Placeholder Image

Subtitles section Play video

  • [MUSIC PLAYING]

  • ELENA NIEDDU: I'm excited to be here

  • and to talk to you about the In Codice Ratio project--

  • that is a project going on at Roma Tre University--

  • and to talk to you about TensorFlow

  • help us build a module that is able to transcribe

  • ancient manuscripts in the Vatican Secret Archive.

  • So some introduction first.

  • This is our team.

  • On the right, you can see paleographers and archivists.

  • And on the left, there is us, a data science team.

  • And that's why I think the name we chose, In Codice Ratio,

  • reflects us very well.

  • Because it's a word play between the Italian

  • and the Latin meaning of the word "codice."

  • Now, in Latin, "in codice ratio" would mean a knowledge

  • through manuscripts.

  • But the word "codice" in Italian also means software code,

  • so it's also knowledge through software,

  • which is exactly what we're planning to do.

  • And so you might ask yourselves, what

  • brings paleographers and archivists and data scientists

  • together?

  • Well, they have one problem in common.

  • They both want to discover knowledge from big data.

  • We are used to think of big data as something

  • that happens in the web.

  • But actually, historical archives

  • are endless source of historical information,

  • of important information, of cultural information.

  • And just to give you a scale of how large this information can

  • be, let's just compare for a second

  • the size of the Vatican Secret Archive

  • to the height of Mount Everest.

  • Now, if you were to take each shelving of a Vatican Secret

  • Archive and stack it one top of the other,

  • you would get to 85 kilometers tall.

  • That is about 10 times the size of Mount Everest.

  • And the content spans the centuries and the continents.

  • For example, there, you have examples

  • of letters coming from China, from Europe, from Africa,

  • and, of course, from the Americas.

  • So what is our goal?

  • Our goal is to build tool and technology that

  • enable historians, archivists, and scholars

  • of the humanities in general to perform large-scale analysis

  • on historical archives.

  • Because right now, the process, let me tell you,

  • is entirely manual.

  • You still have to go there, consult the documents manually,

  • and be able to read that very challenging handwriting.

  • And then, if you find information

  • that may be linked to another collection,

  • then you have to do it all by yourself.

  • But first, we have to face the very first challenge that

  • is when you are dealing with web content-- for example,

  • if you want to extract data from the internet-- well, that's

  • already text.

  • And when we said we're dealing with the historical documents,

  • that's often scans.

  • And traditional OCR is fine for printed text.

  • But then you get to this.

  • This is medieval handwriting.

  • It's Latin, a language nobody uses anymore.

  • It's a handwriting nobody is able to write or read anymore,

  • for that matter.

  • It's heavily abbreviated.

  • And still, you want to get texts out of it.

  • So you might want to train a machine learning module.

  • Of course, you want.

  • But then, we come to the second challenge.

  • And that is scalability in the data set collection process.

  • Now, the graph you see there is a logarithmic scale.

  • And it might show you something that you already

  • know that is known as the zip flow that tells you

  • that there is very few words occurring humongous times.

  • And then, most of the words, they do not occur that often.

  • What does that mean for us?

  • That if we want to collect data, for example, at word level,

  • at vocabulary level, this means that we

  • have to annotate thousands of lines of text, which

  • means hundreds of pages, OK?

  • And similar systems do exist.

  • They are state of the art systems.

  • But most of the paleographers, even when

  • they know of these tools, get discouraged in using them

  • because they say, well, it's not cost-effective for me.

  • Because it can take up to months, or even years, of work

  • on these documents just to get a transcription system that they

  • will maybe use once or twice--

  • I don't know-- whereas they would like to do it faster.

  • So we asked ourself, how can we scale on this task?

  • And so we decided to go by easier step, simpler step.

  • The very first things that we did

  • was to collect data for single characters.

  • And these enabled us not to involve

  • paleographers but people with very less experience.

  • We built a custom crowdsourcing platform

  • that worked pretty much like CAPTCHA solving.

  • What you see there is an actual screen from the platform.

  • So the workers were presented with an image

  • and with a target.

  • And they had to match the target and select

  • the areas inside of the image.

  • And in this way, we were able to involve more than 500

  • high school students.

  • And in about two weeks' work, we made

  • more than 40,000 annotations.

  • So now we had the data, we wanted to build a model.

  • When I started working at the project,

  • I was pretty much a beginner in machine learning.

  • And so TensorFlow helped me put in practice

  • what I was studying in theory.

  • And so it was a great help that I

  • could rely on tutorials and on the community

  • and, where everything else failed, even the source code.

  • So we started experimenting, and we

  • decided to start small first.

  • We didn't want to overkill.

  • We wanted the model to fit, exactly, our data.

  • So we started small and proceeded incrementally

  • and, in this phase, in a constant cycle

  • of tuning hyperparameters and model tuning

  • and choosing the best optimizer, the best thing initializers,

  • the number of layers and the type of layers,

  • and then evaluating and training again.

  • Then we used Keras.

  • It was good for us because it allowed us to keep

  • the code small and readable.

  • And then, this is what we settled for.

  • It might look trivial.

  • But it allowed us to get up to a 94% average accuracy

  • on our test characters.

  • So where does this fit in the whole scheme

  • of the transcription system?

  • It's there in the middle.

  • And it's actually, so far, the only [INAUDIBLE] part,

  • but we are planning to expand.

  • And you will see how later-- we will see how later.

  • And so we have the input image.

  • So far, we're relying on an oversegmentation

  • that is old-school.

  • It's a bit old-school, but it allows

  • us to feed single characters or combinations of characters

  • inside of the classifier, which then produces

  • a different transcription who are ranked according

  • to a Latin language model, which we also build from publicly

  • available sources.

  • How good do we get?

  • We get about 65% exact transcription.

  • And we can get up to 80% if we consider minor spelling errors

  • or if the segmentation is perfect.

  • If we had perfect segmentation, we could get up to 80%.

  • We will see that this can be more challenging.

  • OK.

  • So what are our plans for a future?

  • We're very excited about the integration

  • of TensorFlow and Keras.

  • Because I described the process as being fully Keras.

  • What we actually found out was that sometimes some feature

  • were lagging behind, and sometimes we

  • wanted to get one part of the features from Keras

  • or from TensorFlow.

  • And so we found ourselves doing lots of--

  • I don't know if that's your experience, as well--

  • but we found ourselves doing lots of back and forth

  • between TensorFlow and Keras.

  • And now, we get the best of the two worlds,

  • so we're very excited about that.

  • And so how do we plan to expand our machine learning system?

  • First thing first, we are trying U-Nets

  • for semantic segmentation.

  • These are the same Nets that achieved very good results

  • on medical imaging.

  • And we're planning to use them to get rid

  • of these tricky computer vision, old-school segmentation.

  • And that would also achieve the result of having classification

  • together.

  • Because this is semantic segmentation we're talking of.

  • These are some preliminary examples

  • that work particularly well.

  • Of course, there is work still that we have to do.

  • And then, of course, since there could still be ambiguity,

  • we could do error correction and then transcription.

  • But I think this would be, in itself,

  • a significant improvement.

  • And another thing we're experimenting with

  • is enlarging our data set.

  • Because we don't want to stick to characters.

  • We want to evolve.

  • We want to move to word level, and even

  • sentence level, annotated characters.

  • But still, our focus is scalability

  • in the data set collection.

  • So we want to involve paleographers

  • as little as possible.

  • So for example, this is our generated inputs from GAN.

  • But we are also planning on using,

  • for example, a variational autoencoder

  • so that we can evolve our data set

  • with little human interaction--

  • the less we can.

  • And in the end, this would bring us to actually use

  • sequence model that could take full advantage of the sentence

  • level context, for example, and could even

  • be able to solve things that we couldn't be able to solve

  • with single character classification-- for example,

  • abbreviation.

  • In this kind of text, many words occur abbreviated, for example,

  • just like you would text.

  • In some texts, you would say me too

  • and use two, the number, or 4U.

  • And that's the same with this kind of manuscript.

  • And that's one of the application you could have.

  • Also, we are planning to use sequence models

  • to get to a neural language model because so far,

  • we only have experimented with statistics.

  • And one last thing before I let you go.

  • I mentioned the people in the team,

  • but there is so many people I would like to thank

  • that were not in that slides.

  • And first of all Simone, who should have been here,

  • but he couldn't make it.

  • And he was my machine learning Jedi Master.

  • And then Pi School of AI andbastien Bratiéres and Lukasz

  • Kaiser for their amazing mentoring.

  • And Marica Ascione, who is the high school teacher that

  • actually allowed us to involve those students that

  • were part of the platform.

  • And, of course, all of the graduate

  • and undergraduate students that worked with us and help us

  • achieve what we have achieved and what we

  • plan to achieve in the future.

  • And of course, thank you for your attention.

  • [APPLAUSE]

  • [MUSIC PLAYING]

[MUSIC PLAYING]

Subtitles and vocabulary

Click the word to look it up Click the word to find further inforamtion about it