[Music]
Hi! In this video we will tell you step by step how our research team prepared, transcribed and
trained a specific model to transcribe the Diario de Madrid using historical text recognition tools.
Ready? Let's get started!
To automatically transcribe the Diario de Madrid, we opted for using Transcribus,
a platform specializing in recognizing texts in historical documents.
Among the available models, we selected the Spanish
Golden Age Theatre Print 1.0, created by Álvaro Cuellar.
Although it was trained mainly with works from the Golden Age,
its type of text is very similar to that of the Diario de Madrid.
That is why we thought it was a solid starting point for our project.
Now, before being able to transcribe, Transcribus
needs to understand the structure of each page.
We call this process "Layout Recognition".
Let's see what the layout is and why it is so important.
Basically, the system must correctly identify where the text begins, where there are columns
or tables, and also distinguish between text and decorative elements, such as stains or stamps.
However, we find several problems.
Sometimes Transcribus confused stains with words,
misdetected columns or disordered text lines.
That is why we realized that, to work with columns,
it was better to Mark them as independent regions.
This way, Transcribus could read in the correct order.
Let's see now how the layout annotation was organized.
Once the problems were detected, we realized that
it was essential to annotate the layout coherently.
To achieve this, we developed a common guide.
This guide gave solutions to problems such as the following.
When we found a table, we divided it into several
regions so that it could be read correctly.
If there were stains that were not part
of the content, we simply eliminated them.
And if a word could not be read, we marked it as illegible.
Thanks to this methodology, in a short time we had
already corrected 203 newspapers manually, a total of 436.
Even so, we decided to train a new
model for this newspaper in particular.
The reason?
Well, although the original model was a good first step,
we soon saw that it made mistakes in crucial aspects.
For example, it did not recognize the numbers
well, which are essential for dates and data.
It also had errors in capitalization, which are
important for detecting proper names, places or titles.
For all this, we decided to train a specific model
using real documents from the Diario de Madrid.
According to Transcribus' recommendations, for a good training,
you need at least 75 well-transcribed pages or about 15,000 words.
To measure the quality of our model,
we use CER, the character error rate.
The lower the CER, the better the recognition.
At this point, it was essential to have quality manual transcriptions.
How did we do it?
Well, a team of three students in practice and several
researchers dedicated a total of 148 hours to download documents,
correct layouts and transcribe texts by hand.
The effort paid off. We got 193 pages ready to train
the first specific model of the Diario de Madrid.
Before training, we reviewed all the transcriptions following a common
standardization guide to ensure the coherence of the transcriptions.
Some fundamental criteria were
always keep the original spelling, including
the old uses of letters and capital
letters; convert Roman numbers into Arabic;
and write illegible if a word could not be read.
The last step in working with Transcribus was the export of documents.
Transcribus offers several export modes, but the most
useful for the project is the export to Word or Doc files,
as it allows you to search for text editing tools.
Currently, the download to Word is only free
through the Quick Text Recognition option.
This export is especially interesting because when a
word appears separated by a line at the end of a line,
Transcribus automatically joins it in the Word transcription,
which does not happen with other types of files such as TXT.
And why is this so important?
Because to apply word recognition models we need them to be compact.
If the scripts remain as they are in the export
to TXT, the performance of the models is affected.
Now, in the Clara HD project, we do need
the TXT newspapers for experimentation.
To solve the script problem, we designed a script that
automatically converts Word documents into
TXT files, correcting these separations.
Thanks to this previous process, we were able to train a model that perfectly
adapts to both the graphic style and the language of the Madrid newspaper.
A model that today is a highly valued tool
in historical and humanistic research.
Would you like to know more about this project?
Well, don't miss the next video.
Transcription by ESO; translation by —.