Good morning to all the people here and to
all the people who are watching us online.
We are here with Dr. Valentina Ragni
and we are delighted to have her here,
She comes from the University of Warsaw and we want to
especially thank the help she has received for brief periods
at the Faculty of Philology of the UNED for
researchers from universities of recognised prestige.
So thank you very much to the university.
I want to remind those watching us online
that they can send us questions by
typing them in the box below the video.
We will answer them at the end of the lecture.
I will switch to English. We are absolutely
delighted to have Dr. Valentina Ragni here today.
She is here thanks to the
scholarship she received from this faculty,
so we are very grateful to the faculty to have her here.
Valentina is currently a postdoctoral fellow in
the WatchMe project at the University of Warsaw.
She holds a PhD in AVT from the University of Leeds,
where she investigated mnemonic retention of reverse
subtitles in advanced L2 Italian learners using eye tracking.
She is interested in the cognitive and
psycholinguistic aspect of translation,
both as a learning tool and as a professional practice,
and particularly in subtitle reading
as explored by eye tracking technology.
She is a member of the TRADILEX Research Group,
coordinated by Professor Noa Talaván and based at UNED.
Today she will give a lecture titled "An introduction to eye tracking
in Audiovisual Translation methods and applications".
In this lecture she will introduce eye
tracking technology and its development
and examine a few practical applications in the context of AVT,
focusing primarily, but not only, on subtitling
and its application in foreign language learning.
So, the floor is yours, Valentina.
Thanks so much for being here today.
Thank you very much for your invitation.
Let me just share my presentation with you.
I hope you can see it. Let me make it a little bit bigger.
Please let us know if you can't see it.
Without further ado, there's quite a lot to cover
so I would like to start.
You can see the title of my presentation there.
So, the plan for today is, first of all,
to examine basic concepts and definitions
that are useful in this presentation.
Then I will delve into eye tracking and give a
pretty broad overview of the hardware, software,
of the main eye movements, of the metrics
that can be derived from these eye movements,
from the key hypotheses that exist in the field.
We'll also look at reading patterns.
And then, at the end, we will have a look at a few studies,
probably just one, maybe two, three if we're lucky,
but I don't think we'll get there.
We'll see how we're doing time-wise,
(the studies) that use this technology in audiovisual
translation, primarily, as Jennifer said,
but not exclusively to do with learning.
So, first of all, starting with concepts and
definitions, what is Audiovisual Translation?
Some of you may be aware of this already.
This is only one of the possible definitions,
quite an old one as well, by Katharina Reiss.
Audiovisual Translation can be defined
as the translation of multimedia texts,
so verbal texts which are only part of a larger whole.
Larger whole because there is, of course,
additional information there to be dealt with and to be translated,
which is supplied by a sign system other than language.
So, we have pictures, music, noises as well.
I would add gestures, facial expressions,
built-up scenery on the stage or in the film,
if the film is the material that we're translating, slides, etc.
Then, second relevant question for the
purpose of today is what is eye tracking?
Again, this is only one of the possible
definitions and quite a general one,
but eye tracking can be defined as and has been defined as
an experimental method of recording eye motion,
so the eye movements that the human body makes
and the location of these movements, of the gaze,
across time and across tasks or different tasks as well.
It is a very common method increasingly used
to observe the allocation of visual attention
within and outside of Audiovisual Translation.
Well, why should we use this technology in the first place?
Well, eye tracking is thought or is believed, and
I'd like to stress the words thought or believed,
and you'll see why in a second. So, it's believed
to give researchers a window into the mind.
Importantly, eye tracking has some key advantages
compared to other measures of processing
that we use in the psycholinguistic literature,
primarily that it allows for more natural processing.
It does not require a secondary task
such as a lexical decision task,
and it provides a very rich moment-to-moment source of
data, online processing as it unfolds in real time.
And it is also rather unobtrusive, hence the word natural processing.
It's quite easy, as we will see later,
for these tools to go unnoticed,
especially now that they're getting
smaller and smaller, niftier and niftier.
So, it's quite easy for participants to
forget that they're being tracked,
that they are taking an experiment and
being really absorbed by the task at hand.
So, I mentioned the words believe and thought, right?
This underscores something that we think applies,
but is fundamentally an hypothesis, right?
This is very simple. It's the eye-mind hypothesis that
was initially posited by Justin Carpenter in 1980,
which basically states that there is no appreciable
lag between what is fixated and what is processed.
In other words, where we look is where we pay attention.
Now, you may all already think about
cases in which this doesn't apply, right?
This is called "mindless reading".
You may have had instances where you're, for example, reading a book.
So, you go through the words, the lines
and at some point you hit a word,
and this word makes you think about your last trip to the
Caribbean or your family or what you're going to have for dinner.
And then you start thinking about these other
things whilst your eyes continue to move on the line
until you snap out of this and you realise that you need
to backtrack three or four lines and start reading again
because you weren't paying attention, right?
This hypothesis is not always correct, and
mindless reading, which is what I've just described,
is proof that this is not always the case.
But in the vast majority of cases, it does apply,
and it is an assumption that is taken for granted
and is accepted in many of the fields
in which eye tracking is applied.
Here are some of them.
I'm not going to go through all of them, but you
can see at a glance how wide the applications are.
I'm just going to highlight the ones that
will be and that are relevant for us today.
Psychology, of course, psycholinguistics,
more specifically, reading research,
second language acquisition and foreign language
learning, and at the end, translation studies.
I put translation studies right at the end on purpose
because compared to other disciplines,
our discipline is quite a recent adopter of this technology.
Certainly compared to something like reading research, for example,
which has developed together with the development of technology.
So it is thanks to reading research that
eye tracking technology in part has developed.
We arrive a little bit later, but there are certainly
very useful things that we can do with this technology.
So now that we got this sort of basic overview out of the way
let's just delve into eye tracking.
The first thing I'd like to briefly point out
is that there are a lot of manufacturers.
As we've seen in the previous slide, lots of
different fields and areas of applications,
and therefore, you know, where there is
demand, then supply will come.
So there are lots of commercial and non-commercial,
but primarily commercial companies now
that produce eye tracking hardware and software
and produce tracking solutions in all forms.
Now, you can see some of them here.
However, in our field, including in audiovisual translation
you may come across these three primarily.
SR Research is a company that produces the EyeLink eye tracker.
I will be mentioning this eye tracker a lot in my presentation.
This is a very precise piece of hardware
that we use routinely in our lab.
I'll show you this later.
SMI, which is a company that has been discontinued now, but
you will still come across quite influential studies in AVT
and in translation that use that, and Tobii, which is another company
that is still active and produces eye trackers.
So this is one eye tracker produced by EyeLink.
It's what we use. It's called 1000+.
Then we have the SMI here.
You can see it in the portable.
Well, it's not so portable in the laptop mode
This is the eye tracker right there.
And then similarly, the eye tracker is here on the screen,
under the screen, and it's quite chunky, as you can see.
But now we have latest generation eye trackers, namely the 4C.
So as I said earlier, really non-intrusive.
When you have a kit that is so small, as you can
probably imagine, it just quickly goes unnoticed.
Sorry, it's not really easy to move. I wish I had a keyboard.
So other pieces of hardware that you might need are licence keys.
You may or may not need this.
If you're using EyeLink, you certainly will be using this.
You can only use some of the
software if you have this plugged in.
Is there a way of going back? Great.
So you can only use this if you are --
Thanks for that. That might be helpful.
You can only use the software if these are plugged in.
So if you only have one key and you have a large lab
You might have a problem, right?
Chin rest. This is particularly useful.
If you do have one, please do use it.
You will thank yourself as your data quality -- the
data quality of your project will be higher as a result.
You may or may not need to use an external sound card.
We used one in our experiments.
You may use a reaction time box.
This is what you see here in the image.
It's a simple box where you have usually a couple of buttons,
and you might ask your participants to say, "Yes,
I know this English or foreign language word,
so if I press the green button, I will know it,
and if I don't, I will press the red button."
And then you calculate the reaction time.
So how long do certain words take the participants,
for example, learners, to give you an answer?
You can also connect microphones.
So if you are interested in assessing, for example, how people
produce re-voicing or dubbing activities, this can also be done.
And then there are other gimmicks, other items
that you can use, the interface, for example,
tablets, like in this example, and mobiles.
Then there is another piece of kit that you can use.
This is eye-tracking glasses.
You can see here, this is me trying these
eye-tracking glasses in Poznan in Poland at the lab.
You can see they are much bigger than my actual glasses.
Compared, this is the previous model,
so you can recognise them via the frame.
Now they're a bit chunkier, but they don't have this here.
They have this sort of much larger box here.
I just wanted to show you a little bit in terms of the accuracy
how does it work.
Usually, you might want to know that you
can do with eye-tracking is doing facial analysis, right?
So you will want to see the fixations on eyes,
on the nose, on the mouth, right?
So quite precise.
Whereas here, all you have is one area.
So you can see that the person is looking
at the face, but you can't quite assess where.
So see, faces, faces. Then she looks at the
battery, and then she looks around at the end.
But that's about it.
So the accuracy is not great, but this
can be still used in a lot of experiments.
And this allows you to go out in the real world
and carry out research that you
wouldn't be able to otherwise carry out.
For example, truck drivers or driving in general.
So this is a quickly developing area of application.
Another area of application that we were discussing
before is web-based eye tracking.
Again, this is not particularly accurate
compared to the eye-trackers that we use.
But it may make the difference between doing the research
and having participants and not having participants.
Why? Because the technology involves people being
tracked in their own home via their own cameras on their laptops.
Obviously, this means that you lose completely the control
that you would have when participants come to the lab.
However, then you would be able to test very
many people compared to what you would be
limited to by asking people to come to your
lab and having to pay them more for their time.
So here you can really collect a lot of data.
So this could offset then the quality that you will
have, the sort of quality loss that you will have.
So you might have to discard a few more
people, but you will have a lot more.
These are two brands, RealEye and GazeRecorder.
This is me yesterday trying the GazeRecorder.
I have to say the calibration didn't work really well.
My standards are quite high because we use precise eye-trackers.
And I've not used one of these before in my research, but it is
certainly something that could fit your own research project.
So if you have any questions or if you want to discuss
this, let's talk about it at the end of the presentation.
So here I was testing again the same system.
This is a series of spheres that will move when you look at them.
And I just wanted to draw your attention.
I don't know if you can see it from home,
but there are quite a lot of dots here.
So what is happening is: my face is being mapped.
So my eyes and my facial features, nose and
mouth, and this is helping the tracking.
Same thing is happening here.
This is WebGazer.js. It's another web-based eye-tracking system.
This is created by university researchers.
It's open source and it could be a better choice sometimes.
Certainly one that is worth exploring.
You can freely contact the researchers.
They are coming from Brown University in the U.S.
Generally, researchers are really happy to share
their research and to explain how things work.
So this could also be an option.
I want to draw your attention to the
gaze, which is just this tiny dot here.
We won't have time to look at this
now, but I will share the presentation.
So if you have the link, which is right here,
you can go and look at this video that I screenshotted here,
you will see that the gaze is really jittery.
It moves quite a lot on screen.
It's not as precise, but it could be enough for certain designs.
Now, from these systems back to the more advanced ones,
We talked about hardware. There is also software.
There are typically two types of
software that come with these systems.
The first one is software that you use to design your experiment.
And the second one is software that you use
to visualise your data once you've collected
and to a lesser extent to analyse the data.
Although, generally speaking, you also will extract
the data and carry out full statistical analysis
in a piece of statistical software, so not inside the eye tracking.
But it's certain things you can do.
However, please be aware that you may not
only use these two pieces of software.
There may be additional software, both internal to the eye tracker.
So, for example, Split AVI here is a software
that is a piece of small executable file
that splits your video from your audio.
For example, in a study that we ran last year
we wanted to assess the effect of sound.
If we take away sound from a film with subtitles
does this affect the way in which people read the subtitles?
In order to do that, we split the two,
the video and the sound, using Split AVI.
Weblink is another piece of software that
you have to buy a separate license for.
We just got it, so we're really happy to be
using this in the near future in EyeLink.
It records the whole screen.
So, you can, for example, which is how we
are going to use it, you can assess learning.
We will be assessing learning.
This is a project not with eye tracking, not
with the subtitles. In this particular study
we will be looking at how people learn from writing.
And then we will assess how they use different digital tools,
Chat GPT, a normal gloss and online dictionaries.
So, we need to be able to track the whole screen as they type,
for example, in Microsoft Word or in another text editor,
but we also need to track them as they go onto
websites and pages, for example,
to interrogate Chat GPT or to look up for
translations in Google Translate.
Then you may have things like Python, MATLAB, PsychoPy,
which are third-party software that is used to design experiments.
And Translog is a key login software.
You may be familiar with that.
So, it's not just those two pieces of software that I mentioned here.
There may be quite a bit more that you will need to
familiarise yourself with, but lots of possibilities too.
This is very different depending on the manufacturers.
Some manufacturers produce software that is very
intuitive to use, but often this is a bit more limited.
The manufacturer that we use, iLink, produces
software that is extremely complex to use,
but allows you to do a lot of things.
This is an example right here.
So, this is called Experiment Builder, self-explanatory name.
This is what you use to build your
experiment, to design your experiment.
It does look a little bit 1990s, not going
to lie, but it is really customizable.
You can do whatever outlandish idea for
a study you may have. Here you can implement it.
Can you do it by yourself? No.
You will need to ask for some help and you will
run into issues, but that's part of the fun.
And still in iLink, this is once you have collected the data,
this is what the data visualization software looks like.
We can see here, I hope you can see
those two dots, a green and a pink one.
These symbolise the two eyes
the data was collected binocularly here.
And then as you can see, we have the so-called areas of
interest automatically drawn around each of the subtitle words.
So, really, a lot of possibilities since this became available.
When I was doing my PhD, which was about a century ago,
I had to draw areas of interest around
every single subtitle myself manually.
Imagine having to do that, not for every subtitle as
a whole, but for every single word in the subtitle.
You would be still here at Christmas and it
wouldn't be anywhere near as precise as this.
So, recent developments already in the last 10 years.
This is a very fast evolving field.
So, if you are interested, get into it now and you
will reap the benefits as we go forward for sure.
I wanted to show you a little bit.
This is the setup that we had in our previous lab.
So, this is our colleague Agnieszka and
my future boss from next week as well.
She is resting on this chin rest.
So, her chin and her forehead are in contact with this contraption.
It doesn't hurt, I promise.
And it does increase data quality by quite a lot because it
prevents movement of the head outside of the
so-called tracking box, so the optimal tracking space.
This is the eye tracker, the 1000+ that I mentioned here.
This is our experiment display PC.
So, here the participant is reading the instructions.
We have sound, so if I can find the mouse.
So, you can see the speakers.
And here in the corner, you can see the sound card.
And here, importantly, we have the station of the experiment.
In this case, I was collecting the data.
So, here you perform calibration,
validation, and other steps where you teach
the eye tracker to optimally track the person that you're testing.
We are all very different, so it's very important
and crucial for data quality to perform this step.
Now, what kind of data can you collect with these systems?
Well, there are two types of eye movements.
Fixations, which are pauses of the eye
on a specific area of the visual fields.
They are represented by these dots.
And saccades, which are the rapid movements between the
fixation, these sort of jumps, represented by the lines.
I also would like to point out that generally in European
languages, we read from left to right.
That's the direction of travel.
And so, regressions are eye movements
that go in the opposite direction.
And they can be either saccades, so the
jump is backwards, or fixations, so the
resulting periods of stationary periods in
which the eye is fixating on a word or an image.
Now, from these two main categories of eye movements,
we can extract a lot of eye tracking metrics.
We can count the fixations.
We can assess their duration.
Generally, this is in milliseconds.
We can look at dwell time.
This is the total time.
So, if we have an area of interest, for example,
the face of this, the head of this person, then
dwell time will be all of the fixations on that,
or a particular word, all of the fixations.
Then we can look at mean fixation duration.
This is an average, so it's nothing but the
total time divided by the fixation count.
We can look at gaze duration.
This is relevant in reading.
So, when we read, we might go through a line of
text the first time, or a word for the first time,
then something happens, triggers our interest, or causes some sort
of processing difficulty, so we go back, and then we reread that word.
Gaze duration excludes this rereading, so
only in first pass, only all of the fixations,
so, it might be multiple ones, but not the rereading, right?
Time to first fixation, also quite relevant for subtitles.
So, it's possible that the participant is looking at the
image, enjoying the video, and then the subtitle appears.
So, time to first fixation is basically the lag, how long
it takes for the eye to move from the image to the subtitle.
This is particularly relevant for short subtitles, and
particularly in a time where
subtitles are getting extremely fast.
Because if the subtitle is there for one second, or
half a second, and it, let's say, well, let's say one
second, and it takes me 500 milliseconds to go down
there, well, I don't really have a second to read it.
I have half that time, because it takes
me that amount of time to get there.
So, there's some interesting research here
carried out by my colleague, Jan-Louis Kruger,
and he will be publishing, hopefully soon,
something about this, so watch this space.
We can also obviously look at saccade length,
right, those jumps represented by the lines.
We can look at crossovers or shifts, so how many
times people jump from image area to subtitle area.
We know the subtitles are great gaze attractors.
They really attract the eye.
The eye, when the subtitles appears, wants to
go down there, so we can count these crossovers.
Proportional reading time, which is another measure where we look at the
reading time, so the time spent on the subtitle area,
as a function of the subtitle presentation time.
If I spend 500 milliseconds on a one-second
subtitle, well, that is 50% of the time.
If I spend 500 milliseconds on a six-second, two-line,
full-of-text subtitle, well, that's a fraction of that.
That is not half of the presentation time.
This has meaning, so for the rest of the time, you may
go back to the images and enjoy those, for example.
And then skipping rates, which
words or entire subtitles are skipped.
Now, I have a little test for you all, including the people
at home, so please feel free to write in the chat later.
I have a little short video to show you, and then I
will be asking you all one question, including the people in the room.
Okay, so please pay attention.
Oh, of course.
Okay, hopefully it works.
Right?
And one thing is the different colours represent
different participants here, and the credit goes to my
colleague, Agnieszka Szarkowska, who ran this experiment
and collected this data, and the video is hers.
Okay.
So, my question to you all is: what were they eating?
Can anyone tell?
No.
People in the room cannot tell. No.
People in the room cannot tell.
I can't tell because I see it now.
Is anyone managing to say this in the chat?
Okay.
And then the next question is,: did you look at the subtitles?
I did.
I tried.
I mean, obviously, they were a bit covered by the...
There were so many colours, and it's the first time I
see a video like this, I was a bit overwhelmed.
Of course.
Information, visual.
But you remember looking at the subtitles, and you don't remember
the food, so you definitely look at the subtitles a little bit,
and you didn't look at other parts of the screen, which makes
my point, which is, subtitles are great gaze attractors.
So much so that sometimes between the main faces
of the participants, which are also great gaze attractors,
and the subtitle area, sometimes we miss things, right?
So, we miss things in the video, but also this means the subtitles
are a great tool that we can use for second language learning
and foreign language learning purposes, right?
They do have this potential.
Whether they are useful or not would have to be tested, and we
have plenty of research now that tells us that they are useful.
But this initial sort of observation that they do attract eye
movements is quite relevant, and they do so, research shows,
even when they are in a language or
in a script that we don't understand.
So, if we don't speak Russian, and I put some
Russian subtitles there with a Cyrillic script,
oftentimes it happens that participants
might tell you, "Oh, no, I didn't look at the subtitles.
I cannot understand it," and then you look at the gaze
patterns, and you see that they still went down there, right?
So, this is potential, a really good potential for exploitation
by language teachers in the classroom.
I also wanted to show you a typical eye behaviour again, so
you can see this is just what we were talking about earlier.
This is an interest area on the word, basically a box.
We have the first fixation, we have first pass, which
is this gaze duration, only four and five
because fixation number six goes out and then seven goes
back in, and total time would be all the fixations on this.
This is just what a typical eye movement behaviour
might look like, including subtitle reading.
So, this is the sort of fine-grained information that we have access to.
There are some challenges involved with eye tracking as well.
I am not going to lie, and I would like to, as I said
in my introduction to, in my abstract, to mention these.
Data quality, as I said before, you have to be quite
careful with that, but there are things you can do,
like using a chin rest and calibrate and validate before
you start testing to ensure the data quality is acceptable.
Data analysis, as I also mentioned, is carried out outside
the software, so you need to learn statistics a little bit,
or maybe you have access to a statistician.
I didn't, so I had to learn it myself.
And it was a little bit challenging, but now, you
know, as a result, I'm a better researcher, I think.
As we said also, we have extra hardware
and software items that we made to connect,
that we might need to connect to the system, so something else to learn.
Overall, there is a fairly steep learning curve in
using the system, but once you've overcome this,
you know, it's fairly easy to use the system that
you have, and there are lots of help sources online,
and conferences and things that can help you in that.
You will have to learn to do troubleshooting, so problem
solving usually, because, like in other aspects of life,
things can go wrong, and you have to try and solve them.
Mostly they do because there is participant variability.
Ultimately, we're dealing with human participants,
so sometimes, you know, I had participants come to
the lab with some absolutely unheard of eye condition
that they didn't mention before and then the eye tracker doesn't
want to work, so what do you do then?
So this doesn't happen very often.
However, you may have to operationalise variables,
so you have to know what you are dealing with, and the
same item, the same eye movement, for example, fixations,
can mean two different things in two different fields.
So if I'm a marketing expert, and I want
to investigate attention to two products,
then more fixation on one product than another may indicate interest.
But if I'm comparing two texts, one full
of academic words that are really rare,
compared to one that has really frequent words, then more
fixations on the academic text may be an index of difficulty,
not of interest, right? So the same eye
movement can have different meanings.
And so we always recommend, please define your
variables and what you take them to mean.
And there is a little bit of variability.
The same fixation, the same metric can mean two different things,
and two different eye movements can be, or the same underlying
eye movements, sorry, can be defined in two different ways.
But they're actually talking about the same thing. So
terminological clarity in your
papers, if you use this, is also key.
Now that we've got this sort of broad overview
of eye tracking, I will try and give you one,
possibly we might have time for two studies that we run in our lab.
The first one is about subtitle speed and individual differences.
So we compared L1, so first language, native
language, to L2, second or foreign language,
and we were looking at whether these effects
are different between these two cohorts.
First of all, why subtitle speed? Well, subtitle speed
is basically the rate at which subtitles are displayed,
and is usually measured in either words per minute, or
better, as it's more precise, characters per second.
Now, until recently, speeds that were
deemed reasonable were around 12 to 15 CPS.
However, there has been a lot of evidence that
subtitle speeds are increasing quite dramatically.
And, for example, Netflix has 20 CPS as their
maximum allowed speed, at least in their home market,
so English subtitles for English video in the U.S.
However, research, particularly by Kruger,
Wisniewska and Liao, but others as well,
showed that oftentimes they go above this threshold, right?
Sometimes they're extremely fast, at 30, 40, even 50
characters per second, which is basically unreadable.
So what is the implication of that? What is the concern?
Well, if subtitle speeds are too high,
viewers, and particularly language learners,
may not have sufficient time to actually read the subtitle.
So all of this potential that we talked about before,
all of a sudden goes out of the window, right?
So it is very important to look at subtitle speed.
So we did exactly this in the study.
We took 80 participants, 33 native speakers from the UK,
50 non-native speakers of English from Poland,
who, however, had high proficiency,
C1 of the Common European Framework of Reference for Languages.
We showed them two clips from this film, Don't Look
Up, with Jennifer Lawrence and Leonardo DiCaprio,
a really good film, if you haven't seen it, by the way.
And we looked at gaze duration, total times,
PRT, so proportional reading time, and word skipping with
the iLink 1000+ eye tracker that I showed you earlier.
And we also looked at individual differences
within the subjects, within the participants.
So age of the participants, working memory, video familiarity
means whether they had seen this film before.
Does that affect processing? And experience with
subtitling means, you know, are these people,
are these viewers used to watching subtitled material or not?
And we also looked at verbatim recall,
which was actually a recognition test.
So we gave them four options and we asked them to
recall exactly which words were used in the subtitles.
So this is what the data looked like in the data viewer.
So as you can see, our binocular data
collection and the very precise interest areas.
Let's look at some data. So you have gaze
duration on the left and total times on the right.
And what we can see, and subtitle speed on the X axis.
So we can see the subtitle speed increases as we go left to right.
And then native speakers are the blue line
and non-native speakers are the yellow line.
So here, what we see is the language learners and native
speakers are more or less similarly affected.
The lines are parallel and in both cases they
are decreasing, which means that as speed grows,
the time that they spend on the subtitle area, both in first
pass, right, and overall the total time steadily decreases.
So there is less time to look, to read, and people
don't spend as much time in the subtitles as a result,
which may have effects on processing and on memorization as well.
Then we also looked at proportional reading time and word skipping.
And here you can see the situation is a little bit different.
So what we found was the mean proportional reading time was 45%
So viewers spent more or less half of the time of the
subtitle presentation time to look at the subtitles.
The other half, they went to the images and processed those.
And also we found that as speed increases, so
again, as we go left to right on the X axis,
learners spent proportionally longer
than native speakers on the subtitles.
So this you can see here, right, the line,
the yellow line is above the blue line.
Not only that, but the two are, they're
not parallel, they are crossing, right?
So there is an interaction.
So actually this means that language learners
were more affected than native speakers.
So they tended to just spend so much more time in the subtitle area.
They weren't able, probably because they weren't able to
make use of the audio in the same way as native speakers.
Native speakers can ignore if they want the subtitles.
They don't because there is automatic
reading behaviour that we mentioned earlier,
but they can do without the subtitles, right?
They can fully make use of the sound, whereas
non-native speakers, even at high proficiency,
we know that listening skills are typically quite hard.
So that is our explanation why that is happening.
And we also look at skipping and we saw
that the L2, so non-native speakers,
you see the line is lower here, skip less.
So they really, they need the subtitles more, right?
And therefore, speed affects them more.
Now, so to summarise, the results indicate
that the non-native speakers, the L2ers,
were more affected by fast speeds.
They spent more time in the subtitle area.
They skipped fewer words and they also had lower recall accuracy,
although I cannot present all of the data
today, but I'd still like to mention this.
And we also found, importantly, that factors like
age, working memory, experience with subtitling,
so people who were used to subtitles versus people who were not,
and people who had seen the film versus people who had not,
were all factors that were found to influence the viewing process.
So these are all things to keep in mind when we
investigate learning which is the next step.
We first look at how people process and
then in this future project that we,
the project that I will be part of from this year, we will
be really investigating the learning via the subtitles
now that we know a lot more about the processing.
So we know what to test.
We know that age has an influence.
We know that we should always ask
whether people have seen the videos or not
and perhaps add this to our statistical models.
How much time do I have left?
15 minutes.
So I have 15 minutes.
So perhaps we can look at this study as well.
So here we looked at cognates.
This is another study I ran with a colleague.
It's just come out this year.
Cognates are words that share some surface similarity,
so there is some orthographical overlap
They can be identical or non-identical.
For example, the word chaos in English is
written in exactly the same way in Polish.
It's pronounced chaos, so the pronunciation is
different, but when you go and read the subtitles,
this may activate already the word in your native language.
And the assumption is that this is facilitating in processing,
so you might spend less time on these words because
you take them for granted and you know them.
So in this case, it's an identical cognate,
but then we also have non-identical cognates,
like critical in English with "krytyczny" in Polish, and
here you can see that the words are very different
Perhaps the pronunciation is similar, but the writing,
if you look at the Levenshtein distance
and you know how many characters they have in
common, these words, they're not very close.
And this is an intermediate word, "expert" in English,
It's not exactly identical, but it's quite similar.
We wanted to know how these words are processed by
non-native English speakers watching English subtitles
when there are these similarities with their native language.
These similarities don't appear, we don't have any Polish in the video,
but it might be activated in the mind of
the learner as they process the input.
So we asked a few questions, but we'll only look at one today.
Are cognates in the subtitles processed
faster or slower than non-cognates,
so words that don't share any similarity in surface form?
And again, we looked at speed,
right? By now it should be clear that
speed is very important in the processing of audiovisual material
and therefore potentially in the learning
that can come from the use of these materials.
So we looked at both things and we examined 35 Polish
participants, again with high English proficiency.
We showed them three clips from a BBC documentary called Planet Earth.
We looked at 42 keywords, 21 non-cognates and 21 cognates,
which were known to the participants and which were
matched for lots of things, frequency, word length, etc.
So they were directly comparable.
We looked at gaze duration, total reading time
and fixation counts with our 1000+ eye tracker.
And we used mixed effect modelling,
which is a type of regression analysis.
We looked at the effect of three variables,
cognateness, whether the word was a cognate or not,
subtitle speed, which was a categorical variable.
We used 12 CPS, 20 and
28, which is an extremely fast speed.
And then cognateness, the cognateness and speed interaction.
We wanted to know if the two variables interacted.
So are cognates processed faster or slower than non-cognates?
Well, when it comes to gaze duration,
this is an early measure.
The first time you go through the subtitle words without
the re-reading afterwards, we found absolutely nothing.
There was no effect of, so you can see here, speed,
the three speeds, 12, 20 and 28.
And this is gaze duration in milliseconds.
So we found, although you can see some differences
here, these differences were not significant.
So no effect for cognateness, no effect
for subtitle speed and no interaction.
Moving on to total reading time.
Again, we found no effect of cognateness.
So whether the word was a cognate or
non-cognate, it was processed similarly.
No interaction with speed and no difference
between the fastest two speeds from 20 and 28.
However, we did find a significant difference going from 12 to 20.
So there seems to be a threshold after which
things get fast and therefore processing decreases.
And then after 20, it wasn't that.
The further increase from 20 to 28 wasn't significant.
Then similarly with fixation count, no interaction,
no effect of cognateness, but even more significant,
as you can see from the steepness of this line.
So these lines are almost flat, right?
Cognate and non-cognate.
But here there is a steep decrease and it
is even more significant than here, right?
So what does it all mean?
Well, there was no effect of cognateness, as
we said, in any of the measures, early or late.
There was no effect of speed and no speed and
cognateness interactions in the early measures.
And however, keywords, all of them, regardless of their
type, showed significantly longer total reading times
and higher fixation counts in the slower
subtitles than in the faster ones, right?
Only, however, in the total measures, not in the early measures.
What does this actually mean?
That when viewers have more time, so at the lowest speed,
they may not fixate on these keywords in first pass.
So at the beginning, it looks like there are
no differences, but they tend to revisit.
So they might go back to those later.
Cognates and non-cognates alike, right?
So this actually means that if you don't have time, so
if speed is too high, you may not be able to do this.
And learning might therefore be impacted.
This is yet to be tested, whether learning was affected.
We only looked at processing, but it gives us an idea of
what might be happening, how might learning be affected.
So slower subtitle speeds, unlike high subtitle
speeds, allow for rereading where it's necessary.
So imagine that you have some difficulty, you can go
back and clarify and use that subtitle to then
understand better what is going on or the
subtitle might be able to tell you what you've
just heard and you go back to that word.
There's no main effect of cognateness, possibly because
there was a low proportion of identical cognates.
So those words tend to be processed much faster.
And we only had three of them in there.
Most of them had some overlap or were inidentical.
So there's space for some more research with
higher proportion of identical cognates,
and we may find an effect of cognateness there.
And also, importantly, in terms of methodological development,
we should always look or we should strive to look at both
early and late eye tracking measurements,
because if we had only looked at gaze duration,
we might have erroneously concluded that
it doesn't matter. Speed doesn't matter, right?
Because in gaze duration, there was no effect anywhere.
But then in later measures and in the counts
and the durations, we did find differences.
Making sure that you collect
data on more than one eye tracking variable is a really good idea.
Do we have time for the last one?
Yes, yes.
Wonderful. I must have spoken even faster than usual.
Sorry about that. So the last study is very brief.
Only three slides, and it is not about learning.
And it is not about subtitling. It's about dubbing.
So it starts from a piece of research that
was carried out by a Spanish researcher,
which you may be familiar with, Pablo Romero Fresco
who looked at whether viewers,
Spanish viewers, avoid looking at the
actor's mouth when a film is dubbed.
So what he did is he took the film Casablanca and
compared English participants watching the original,
obviously English, English without subtitles,
just with English original soundtrack
and then compare this to the version of Casablanca dubbed
into Spanish shown to Spanish participants.
So there were no subtitles anywhere.
What he noticed is that English viewers watching the original spent
75 percent on the eyes and 25 percent of the time on the mouth.
However, when you are in the situation where you
have the original video, but with a Spanish dub,
what's happening there is that what you hear
doesn't match the lips of what you're seeing.
And he found that Spanish viewers fixated
only five percent of the time on the mouth.
Obviously, then as a sort of form of control, he also looked
at Spanish people watching a Spanish video with Spanish audio.
So where the lips would match.
And he found that they would,
they would fixate normally like the English.
So he turned this the dubbing effect.
Now, one of our PhD students at the University of Warsaw
has looked at this in the Polish context.
And this is actually I mean, if there are any master's
students, I think this was her master dissertation.
So she was quite adventurous to embark on something like this.
But the study is very simple, really.
So if you are intending to do this, you can.
I wanted to provide this example for your students as well.
This can be done at master's level as well. Right. It's not impossible.
And it was even published. So, she looked at
the Casablanca version with the Polish lector.
Of course, in Polish. Well, I don't know if you're
aware of this, but in Polish they don't dub as such.
What they do is they have this lector, a person.
It's really strange for me being Italian and probably for Spanish
speakers as well to have just one person dubbing all the voices.
But this is this is what happened.
What usually happens.
She created areas of interest on the eyes and on the mouth.
And then she tracked participants
to see if they avoid looking at the mouth.
And she found that they do not.
I want to just keep it simple.
So no graphs or anything. I just wanted to mention this.
You can find full details at this publication here.
I just wanted to give very brief example of something that isn't
subtitling, isn't for language
learning, but still uses eye tracking.
Eye tracking can be used in audio
description as well in live subtitling,
in subtitle creation, not only in reception
studies, but also in production studies.
So lots of possibilities there. I think I will conclude here.
I have a couple of slides of acknowledgements.
One is for the AVT lab, which is the laboratory where I worked
for the last three years, where this research is coming from.
So you can see the various PhD students
and other researchers that work there.
And as part of this project I worked on
Well, as part of this lab, I worked on the WatchMe project, which is an
international project that includes other
researchers that you may be aware of.
Sharon Black from the University of East
Anglia, Professor David Orrego-Carmona
from Warwick and Jan-Luis Kruger from Macquarie University in Sydney.
All of which collaborated with us on a number of
publications, some of which are yet to come out.
So watch this space. And then I also would like to
acknowledge the work that my future
research team, some of the people are
So you see some of the people are the
same. So the cognate study that I mentioned
was run by, whose first author, my
colleague, Bruno Silva and by ourselves.
There will be a lot more coming. This is the exciting part.
If you're interested in language
learning, we will be publishing a lot,
hopefully, about language learning via subtitling and via audiovisuals.
So I think I will stop here. Some publications that came out recently.
This is 2024. One of the papers on subtitle speed.
This is a replication study. The Cognate study that I mentioned
just now has come out in language learning and technology.
So it's freely available if you want to go and browse.
And then last but by no means least,
I'd like to also mention a chapter that I
wrote recently for a book that's just come out.
And I'm sure that some of the people connected online, certainly some
in the room will know this because Jennifer wrote a chapter as well.
And Professor Talaván wrote the preface for the for the volume.
And there's a couple, there's at least one
study that comes from from Tradit, right?
I think so. There is this, yes, which is a brainchild
of UNED, of the very university that we are at now.
So certainly relevant. I will stop here.
There is a lot of literature on eye tracking methodology.
So if you feel a bit overwhelmed by the amount
of things that you have seen today, don't worry.
You can read it in your own time. Not listening
to some absolutely fast speaker like myself.
I apologise. I will leave the presentation with
you and I included the references for this in the bibliography.
So you can go and browse, browse this if you so wish.
So I will stop here.
Thank you very much for your attention.
And if you have any questions, please fire away.
Valentina, thank you very much for the
informative and inspiring presentation.
So I would just ask the audience if they have any question.
And of course, people listening online.
If they have any question, please just click
on "consultas" and send your questions there.
There was a comment at the beginning.
I got this email from one of the people online asking for the.
Well, I'm going to read it.
(Spanish) I guess.
Can you repeat the beginning of the slides?
Yes, absolutely. I can leave the slides.
So you have all the references.
You can, as I said, you can go through it in your own time.
I mean, it's quite a lot to take in.
I'm not going to. Well, not about the certificate.
So of course. Where will they be able to download
the certificate after submitting the summary.
We will leave the presentation along with the link to the video.
Exactly. So basically, the people who registered for this seminar,
they will receive an email with all the information where
they can find the link to the slides and also the certificate.
So please be aware that you can download the certificate as
soon as you submit a brief summary of this seminar
I'm glad that someone wants the slides. That's a good sign.
Yes, indeed. It was very interesting, Valentina.
And of course, the head was spinning,
thinking about all possible applications.
And since you've been so generous to show a lot of studies
and very recent studies as well as forthcoming studies.
So my question would be, what would you recommend?
Where SLA research with eye tracking technology should
go? Which is the direction, at least your interest.
So in my interest, I think is increasingly in cognates because
we are, I think in particular in individual translation,
we tend to not have very precise theories.
And what contributes to having precise theories
usually is this concept of cognates.
So in the theories that exist. So when people have tried so
far to include, to create theories, cognates appear there.
Right. Because this tells us something about the
mental lexicon which is an assumption.
How does the brain work?
How does the brain deal with linguistic information?
Also. Through the subtitles or through audio,
audiovisuals in general.
I am interested in cognates because this is going
to allow us to refine our theoretical assumptions.
As a researcher that
that looks at testable hypotheses,
I think it's important to have theories
that you can really put to the test.
So I'm hoping that we will be doing a lot more research
on cognates. that you can really put to the test.
So I'm hoping that we will be doing a lot more research on cognates.
There is already a lot of research on
cognates out there without subtitles.
Now, the moment you add: images, sound,
proxemics, facial expressions, all of this.
So this beautiful semiotic system that is
a film or an audiovisual material, things may change.
So I find this area
particularly interesting.
And then perhaps also devising ways in
which we can test learning.
This is next challenge for us.
So we currently have this pre-test, post-test format.
We are going to start to look at that within
the subtitle, within subtitling with eye tracking.
I think essentially to put together these two things will
enable us to go one step further than what we've done so far.
So, for example, yourself, you have run some amazing studies
about learning from subtitles without the eye tracking components.
We have run a lot of eye tracking studies about
subtitling without the learning component.
What I would like to see more of is the two together. Right.
Because then you can use one to explain the other.
And the two really will give us a more comprehensive
picture Because then you can use one to explain the other.
And the two really will give us a more comprehensive picture
of how our students can or should,
if we have some clear findings, approach these audiovisual
materials to better exploit them to their advantage.
That sounds great. And especially because
we have you here. So that would be great.
If I'm allowed, just one last question. What about. Oh, sorry.
Yeah. Thank you very much for the conference.
I would like to ask you, you have done research with
children, obviously children who can read and teenagers.
Thank you. So not myself, but and let me just.
So I don't work with children myself.
Children are extremely difficult to work with.
And so absolute respect to whoever is doing this
which is my colleague Sharon Black.
So I think in the journal of audiovisual translation.
So she works with teenagers.
Right now, I can't think of people who work with younger
children, but certainly with teenagers, she has done some work.
And this is the very beginning.
Not many people have done this.
This is a very interesting question.
I'm very grateful for it because we know.
There is something called the
critical period. So we may become bilingual.
So we may actually have. But if we learn in these early stages
of life, rather than if we learn a foreign language as adults,
we may get really good at both languages.
When when the brain is still developing.
So I think it is really essential to look at how
children When when the brain is still developing.
So I think it is really essential to look at how children
and teenagers learn from audio visual material.
And it's a shame that there's not that much there.
But the first steps are being made.
And I'm very glad that it's one of our
But the first steps are being made.
And I'm very glad that it's one of our
our project partners that is doing this.
So please check her out.
I think she has something published.
Thanks. Thanks for the question. That's that's really great.
It's 12. Do you have any other questions online?
There is a question online. It's just in terms of the beginning.
It's about confirmation of the registration.
I'm sure it will be fine. It was going to check. But in
case anybody is that doesn't get the email that you're
going to send them on Monday with all the
information they can write to either you or me.
And you have the list, right? Exactly.
I will email all the people registered.
But if you have any problem, you can
email me and/or Professor Noa Talaván.
She will reply also.
OK, well, I think we are closing because it's 12.
So thank you very much, Valentina, for being here.
Thank you very much for being so generous with all the presentation.
Thank you. Thank you.
Thanks.