We may think that when we speak to another person, the sound of their voice is the most important way we understand them. But masks are teaching us that this might not be the case.
In fact, a large part of verbal communication involves reading the context of what a person is saying – analysing their facial expressions, hand gestures, tone of voice – and it also means reading their lips.
You may not have realized that you read lips, until the point when everyone’s face was covered by a mask during the Covid pandemic. Suddenly, it has dawned on humans that we ALL read lips, to some extent.
“While a normal speech is heard by a person, it is also simultaneously visualized, as the processing areas in the brain combine the visual and auditory data related to the speech.
“The visual data received from the face and mouth of the speaker has an important role in speech comprehension.”
[Source: A Review of the Roll of Lip Reading in Verbal Communication and Lip Reading Techniques, Seyedeh Faeezeh Fazelian, Ali Mohammadzadeh]
What is good and bad about this?
On the downside, it makes comprehension challenging. We’ve all experienced that moment where we cannot understand something a person has said to us in a shop, restaurant, etc., from behind their face mask. It can feel embarrassing to ask someone to repeat what they’ve said (in my case, multiple times!)
On the upside, it demonstrates that we all have a skill we didn’t realise – lipreading.
Furthermore, it gives us greater empathy for people with auditory, visual or verbal impairments – deafness, blindness, inability to speak due to a medical condition, etc.
For us, lipreading technologists, it underlines an important message: If human brains can lipread quite readily, then we can teach AI machines to do the same.
Why does machine-based lipreading matter?
There are many beneficial accessibility applications of automated lipreading. These applications can sometimes transform the lives of people who are unwell, or disabled.
- A simple app that reads lips and ‘voices’ the phrase to the person you are speaking with – useful for anyone who is voice impaired
- An app that can type out a message by reading lips – could be popular with children or video gamers – or when you’re in a crowded and noisy environment like a party
- An app that could take a video of someone speaking, where there is no audio, and tell you if they said a certain word – think Premier League soccer player shouting at a referee!
- An app that allows you and your friends to communicate silently on video chat – your lip movements being converted to on-screen text
- An app for communicating sensitive information on-line, silently in a public setting
And of course, the AI engine behind these technologies means that they can be improving all the time, as people use them.
What’s available right now in lipreading?
Although some of the above are merely at the “ideas stage”, we do have a working automated lipreading app now available.
Check out our SRAVI app to see how it is helping ICU patients at Royal Preston Hospital. Anyone can join our trial – just visit the website to learn more. www.sravi.ai
Or watch this short video that describes our progress with developing the AI-based SRAVI app.
Read more about our technology
LipRead VSR Platform
Liopa’s mission is to provide an accurate, easy-to-use and robust Visual Speech Recognition (VSR) platform. Known as LipRead, it will be focused on the Speech Recognition market. Liopa is a spin out from the Centre for Secure Information Technologies (CSIT) at Queen’s University Belfast (QUB). Liopa is onward developing and commercialising ten years of research carried out within the university into the use of Lip Movements (visemes) in Speech Recognition. The company is leveraging QUB’s renowned excellence in the area of speech, speaker and dialogue modelling to position in the market as a leading independent provider of VSR technology.