I was intrigued by an internship vacancy from Liopa, a company that, at its core, uses machine learning to effectively read peoples’ lips from videos. So I made an application for the role – and, after a successful interview, I happily accepted the post.

I have now been working at Liopa for a little over nine months. From creating custom neural network animations in Matlab, web servers using flask, making use of the Google Speech API, complete Web SDKs in JavaScript to creating custom data augmentation algorithms for videos in python, it is safe to say that every day is different.

I’m based in the Queen’s University Institute of Electronics, Communications and Information Technology  (ECIT) building. There are a number of options for lunch and coffee in the building or a mere stones throw away. Buying your own coffee can prove difficult as the CEO and COO are always buying coffee for the team, just one of many company perks.

Working at Liopa has been and continues to be a challenging yet rewarding experience. Employees are given a lot of flexibility and allowed to independently to work through issues for themselves without being micromanaged. Additionally, the company also has a strong desire to let employees grow by learning new languages, tools and frameworks. This is in order to achieve a company objective whilst simultaneously helping the team gain valuable knowledge and expertise.

Would I recommend an internship at Liopa? Absolutely!



Belfast AI Startup Liopa Raises Seed funding on the Syndicate Room crowdfunding platform

A spin-out from Queen’s University Belfast and the Centre for Secure IT, Liopa is developing lipreading technology to enable visual speech recognition. 

AI-based technology startup Liopa has completed a very successful fund-raising campaign on the SyndicateRoom crowdfunding website. The company raised 2.5 times the target amount in a 4 week period. It received strong backing from a number of angel investors and the Fund Twenty8 EIS fund. 

Founded in 2015, Liopa is commercialising over 10 years of research in the field of Speech and Image processing. The company’s technology can determine speech by analysing the movement of a user’s lips as they speak into a camera. In addition, the technology can be used to prevent “spoofing” and security issues in facial recognition systems. 

Liopa’s primary focus is improving the accuracy of voice driven applications which have risen in popularity. Virtual assistants such as Apple’s Siri and Amazon’s Alexa have brought voice interaction into the mainstream. And now corporations such as Google and Sonos are following their lead. These voice driven systems, however, rely on audio speech recognition (ASR) to determine speech. This means their accuracy deteriorates with the increase of real-world audio noise – for example, in a busy restaurant or outside on a windy day. 

Liopa’s visual speech recognition technology, LipRead, is designed to decipher speech from lip movements. It is therefore agnostic to audio noise. Liopa hopes to augment existing voice driven systems in real-world environments, improving accuracy when background noise is present. The company calls this usage of LipRead “ASR-Assist”. 

Speaking about the funding round, Liopa’s CEO Liam McQuillan said: “This investment will allow us to grow our engineering capability and AI talent.” He continued, “We’ll be able to accelerate the exciting developments we have planned in our roadmap, and protect our valuable IP.”

SyndicateRoom’s co-founder, Tom Britton commented, “It’s no wonder the technology being developed by Liopa is so incredible. The team behind it have spent a combined 50+ man years researching or developing the technology. Their backgrounds range from Senior Academics to C- suite Commercial ranging from startups to the likes of Intel. The applications for their platform are wide ranging, everything from helping law enforcement decipher what’s been said on CCTV footage to giving those who have lost their ability to vocalise a new way to easily communicate. We’re delighted to play a role in such an innovative technology that is applying machine learning and AI for an ultimate good.” 

Launched in September 2013, SyndicateRoom is an online investment platform. It has helped 170+ early-stage UK businesses secure more than £215 million in funding through its investor-led equity crowdfunding model.  



What is lipreading?

Lipreading is a communication technique used by the hard-of-hearing.  Unlike sign language it doesn’t require both parties to be trained in the technique. From an accuracy perspective however, human lipreading is generally poor. Indeed, it requires intense levels of concentration for the lipreader. As such, it is not a favoured communication technique for the hard-of-hearing.  

The challenges

The optimal scenario for the lipreader is a face-to-face engagement with someone they know – ideally whose lip movements are familiar. Lipreading strangers is much more challenging. More often than not, interactions and environments are not ideal. Lipreading multiple speakers in a group is virtually impossible. People do not turn to face the lipreader, or speak one at a time in an orderly fashion!  Additionally, different mouth shapes, facial hair, rate of speech and distance from speaker all create problems for even the best trained lipreaders.  

Low accuracy

As a result, the accuracy of human lipreading is unfortunately very low.  Most lipreaders actually try to pick out keywords and ‘fill in the blanks’ given the context of the conversation.  In fact, lipreading is said to be 80% guesswork! Studies have shown that the best performing lipreaders struggle to achieve greater than 50% accuracy in ideal conditions.  These accuracy levels tail off markedly in longer tests, as the lipreader tires. Amongst the standard hearing population lipreading accuracy is about 10% – 1 word in 10.    

Why being lipreader “friendly” is important

Effective communication techniques allow the hard of hearing to stay connected to the world around them. They build confidence and develop social and communication skills. Not being able to understand what is being said can be frustrating and lead to a sense of isolation. Communication is part of human contact and is vital for mental well-being.  It is important that the hearing population are aware of the difficulties the hard of hearing have in using techniques such as lipreading. They should ensure, where possible, that they communicate in such a way that is best for the lipreader.  

You can read some suggestions on how to more lipreader “friendly”:  Healthy Hearing: Resolve to Improve Your Lipreading Skills and this Deaf Expressions blog: Five Tips to make Lipreading Easier. 

What we’re doing to help

Liopa is developing an automated lipreading platform – LipRead. We use videos of people speaking to train our AI-based LipRead platform to recognise speech from lip movements. LipRead is initially targeted at constrained vocabularies. This includes the command set for an in-vehicle voice-activation unit, for instance. It can be used to  improve the accuracy of other current speech recognition technologies, especially which analyse audio and are susceptible to background noise. 

Over time, LipRead will support larger vocabularies and more languages with increasing accuracy. With these additional capabilities, we plan to provide a smartphone application that can assist the hard-of-hearing in the difficult task of lipreading.

Contact us to find out more! 

 

 



Investment secured

Liopa to bring visual speech-recognition platform to market. Belfast’s Liopa has raised $1m in funding led by Techstart NI and QUBIS to commercialise its LipRead platform for a global audience. Read the article in full here.

It doesn’t matter if you’re from Bangor in Northern Ireland or Wales, or Perth in Scotland or Australia: when it comes to lip-reading technology, accents don’t matter – only the words you say.
The end-goal for an innovative start-up based in Belfast is to have a real-time lip reader than can recognise very large vocabularies across multiple languages. That’s a little way off but there are many nearer term use-cases that Liopa can support in the interim.

Read the PwC disruptive technologies blog in full here.