Lero researchers at UCD announce programme with Microsoft to enable future communications between deaf and non-deaf people

Lero, the SFI Irish Software Research Centre and University College Dublin (UCD) have announced a research initiative with Microsoft Skype designed to enable future communications between deaf and non-deaf people.

The research project, which will develop a virtual Irish sign language interpreter, is backed by Science Foundation Ireland in collaboration with Microsoft. It incorporates various Microsoft technologies that enable interaction without the need for a human interpreter or sign language knowledge by the non-deaf individual.

“This is an exciting research programme which could transform the ability of deaf people in the future to communicate across education, work, social and family situations with non-deaf individuals and groups who do not know sign language,” commented Dr Anthony Ventresque, Lero researcher and Director of the UCD Complex Software Lab who is leading the programme with PhD students Elayne Ruane and Thomas Laurent.

“Our business is all about communications which is why we began to think that deafness should not be a barrier,” commented Ross Smith, Director of Engineering at the Microsoft Skype Division. “The goal of the three to five year programme is to develop a solution that could be made available to deaf people worldwide as part of our Skype4Good global initiative.”

Lero researchers at UCD have already developed a prototype using Microsoft tools including HoloLens mixed reality smart glasses, LUIS.AI language understanding, Azure cognitive services and Xbox depth camera technologies. An avatar translates the sign language of the deaf person into speech and vice versa.

Dr Ventresque commented, “To date, we’ve developed a sign language interpreter that focuses on hand gestures; the next step is to make the prototype realistic and useful to the end user by integrating facial expression recognition and generation into the interpreter, which is a critical feature missing from current offerings in this area. We feel we also have a head start in this space as we’ve developed an innovative solution to emotion detection in the context of another joint Lero Skype project.”

Dr Darrin Morrissey, Director of Programmes and Investments for Science Foundation Ireland, welcomed the initiative. “This is an excellent example of some of the world-leading research that is being conducted in Ireland across the SFI Research Centre network with our industrial partners. It has huge potential to contribute to the common good globally and positively impact the lives of the deaf community.”

Commenting on the development, Philip Power, who is a deaf law student at UCD said, “My reaction when I first used the prototype was ‘Wow! Absolutely fantastic!’ I think it would be a great benefit for myself, for students, for everyday life. It is s a very good idea.”

Leah Ennis McLoughlin, a deaf UCD science student added, “We’re all part of the same world so it would be fantastic if this smashes linguistic barriers which currently exist for deaf people across work and social interaction.”