Sign Language Translation: Why It Is Hard to Translate Hand Gestures to Spoken Words

Why It Is Hard to Translate Hand Gestures to Spoken Words

The sign language translation technology is on its way.

According to the Ethnologue guide, there are 7,151 spoken languages in the world. While the number might be surprising, did you know that there are also more than 300 sign languages worldwide, with American sign language (ASL) as the most widely used one?

As of 2021, 430 million people globally are suffering from some form of hearing loss, and that number is likely to double by 2050. The enormous number suggests that more people may need to learn sign language to communicate with deaf people in the near future. Some may even feel discouraged from learning the skill, since mastery of sign language takes time and extensive practice. This is where sign language translation comes into play. However, despite strenuous efforts made in developing sign language translation technology, it remains challenging to perfect the task. Read on to learn more about the obstacles facing sign language translation and how they are addressed by the newly-invented tech solutions.

Why are sign languages difficult to translate?

There is no universal sign language

As mentioned earlier, there are over 300 sign languages worldwide. From a linguistic perspective, a sign language is similar to a distinct spoken language in that they’re not mutually intelligible among themselves. To put it simply, an ASL user cannot understand British sign language (BSL) and vice versa because sign languages are developed based on regions’ dialect and culture.

Since sign language translation remains relatively experimental, there hasn’t been any system or device that translates ASL to BSL or allows users to translate sign language to any foreign language. Researchers from around the world, however, are developing systems that translate their regions’ sign language.

For instance, researchers from Complex Software Lab (a software engineering lab at University College Dublin, Ireland) have put together a new artificial intelligence (AI) -based technology that can translate Irish sign language (ISL) into spoken words. The team behind the AI understands that about 70% of communication in sign language comes from facial expressions. So, they also leverage computer vision and deep learning to capture facial expressions for more accurate translation. We will delve more into the other elements of sign language in the later part of this article.

Besides ISL translation, researchers from Australian video consultation services company Coviu have also developed a web application that translates Auslan (the national sign language of Australia) alphabets using machine learning. The team started off with building their own Auslan alphabet image dataset containing photos of different people signing the alphabets. Those photos served as data “teaching” the computer to match the signs with the corresponding alphabets. When signers sign in front of the computer webcam, machine learning helps the computer translate the Auslan alphabets in real-time.

Understanding individual signs is not enough

Except when we are replying or chit-chatting with someone (e.g., “yes,” “no,” “sure,” “how’s it,” etc.), we normally make complete sentences when communicating. The same also applies to sign language, and hence a sign language translator or software should be able to translate not only single, individual signs but also complete sentences.

Back in 2016, two sophomores from the University of Washington won the Lemelson-MIT Student Prize for their invention of a pair of sign language translation gloves named SignAloud. Inside the gloves, there are sensors measuring hand position and movement. The sensor data is then sent to a nearby computer via Bluetooth that matches the hand movements with corresponding gestures. If the data matches a gesture in the system, the translated signs will be read aloud through speakers. While the idea of translating sign language via gloves is undoubtedly groundbreaking, translating only words or phrases is far from enough.

Three years later, in 2019, researchers from Michigan State University rolled out a deep learning-backed sensory device called DeepASL, which can translate complete ASL sentences without requiring users to stop after each sign. To boost system performance, the team collected approximately 7,000 samples covering 56 common ASL words and 100 sentences to train the algorithm. The DeepASL device is powered by leap motion (a small movement tracking computer interface) which maps and tracks human hand movements through its cameras.

Sign languages involve more than just gestures

Since we see signers mainly using their hands while signing, we may think sign language is only about hand gestures. The truth is, sign language comprises three elements: hand gestures, body movements and facial expressions. All these elements help signers express meaning, such as raising the eyebrows to turn a phrase into a question. The importance of non-verbal cues in sign language communication suggests that researchers must consider more than hand gestures when developing their applications; otherwise, the translation can never be accurate.

SignAll, a Hungary-based startup that strives to enable spontaneous communication between deaf people and the hearing, has put together a device that translates sign language using computer vision and natural language processing. To unpack some technical terms, computer vision refers to enabling computers to process and analyze information (e.g. videos and pictures) through using high-performance camera and image processing software. As for natural language processing, it is the use of technology, like AI, to understand text and spoken words as humans do. 

For translation, signers need to wear gloves and sign in front of three cameras that capture their hand gestures, facial expressions and body movements. This data is then sent and processed by a central computer that transcribes complete ASL sentences for the hearing. On the other hand, the computer can also transcribe speech for deaf people using natural language processing.

Speech to signs—sign language translation goes both ways

Researchers from the Complex Software Lab at University College Dublin also noticed that sign language communication should be for both deaf people and those who can hear. Therefore, their technology focuses on more than just transcribing sign language into spoken language but also the other way around. When the software is translating speech to ISL, the signers will see an avatar signing on their screen. If signers want to translate signing to speech, all they need to do is sign in front of a Microsoft Kinect (a motion-sensing device) and let the system transcribe for the hearing.

While many people welcome and advocate sign language translation technology, some remain skeptical about the invention because sign language is too sophisticated to be translated accurately. We should, however, appreciate the efforts and time researchers have invested in building the technology and their attempts to enable effective communication between deaf people and the hearing.

Also read:

Header image courtesy of Freepik


Share on facebook
Share on twitter
Share on linkedin
Share on email


Is Using AI for Academic Writing Cheating?

Is Using AI for Academic Writing Cheating?

From films written and directed by ChatGPT to Kindle authors writing books with artificial intelligence (AI), we are seeing more and more reportage on AI writing tools these days. While writing with an AI may seem harmless and possibly helps improve work efficiency, the use of these tools in the sphere of academic writing has generated serious concern. Teachers are worried that the use of AI writing tools is equivalent to plagiarism or cheating.

7 Creative Ways to Reward Employees without a Salary Raise

7 Creative Ways to Reward Employees without a Salary Raise

Giving your employees a raise is a significant way to show your employees that you value their work and compensate them appropriately for their contributions. Failing to acknowledge the work of your employees can be ‌costly for startups. A OnePoll survey of 2,000 workers found that almost half of ‌American workers (46%) have quit a job due to feeling unrecognized

The Pros and Cons of Human Cloning

The Pros and Cons of Human Cloning

Clones, human or otherwise, are genetically identical copies of a living organism produced artificially in a laboratory. Over the years, many animals have been cloned. Starting from the cloning of a tadpole in 1952 to the cloning of a sheep named Dolly from adult mammal cells in 1996, cloning has come a long way.

Reform of Hong Kong's Offshore Tax System

Reform of Hong Kong’s Offshore Tax System

Further to the introduction of the proposed Hong Kong’s new Foreign-Source Income Exemption (FSIE) regime as discussed in our article A Challenge to Hong Kong’s Long-Established Offshore Tax System in October 2022, the relevant legislation has been enacted and became effective on January 1, 2023.

Mark Cuban’s Tips on How to Get Rich

Mark Cuban’s Tips on How to Get Rich

A popular adage suggests, money makes the world go round. People are always looking to do better financially so that their lives can be more secure. To make that happen, some seek out the advice of millionaires—the ones who have been there, done that.

What India's EdTech Industry Must Prioritize in 2023?

What India’s EdTech Industry Must Prioritize in 2023?

2022 was a tough year for the majority of education technology (EdTech) in India, with the “Tracxn Geo Annual Report: India Tech 2022” reporting a 39 per cent decrease in funding during the January-November period compared to 2021. To overcome this, many businesses, including industry-leading players like BYJU’S, Unacademy and Vedantu, had to resort to job cuts.