robot hand

Wrist-mounted tracker accurately models 3D hand movements

Image credit: Dreamstime

A wrist-mounted device that continuously tracks the entire human hand in 3D has been developed for use in sign language translation, virtual reality, mobile health, human-robot interaction and other areas.

Developed by researchers at Cornell University, the FingerTrak bracelet can sense and translate into 3D the many positions of the human hand, including 20 finger-joint positions, using three or four miniature, low-resolution thermal cameras that read contours on the wrist.

“This was a major discovery by our team - that by looking at your wrist contours, the technology could reconstruct in 3D, with keen accuracy, where your fingers are,” said researcher Cheng Zhang. “It’s the first system to reconstruct your full hand posture based on the contours of the wrist.”

Previous wrist-mounted cameras have been considered too bulky and obtrusive for everyday use, and most could reconstruct only a few discrete hand gestures.

FingerTrak’s breakthrough is a lightweight bracelet that allows greater freedom of movement. Instead of using cameras to directly capture the position of the fingers, the focus of most prior research, FingerTrak uses a combination of thermal imaging and machine learning to virtually reconstruct the hand.

The bracelet’s four miniature thermal cameras - each about the size of a pea - snap multiple “silhouette” images to form an outline of the hand.

A deep neural network then stitches these silhouette images together and reconstructs the virtual hand in 3D. Through this method, Zhang and his fellow researchers were able to capture the entire hand pose, even when the hand is holding an object.

While the technology has a wide range of possible uses, Zhang said the most promising is its potential application in sign language translation.

“Current sign language translation technology requires the user to either wear a glove or have a camera in the environment, both of which are cumbersome,” he said. “This could really push the current technology into new areas.”

FingerTrak could also have an impact on health care applications, specifically in monitoring disorders that affect fine-motor skills, said Yin Li who contributed to the software development.

“How we move our hands and fingers often tells about our health condition,” he said. “A device like this might be used to better understand how the elderly use their hands in daily life, helping to detect early signs of diseases like Parkinson’s and Alzheimer’s.”

Sign up to the E&T News e-mail to get great stories like this delivered to your inbox every day.

Recent articles