5 min
Robust real-time hand perception is one of the most challenging and complex computer vision projects right now. As human hands are flexible as well as highly articulated, they lack high contrast patterns. Therefore, one of the important reasons to develop hand gesture recognition is to establish a robust interaction between humans and machines.
This week, Google announced the release of a new approach to hand perception — an ML Pipeline for hand-tracking and gesture recognition. Earlier in June this year, Google previewed this new technique in the Computer Vision and Pattern Recognition conference 2019.
Behind the Architecture
The hand perception functionality is implemented in MediaPipe which is an open-source cross-platform framework for building pipelines in order to process perceptual data of different modalities, like video and audio. It provides high-fidelity hand and finger tracking by employing ML to infer 21 3D keypoints of a hand from just one single frame.
The hand tracking solution utilises a machine learning pipeline which is constituted of three models as mentioned below
- BlazePalm: BlazePalm is a single-shot detector model that operates on the full image and returns an oriented hand bounding box. To achieve this, the model follows some specific strategies such as training a palm detector instead of a hand detector, using an encoder-decoder feature extractor for bigger scene context-awareness even for small objects and lastly, minimising the focal loss during training. BlazePalm is mainly used to detect the occluded and self-occluded hands. So far, this method has achieved an average precision of 95.7% in palm detection.
- Hand Landmark Model: This model operates on the cropped image region defined by the palm detector and performs precise keypoint localisation of 21 3D hand-knuckle coordinates inside the detected hand regions via regression. In order to get robust hand poses, the researchers manually annotated approximately 30,000 real-world images with 21 3D co-ordinates. A high-quality synthetic hand model is also rendered over various backgrounds and map it to the corresponding 3D coordinates.
- Gesture Recogniser: This model classifies the previously computed keypoint configuration into a discrete set of gestures. The researchers used accumulated angles of joints to determine the state of the fingers and it is then mapped to a set of pre-defined gestures. This technique helped the researchers to estimate basic static gestures with reasonable quality.
How It Is Different
This approach is different from the existing state-of-the-art approaches. The existing approaches rely primarily on powerful desktop environments for inference, while the new approach not only achieves real-time performance on a mobile phone but also has the capability to scale more than one hand.
Application of This New Approach
The hand perception approach can be used in various cases and few of them are mentioned below- It can be enabled to the overlay of digital content and information on top of the physical world in augmented reality.
- Will help the differently-abled with sign language
- It can be applied in VR systems to manipulate virtual objects.
- It can also be utilised to control intelligent robots.
- Why AI is Eating Google Search’s Ad Revenue While Boosting Meta’s
- Anthropic Makes $2.08 in Revenue for Every $1 Spent on Compute — Here’s the Full IPO Story
- OpenAI Says AGI is Here. But Not Everyone is Buying It
- Why Oracle is Bringing Post-Quantum Security to Java
- What Really Happens When You Turn Your Selfie Into a 1980s AI Pic?
- AI Agents Are Making Data Security A Bigger Enterprise Problem