GDTII: Gesture Driven Text Input for Immersive Interfaces

Download

Volume 5 Issue 2 2024

Author(s):

Nizamuddin Maitlo* Dept. Engenharia De Computadores E Telematica, University of Aveiro, Portugal , nizamuddin@ua.pt

Samia Karim Bhutto Institutue of Computer Science, Shah Abdul Latif University, Khairpur,Pakistan , Bhuttosamia775@gmail.com

Muntazir Mahdi Institutue of Computer Science, Shah Abdul Latif University, Khairpur, Pakistan , mahdibaidani@gmail.com

Samar Abbas Mangi Institutue of Computer Science, Shah Abdul Latif University, Khairpur, Pakistan , mangisamar@gmail.com

Abstract Hand gesture-based text input can be used as an easy and more natural way of human-computer interaction, especially in AR (Augmented Reality) and VR (Virtual Reality) immersive environment. We present GDTII (Gesture-Driven Text Input for Immersive Interfaces), a novel approach that utilizes both static and dynamic hand gestures to facilitate an efficient and accurate text entry. The proposed system works in three essential steps: 1) A hand is recognized from the original RGB video 2) A hand segmentation model based on adaptive background subtraction is used and 3) Trajectory classification for gesture recognition is done using deep learning-based models. Static gestures as well as dynamic hand movements are identified by a CNN and optimized convex hull trajectory-mapping algorithm. Then, the extracted trajectories are processed so that the network reconstructs the handwritten character, which goes through a character recognition network and, consequently, text generation. The proposed system is thoroughly tested on real-world datasets, obtaining higher classification accuracy as well as proving to be more resilient against the variety of lighting conditions and has a better real-time performance compared to traditional gesture recognition approaches. The results show that GDTII is a practical and reliable solution for gesture-driven text input and enables effortless interaction in AR/VR environments and other scenarios that need non-contact text entry.
Keywords augmented reality, virtual reality, hand segmentation, hand gesture recognition, text-based input, immersive interfaces, CNN
Year 2024
Volume 5
Issue 2
Type Research paper, manuscript, article
Recognized by Higher Education Commission of Pakistan, HEC
Category
Journal Name ILMA Journal of Technology & Software Management
Publisher Name ILMA University
Jel Classification --
DOI -
ISSN no (E, Electronic) 2790-590X
ISSN no (P, Print) 2709-2240
Country Pakistan
City Karachi
Institution Type University
Journal Type Open Access
Manuscript Processing Blind Peer Reviewed
Format PDF
Paper Link https://ijtsm.ilmauniversity.edu.pk/arc/Vol5/i2/pdf4.pdf
Page 35-40