GDTII: Gesture Driven Text Input for Immersive Interfaces
Download| Volume 5 Issue 2 2024 | |
|---|---|
| Author(s): |
Nizamuddin Maitlo* Dept. Engenharia De Computadores E Telematica, University of Aveiro, Portugal , nizamuddin@ua.pt Samia Karim Bhutto Institutue of Computer Science, Shah Abdul Latif University, Khairpur,Pakistan , Bhuttosamia775@gmail.com Muntazir Mahdi Institutue of Computer Science, Shah Abdul Latif University, Khairpur, Pakistan , mahdibaidani@gmail.com Samar Abbas Mangi Institutue of Computer Science, Shah Abdul Latif University, Khairpur, Pakistan , mangisamar@gmail.com |
| Abstract | Hand gesture-based text input can be used as an easy and more natural way of human-computer interaction, especially in AR (Augmented Reality) and VR (Virtual Reality) immersive environment. We present GDTII (Gesture-Driven Text Input for Immersive Interfaces), a novel approach that utilizes both static and dynamic hand gestures to facilitate an efficient and accurate text entry. The proposed system works in three essential steps: 1) A hand is recognized from the original RGB video 2) A hand segmentation model based on adaptive background subtraction is used and 3) Trajectory classification for gesture recognition is done using deep learning-based models. Static gestures as well as dynamic hand movements are identified by a CNN and optimized convex hull trajectory-mapping algorithm. Then, the extracted trajectories are processed so that the network reconstructs the handwritten character, which goes through a character recognition network and, consequently, text generation. The proposed system is thoroughly tested on real-world datasets, obtaining higher classification accuracy as well as proving to be more resilient against the variety of lighting conditions and has a better real-time performance compared to traditional gesture recognition approaches. The results show that GDTII is a practical and reliable solution for gesture-driven text input and enables effortless interaction in AR/VR environments and other scenarios that need non-contact text entry. |
| Keywords | augmented reality, virtual reality, hand segmentation, hand gesture recognition, text-based input, immersive interfaces, CNN |
| Year | 2024 |
| Volume | 5 |
| Issue | 2 |
| Type | Research paper, manuscript, article |
| Recognized by | Higher Education Commission of Pakistan, HEC | Category | Journal Name | ILMA Journal of Technology & Software Management | Publisher Name | ILMA University | Jel Classification | -- | DOI | - | ISSN no (E, Electronic) | 2790-590X | ISSN no (P, Print) | 2709-2240 | Country | Pakistan | City | Karachi | Institution Type | University | Journal Type | Open Access | Manuscript Processing | Blind Peer Reviewed | Format | Paper Link | https://ijtsm.ilmauniversity.edu.pk/arc/Vol5/i2/pdf4.pdf | Page | 35-40 |