Vision-Driven framework For Interpreting Human Gestures into Natural Language
DOI:
https://doi.org/10.68104/ijasit.v1.i3.64Keywords:
Sign Language Recognition, Computer Vision; Gesture Recognition, Deep Learning, Human–Computer Interaction, Assistive Technology, AccessibilityAbstract
Communication limitations between sign language and the hearing community have remained an impediment for inclusion and access of deaf and hard-of-hearing persons into vital and basic life requirements. Towards overcoming this major challenge, this paper advances a vision-based framework for interpreting and processing natural language from human gestures using computer vision and machine learning approaches. The proposed framework, chiefly, employs camera devices in capturing real-time gestures and understanding hand, body, and facial movements. Efficient techniques have been incorporated, such as detecting gestures and landmarks using OpenCV and MediaPipe, and further analyzing and processing gestures with landmark detection techniques and machine learning-based classification using TensorFlow. Experimental results indicate the accuracy and reliability of a proposed system for vision-based natural language translation. Thus, the proposed vision-based framework can be a potential and inclusive technology in addressing the challenges and limitations in natural language understanding and accessibility, towards promoting inclusive and equitable societies by ensuring accessibility for all, thus promoting and sustaining inclusive growth and social stability.
Downloads
References
A. Al-Khazraji, M. Al-Ali, and S. Al-Maadeed, “JUMLA-QSL-22: A Novel Qatari Sign Language Continuous Dataset,” in Proceedings of the 2023 International Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2023, pp. 1–6.
Vision-Base Y. Zhou, H. Shi, and L. Wang, “An Efficient Two-Stream Network for Isolated Sign Language Recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2022, pp. 1234–1242. d Continuous Sign Language Spotting Using HMM (2022)
A. K. Talukdar and M. K. Bhuyan, “Vision-Based Continuous Sign Language Spotting Using Gaussian Hidden Markov Model,” IEEE Sensors Letters, vol. 6, no.7, pp. 1–4, July 2022.
R. Singh, P. Kumar, and S. Verma, “Smart IoT System for Indian Sign Language Translator,” in Proceedings of the International Conference on Internet of Things and Smart Systems (IoTSS), 2024, pp. 1–6
M. Ahmed, S. Khan, and R. Ali, “Sign Language Recognition and Translation Systems for Enhanced Communication,” in Proc. Int. Conf. Artificial Intelligence and Communication Technologies (AICT), 2023, pp. 1–6.
C. Liang, C. Wu, and S. Shimamoto, “Research on Sensor-Integrated Smart Gloves for Sign Language Recognition,” in Proc. Asia-Pacific Conf. Communications (APCC), 2025, pp. 1–5.
M. S. Amin, S. T. H. Rizvi, and M. M. Hossain, “A Comparative Review on Applications of Different Sensors for Sign Language Recognition,” J. Imaging, vol. 8, no. 4, art. 98, 2022.
N. C. Camgoz, S. Hadfield, O. Koller, and R. Bowden, “Neural Sign Language Translation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 7784–7793.
K. Simonyan and A. Zisserman, “Two-Stream Convolutional Networks for Action Recognition in Videos,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2014, pp. 568–576.
A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks,” in Proc. Int. Conf. Mach. Learn. (ICML), 2006, pp. 369–376.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Karthikeyan V, Hemlathadhevi A, Jackulin C, Dhanush Abhishek J, Balamurugan P, Jabasheela L
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
