A study of the Sign language identifying using the OpenCV, MediaPipe, and Scikit-learn modules
DOI:
https://doi.org/10.29070/nfke6755Keywords:
Sign language, MediaPipe hand landmark detection, Python, Audio, OpenCv libraryAbstract
Sign language is manual communication commonly used by people who are hard of speaking and hearing. These languages use the visual-manual modality to convey meaning, instead of spoken words. Sign languages are expressed through manual articulation using hand gestures, facial expressions, and body language to describe the intended message as well as some non-manual markers. This paper proposes a novel approach to interpreting sign language using the camera of a phone or a laptop breaking the communication barrier between a mute person and a person who does not know sign language. In this approach, the model has been trained to identify some signs using the OpenCV, MediaPipe, and Scikit-learn modules. The hand landmarks from the image data set were extracted using the media pipe module to train the model. The model can identify signs and once the sign has been identified it can play the corresponding sign in the form of audio.
References
Wikipedia contributors. (2024, July 16). Sign language - Wikipedia. https://en.wikipedia.org/wiki/Sign_language
MediaPipe Solutions guide. (n.d.). Google for Developers. https://ai.google.dev/edge/mediapipe/solutions/guide
Hand landmarks detection guide. (n.d.). Google for Developers. https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker
Boesch, G. (2024, June 21). MediaPipe: Google’s Open Source Framework (2024 Guide). viso.ai. https://viso.ai/computer-vision/mediapipe/#:~:text=for%20your%20 organization.-,What%20is%20MediaPipe%3F,currently%20in%20alpha%20at%20v0
OpenCV. (2024, July 12). OpenCV - Open Computer Vision Library. https://opencv.org/
GeeksforGeeks. (2024, April 15). What is OpenCV Library? GeeksforGeeks. https://www.geeksforgeeks.org/opencv-overview/
GeeksforGeeks. (2022, April 18). Python Text to Speech by using pyttsx3. GeeksforGeeks. https://www.geeksforgeeks.org/python-text-to-speech-by-using-pyttsx3/
pyttsx 3. (2020, July 6). PyPI. https://pypi.org/project/pyttsx3/
Grzejszczak, T., Kawulok, M., & Galuszka, A. (2016). Hand landmarks detection and localization in color images. Multimedia Tools and Applications, 75, 16363-16387.
Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C. L., & Grundmann, M. (2020). Mediapipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006.10214.
Sánchez-Brizuela, G., Cisnal, A., de la Fuente-López, E., Fraile, J. C., & Pérez-Turiel, J. (2023). Lightweight real-time hand segmentation leveraging MediaPipe landmark detection. Virtual Reality, 27(4), 3125-3132.
Luna-Jiménez, C., Gil-Martín, M., Kleinlein, R., San-Segundo, R., & Fernández-Martínez, F. (2023, October). Interpreting sign language recognition using transformers and MediaPipe landmarks. In Proceedings of the 25th International Conference on Multimodal Interaction (pp. 373-377).
Priya, K., & Sandesh, B. J. (2023, March). Hand Landmark Distance Based Sign Language Recognition using MediaPipe. In 2023 International Conference on Emerging Smart Computing and Informatics (ESCI) (pp. 1-7). IEEE.
Bora, J., Dehingia, S., Boruah, A., Chetia, A. A., & Gogoi, D. (2023). Real-time assamese sign language recognition using mediapipe and deep learning. Procedia Computer Science, 218, 1384-1393.
Samaan, G. H., Wadie, A. R., Attia, A. K., Asaad, A. M., Kamel, A. E., Slim, S. O., ... & Cho, Y. I. (2022). Mediapipe’s landmarks with rnn for dynamic sign language recognition. Electronics, 11(19), 3228.
Luna-Jiménez, C., Gil-Martín, M., Kleinlein, R., San-Segundo, R., & Fernández-Martínez, F. (2023, October). Interpreting sign language recognition using transformers and MediaPipe landmarks. In Proceedings of the 25th International Conference on Multimodal Interaction (pp. 373-377).
Remiro, M. Á., Gil-Martín, M., & San-Segundo, R. (2023). Improving Hand Pose Recognition Using Localization and Zoom Normalizations over MediaPipe Landmarks. Engineering Proceedings, 58(1), 69.
Priya, K., & Sandesh, B. J. (2023, March). Hand Landmark Distance Based Sign Language Recognition using MediaPipe. In 2023 International Conference on Emerging Smart Computing and Informatics (ESCI) (pp. 1-7). IEEE.
Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C. L., & Grundmann, M. (2020). Mediapipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006.10214.
Subashini, V., Someshwaran, B., Sowmya, S., & Kumar, S. A. (2024, March). Sign Language Translation Using Image Processing to Audio Conversion. In 2024 Third International Conference on Intelligent Techniques in Control, Optimization and Signal Processing (INCOS) (pp. 1-6). IEEE.
This work is licensed under a 