Defense of the dissertation of Amangeldi Nurlan for the degree of Doctor of Philosophy (PhD) in the specialty «6D060200 - Информатика»
Defense of the dissertation of Amangeldi Nurlan for the degree of Doctor of Philosophy (PhD) in the specialty «6D060200 - Информатика»
L.N. Gumilyov Eurasian National University, a dissertation defense for the degree of Doctor of Philosophy (PhD) by Amangeldi Nurlan on the topic «The methods and algorithms of Kazakh sign language recognition» in the field of «6D060200 – Информатика».
The dissertation was carried out at the «Artificial Intelligence Technologies» of L.N. Gumilyov Eurasian National University.
The language of defense is kazakh
Reviewers:
Еримбетова Айгерим Сембековна - Master
Аканова Акерке Сапаровна - Doctor of Philosophy (PhD), senior lecturer
Temporary members of the Dissertation Committee:
Кожирбаев Жанибек Мамбеткаримович - Doctor of Philosophy (PhD)
Маткаримов Бахыт Турганбаевич - Doctor of Science, Professor, Teacher-researcher
Дауренбеков Куаныш Койшыгулович - Deputy Head
Мансурова Мадина Есимхановна - Candidate of Science, Assistant Professor, Head of the cafedra
Гильмуллин Ринат Абрекович - Candidate of Science, Director of the Institute
Исраилова Нелла - Candidate of Science, Assistant Professor
Academic Advisors:
Kudubaeva Saule Alzhanovna - candidate of technical sciences, senior lecturer of the department "Artificial Intelligence Technologies", faculty of Information Technologies, L.N. Gumilyov Eurasian National University (Astana)
Krak Yuri Vasilyevich - Corresponding Member of NAS of Ukraine (Computer Science), doctor of physical and mathematical sciences, professor, Head of the Department of "Theoretical Cybernetics", faculty of Informatics and Cybernetics, Taras Shevchenko National University of Kyiv (Kyiv, Ukraine)
The defense will take place on January 15, 2024, at 03:00 PM in the Dissertation Council for the training direction «8D061 – %!s(*string=0xc0034723f0)» in the specialty «6D060200 – Информатика» of L.N. Gumilyov Eurasian National University. The defense meeting is planned to be held online.
Link: https://teams.microsoft.com/l/meetup-join/19%3ab4OWgn9w0BU7i4VgTMspGg4LykYCo4gkpJ0M1aiEJ2M1%40thread.tacv2/1702010534128?context=%7b%22Tid%22%3a%225a2fd781-9261-485a-af2b-6171d0efab73%22%2c%22Oid%22%3a%22309bb2ca-1ef1-4604-912b-0411910d9d0b%22%7d
Address: 010000, Астана қаласы, Сәтпаев көшесі, 2, оқу-әкімшілік ғимараты, №302 ауд.
Abstract (English): SUMMARY of the PhD Thesis on the Specialty «6D060200 – Computer Science» Nurzada Amangeldi «The methods and algorithms of Kazakh sign language recognition» Research relevance: In recent decades, there has been a growing emphasis on developing technologies aimed at eliminating communication and interaction barriers for individuals with disabilities. In particular, inclusive technologies for those with hearing and speech impairments are gaining significant importance in the modern world. These technologies play a crucial role in integrating this group into the socio-cultural and educational processes of society. Artificial intelligence, leveraging computer vision capabilities, is emerging as a primary tool for individuals with hearing and speech impairments. It enhances their interaction and communication with the surrounding world, ultimately improving their quality of life and education. The research outcomes have a substantial impact on economic development and public relations as a technological tool in an inclusive environment. The establishment of intelligent systems for Kazakh Sign Language recognition and the expansion of digital services in the Kazakh language, including e-government and state services, play pivotal roles in this advancement. Intelligent systems of automatic sign language translation based on datasets of Kazakh Sign Language, machine learning algorithms, and computer vision can serve as a scientific foundation for developing multimodal corpora of Kazakh Sign Language. These advancements stand to assist people with hearing impairments in achieving seamless interaction with society. Dissertation objective: The development of methods and algorithms for recognizing Kazakh Sign Language using modern computer vision technologies, aimed at providing an accurate, efficient, and reliable means of communication for people with hearing and speech impairments in Kazakhstan. Research tasks: − Theoretical analysis of methods and algorithms for sign language recognition based on computer vision, identifying the most effective methods; − Development of a method for creating datasets for Kazakh Sign Language, including the creation of the datasets themselves; − Development of an adaptive multi-class SVM classifier for recognizing the alphabet of Kazakh Sign Language, and building a model based on this classifier; − Development of algorithms for variable recognition of sign language using deep learning, and constructing models based on these algorithms; − Creation of algorithms for continuous recognition of words in sign language using deep learning, and building a model based on these algorithms; − Integration of the developed models into real systems and their testing. Research methods: Theoretical analysis: An extensive study and analysis of scientific literature, works, and research in the field of computer vision were conducted. This analysis encompassed the examination of the current state of knowledge, as well as theories and models related to sign language recognition; Comparative analysis: A comparison of Kazakh Sign Language with other sign languages, such as Russian, English, and Turkish, was carried out in terms of presentation form. The aim was to identify unique demonstration structures characteristic of Kazakh Sign Language; Experimental research: Development and application of experimental data collection methods based on computer vision techniques, including deep learning, were carried out for accurate motion recognition; Programming and modeling: Software and computer vision models were developed to implement various recognition algorithms, including SVM, LSTM1024, and 2DCNN; Integration and application: The developed methods and models were integrated and tested to demonstrate their practical application in real-world systems. These research methods provided a comprehensive approach to studying and developing effective solutions for recognizing Kazakh Sign Language, supported by both theoretical and practical aspects. In the dissertation titled «The methods and algorithms of Kazakh sign language recognition» the following key propositions, contributing new knowledge to the field of sign language recognition, have been formulated and proven: Uniqueness of kazakh sign language: The hypothesis about the unique characteristics of Kazakh Sign Language, distinguishing it from Russian, English, and Turkish sign languages in terms of demonstration form, has been proven. This expands the understanding of the diversity of sign languages and underscores the importance of culturally specific research; Innovative data collection methods: A new method for creating datasets has been developed and tested, utilizing advanced computer vision technologies. This confirms the capability of modern technologies to enhance the process of data collection and processing for the study of sign language; New method for recognizing the dactyl alphabet: A new method for recognizing the dactyl alphabet of Kazakh Sign Language has been proposed and tested, significantly improving the accuracy and efficiency of gesture recognition; Variable real-time dynamic movement recognition: The development of a method for variable real-time dynamic movement recognition based on deep learning represents a significant advancement in artificial intelligence and motion recognition; Continuous recognition of words in sign language: The development of a method for continuous recognition of words in sign language based on deep learning opens up new possibilities for creating more complex and effective sign language recognition systems. These propositions and conclusions are an important contribution to the scientific community, particularly in the study and application of sign language recognition, and open new areas for future research and development. Description of the Main Research Outcomes: Outcome 1. Proof of the uniqueness of kazakh sign language presentation: Demonstrated through comparative analysis with other sign languages, this finding significantly contributes to linguistic science. It broadens the understanding of sign language diversity and reinforces the cultural and linguistic identity of the deaf community in Kazakhstan; Outcome 2. Development of a new data collection method: Marking progress in artificial intelligence, this method for recognizing human body movements and sign language words using computer vision is applicable in various fields, including security, identification, and healthcare; Outcome 3. Creation of an SVM classifier for kazakh sign language alphabet: This development enhances technologies to assist people with hearing impairments and acknowledges Kazakh Sign Language; Outcome 4. Development of a 2DCNN architecture for real-time dynamic sign language word recognition: An important achievement in machine learning, improving interaction between hearing and deaf individuals and facilitating their social integration; Outcome 5. Development of a new LSTM1024 architecture: Enhances the creation of effective systems for sign language communication, strengthening continuous interaction between hearing people and those with hearing and speech limitations; Outcome 6. Application of developed models in real systems: Opens opportunities for new innovations, significantly impacting society and technological progress. Justification of the Novelty and Importance of the Obtained Results: Outcome 1. Uniqueness of kazakh sign language. Novelty: focuses on the lesser-studied kazakh sign language in the context of computer vision and machine learning. Importance: contributes to preserving cultural and linguistic values and ensuring accessibility for the deaf community in kazakhstan; Outcome 2. Innovative data collection methods. Novelty: automates the data collection process in computer vision for sign language recognition tasks. Importance: application in various fields, enhancing interactive systems, monitoring, and security systems; Outcome 3. Adapted SVM classifier. Novelty: adapts the SVM classifier for kazakh sign language. Importance: foundation for recognizing gesture-based sign language and related scientific research; Outcome 4. Variable gesture recognition method. Novelty: development of algorithms using 2DCNN for diverse gesture recognition. Importance: improves accuracy for practical application; Outcome 5. Continuous word recognition. Novelty: development of a method using LSTM1024 for continuous word recognition. Importance: enhances communication between deaf and hearing individuals, a basis for automatic translation systems; Outcome 6. Integration and testing of models. Novelty: translates research into practical application. Importance: improves accessibility of information and services for the deaf and hard of hearing, promoting their social integration. Alignment with scientific development directions or state programs 4. Information, communication, and space technologies . 4.1 artificial intelligence and information technologies. 4.1.3 pattern recognition and image processing; 4.1.5 machine learning Contribution of the candidate in the preparation of each publication. Within the framework of the dissertation titled «The methods and algorithms of Kazakh sign language recognition» three articles were presented: 1. "Sign language recognition method based on palm definition model and multiple classification": this article proposes a method for recognizing the letters of kazakh sign language, achieving high accuracy in recognizing both the alphabet and numbers; 2. "A real-time dynamic gesture variability recognition method based on convolutional neural networks": this research focuses on recognizing words using convolutional neural networks, demonstrating improved accuracy in dynamic gestures applicable to various sign languages; 3. "Continuous sign language recognition and translating it into intonation-colored speech": dedicated to the continuous recognition of sign language and its translation into speech with intonation coloring. The model showed high accuracy in cross-validation, emphasizing the importance of the connection between gestures and intonation in speech; The dissertation author is either the first or corresponding author in all of these articles, confirming their direct and comprehensive involvement in the conducted research. These publications thoroughly disclose the essence of the research, are logically interconnected, and cover various aspects and methods of recognizing kazakh sign language, making a valuable contribution to this field of scientific research. Publications. Publications in Journals Listed in the Committee on Control in Education and Science of the Ministry of Education and Science of the Republic of Kazakhstan: 1. Nurzada Amangeldy, Saule Kudubayeva. "Identification of connected areas and correlational methods in phrase recognition in Kazakh Sign Language". Kazakh National Technical University Reporter No. 5 2020, pp. 172–177. ISSN: 1680-9211. 2. Nurzada Amangeldy, Saule Kudubayeva. "Review of the subject area for Kazakh Sign Language recognition problem". Kazakh National Technical University Reporter No. 5 2020, pp. 177–182. Publications in Journals and Conferences Indexed in Web of Science and Scopus: 3. Nurzada Amangeldy, Saule Kudubayeva, Akmaral Kassymova, Ardak Karipzhanova, Bibigul Razakhova, Serikbay Kuralov. Sign Language Recognition Method Based on Palm Definition Model and Multiple Classification. Sensors. Издатель: Multidisciplinary Digital Publishing Institute. ISSN 14248220. DOI: 10.3390/s22176621. Sensors 2022, 22(17), 6621. (Article) 4. Amangeldy Nurzada, Ukenova Aru, Bekmanova Gulmira, Razakhova Bibigul, Milosz Marek, Kudubayeva Saule. Continuous Sign Language Recognition and Its Translation into Intonation-Colored Speech. Sensors. Издатель: Multidisciplinary Digital Publishing Institute. ISSN 14248220.DOI: 10.3390/s23146383. Sensors 2022, 23(14), 6383 (Article) 5. Nurzada Amangeldy, Marek Milosz, Saule Kudubayeva, Akmaral Kassymova, Gulsim Kalakova, Lena Zhetkenbay. A Real-Time Dynamic Gesture Variability Recognition Method Based on Convolutional Neural Networks. Applied Sciences. Издатель: Multidisciplinary Digital Publishing Institute. ISSN 2076-3417. DOI:10.3390/app131910799. Appl. Sci. 2023, 13(19), 10799 (Article) 6. Amangeldy Nurzada, Kudubayeva Saule, Razakhova Bibigul, Assel Mukanova, Nazira Tursynova. Comparative Analysis Of Classification Methods Of The Dactyl Alphabet Of The Kazakh Language. Journal of Theoretical and Applied Information Technology. Издатель: Little Lion Scientific. ISSN 1992-8645. Volume 100, Issue 19, Pages 5506 - 5513 (Article) 7. Saule Kudubayeva, Nurzada Amangeldy, Ainur Sundetbayeva, Assiya Sarinova. The use of correlation analysis in the algorithm of dynamic gestures recognition in video sequence ACM International Conference Proceeding Series, 5th International Conference on Engineering and MIS, ICEMIS 2021, 6-8 июня 2019. Номер статьи 149162. ISBN: 978-145037212-1. DOI: 10.1145/3330431.33. 30439, Pages 1–11. (Conference Paper) 8. Bekmanova Gulmira, Nazyrova Aizhan, Amangeldy Nurzada, Sharipbay Altynbek, Kudubayeva Saule. A New Approach to Developing a Terminological Dictionary of School Subjects in the Kazakh Language Proceedings - 7th International Conference on Computer Science and Engineering, UBMK 2022, 14-16 сентября 2022, Номер статьи 183844. ISBN:978-166540618-5. DOI: 10.1109/UBMK 55850.2022.9919581. (Conference Paper) Publications in Foreign Scientific Journals: 9. S. Kudubayeva, N. Amangeldy, А. Zakirova. "Kazakh sign language recognition system based on the Bernsen method and morphological structuring". SPEECH TECHNOLOGY 1-2/2020. In Proceedings of International Scientific-Practical Conferences: 10. S. Kudubayeva, N. Amangeldy, et al. "Different Methods of General Recognition of Sign Language for Solving the Problem of Recognizing Kazakh Sign Language". VI International Scientific-Practical Conference "Europe and the Turkic World: Science, Engineering, and Technology", Bursa (Turkey), May 2021. 11. Kudubayeva S, Amangeldy N. "The use of correlation analysis in the algorithm of dynamic gestures recognition in video sequence". ICEMIS 2019, L.N.Gumilyov Eurasian National University, Astana, Kazakhstan. 12. N. Amangeldy, Y.V. Krak, S.A. Kudubayeva. "Classification of gesture demonstration forms based on an ontological model of the subject area". 2020 IEEE International Conference on Advanced Trends in Information Theory ATIT, Kyiv, Ukraine. 13. Amangeldy, N.; Kudubayeva, S.A.; et al. "Comparative analysis on the form of demonstration words of Kazakh sign language with other sign languages". TURKLANG 2022, 113. Author Certificates, Patents: 14. "Automatic Translation System of Kazakh Dactyl Alphabet". No. 29937, November 3, 2022. Amangeldy Nurzada, Saule Kudubayeva, Bekbolat Kurmetbek, Serikbay Kuralov. 15. "Automatic Translation System of Kazakh Sign Language". No. 29874, November 1, 2022. Amangeldy Nurzada, Saule Kudubayeva, Bekbolat Kurmetbek. 16. "Intelligent System for Human Motion Analysis". No. 38947, September 7, 2023. Amangeldy Nurzada, Bekbolat Kurmetbek. 17. Intellectual System for Converting Gestures into Text. No. 38966, dated September 8, 2023. Amangeldy Nurzada, Bekbolat Kurmetbek
Conclusion of the Research Ethics Committee
Defense of the dissertation: https://youtu.be/IpYD8Xw5rvU
'/%3e%3cpath%20fill-rule='evenodd'%20clip-rule='evenodd'%20d='M17.5427%2012.684C17.1383%2011.7889%2016.7126%2011.771%2016.3278%2011.7553C16.0132%2011.7419%2015.6529%2011.7427%2015.2932%2011.7427C14.9333%2011.7427%2014.3483%2011.8774%2013.8537%2012.4152C13.3587%2012.9531%2011.9639%2014.2533%2011.9639%2016.8978C11.9639%2019.5426%2013.8987%2022.098%2014.1684%2022.4569C14.4383%2022.8153%2017.9033%2028.4156%2023.3907%2030.5701C27.9513%2032.3606%2028.8793%2032.0045%2029.8692%2031.9149C30.8591%2031.8253%2033.0633%2030.6149%2033.5133%2029.3597C33.9632%2028.1048%2033.9632%2027.0291%2033.8282%2026.8043C33.6932%2026.5803%2033.3333%2026.4459%2032.7933%2026.1771C32.2534%2025.9083%2029.5992%2024.6078%2029.1043%2024.4286C28.6093%2024.2494%2028.2494%2024.1598%2027.8894%2024.6979C27.5295%2025.2355%2026.4954%2026.4459%2026.1804%2026.8043C25.8655%2027.1635%2025.5505%2027.2083%2025.0105%2026.9395C24.4706%2026.6699%2022.7318%2026.1029%2020.6691%2024.2717C19.0641%2022.8472%2017.9806%2021.0877%2017.6656%2020.5496C17.3506%2020.012%2017.6318%2019.7209%2017.9027%2019.4529C18.1451%2019.2121%2018.4426%2018.8254%2018.7126%2018.5116C18.982%2018.1977%2019.072%2017.9737%2019.252%2017.6153C19.4319%2017.2564%2019.342%2016.9425%2019.207%2016.6737C19.072%2016.405%2018.0228%2013.7468%2017.5427%2012.684Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_181_11959'%20x1='22.8525'%20y1='42.8177'%20x2='22.8525'%20y2='0.921701'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2320B038'/%3e%3cstop%20offset='1'%20stop-color='%2360D66A'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e)
