Defense of the dissertation of Мауленов Қалыбек Сапарұлы for the degree of Doctor of Philosophy (PhD) in the educational program «8D06101 - Computer Science»
L.N. Gumilyov Eurasian National University, a dissertation defense for the degree of Doctor of Philosophy (PhD) by Мауленов Қалыбек Сапарұлы on the topic «Development of an information-algorithmic model of a system for registering biometric data and searching for persons using bar coding technology and deep neural networks» in the educational program of «8D06101 – Computer Science».
The dissertation was completed at the department of «Information Systems» of AKHMET BAITURSYNULY KOSTANAY REGIONAL UNIVERSITY.
The language of defense is russian
Official reviewers:
Bapiyev Ideyat Melsovich - Doctor of Philosophy (PhD), Acting Associate Professor of the Higher School of Information Technology, WKATU Zhangir Khan (Oral);
Аkanova Аkerke Saparovna - doctor of philosophy (PhD), senior lecturer of the department «Computing and software», faculty of computer systems and professional education, S.Seifullin Kazakh Agro Technical Research University (Astana).
Temporary members of the Dissertation Council:
Rakhimova Diana Ramazanovna - Doctor of Philosophy (PhD), Senior Lecturer at the department of Information Systems, Al-Farabi Kazakh National University (Almaty);
Kozhirbayev Zhanibek Mambetkarimovich - Doctor of Philosophy (PhD), Senior Researcher at National Laboratory Astana, Nazarbayev University (Astana);
Farkhadov Mais - doctor of Technical Sciences, chief Researcher at the V. A. Trapeznikov Institute of Management Problems of the Russian Academy of Sciences (Moscow, Russian Federation) (replaced instead of Cariow Aleksandr - Doctor of Science in Computer Science, Professor of the Department of Computer Architecture and Telecommunications, West Pomeranian University of Technology (Szczecin, Poland)).
Scientific consultants:
Kudubaeva Saule Alzhanovna - candidate of technical sciences, senior lecturer of the department «Artificial Intelligence Technologies», faculty of Information Technologies, L.N. Gumilyov Eurasian National University (Astana).
Matveev Yuri Nikolaevich is a leading researcher at the corporate Laboratory of Human-machine Interaction Technologies, Doctor of Technical Sciences, Professor at the ITMO National Research University (St. Petersburg, Russian Federation).
The defense will take place on May 22, 2024, at 03:00 PM in the Dissertation Council for the training direction «8D061 – Information and communication technologies» of L.N. Gumilyov Eurasian National University. The defense meeting is planned to be held offline and online.
Address: Astana, Satpayev Street, 2, room №302.
Abstract (English): Dissertation thesis of Maulenov Kalbek Saparuly for the degree of Doctor of Philosophy (PhD) in the educational program 8D06101 – Сomputer science on the topic "Development of an information-algorithmic model of a system for registering biometric data and searching for persons using bar coding technology and deep neural networks" Relevance of the Dissertation Topic. Ensuring the security of state borders is becoming a priority task for governments of all countries in the world. This is largely due to the increase in the flow of refugees from neighboring countries and mass attempts of illegal immigration by foreign mercenaries and terrorist groups disguised as refugees. As the events of January 2022 showed, Kazakhstan found itself at the forefront of the fight against international terrorism. Leading technology nations are spearheading global efforts to define border security best practices and are actively deploying biometric solutions in automated border control systems (E-Borders, eGates). They are also pioneers in successfully integrating facial recognition technology for person identification and authentication. The widespread use of facial biometrics hinges on secure and effective data practices. This includes robust mechanisms for collecting, storing, and transmitting biometric data responsibly. In addition, currently, face recognition relies heavily on neural networks and deep learning. However, the recent "FAWKES" technology for de-identifying faces poses a challenge. This tool distorts facial images, rendering them unusable for recognition by existing deep learning methods. Reversing these distortions is nearly impossible as "FAWKES" itself utilizes neural networks. The potential presence of such disguised images in biometric and international search databases raises serious concerns. Comprehensive research is crucial to understand and address this issue, potentially enabling robust face recognition even with "FAWKES" interference. Therefore, the research and development of more advanced methods and algorithms for collecting, representing, storing, and transmitting full facial biometric information (including facial images), investigating the problem of recognizing facial images that have undergone the "FAWKES" procedure, and integrating these solutions within the information-algorithmic model of the registration system becomes an urgent problem requiring solutions. The purpose of the dissertation research: Research and development of methods, algorithms, and software for collecting, representing, storing, and transmitting full facial biometric information, and integrating these solutions within the information-algorithmic model of the biometric data registration system. Research Tasks: 1. Review of modern methods for biometric registration of people in border control tasks and typical operational scenarios. 2. Systematization and analysis of facial biometric data acquisition and utilization methods in typical registration system operation scenarios; identifying promising solutions for the information-algorithmic model of the biometric data registration system. 3. Development of a method for precise head alignment in facial biometric systems, ensuring the robustness of facial features extraction across diverse head poses and sizes. 4. Implementation and testing of an algorithm for swift generation of multimedia files (MFs) with comprehensive facial information by efficiently embedding a color QR code within the least significant bits (LSBs) of the original facial image. 5. Development of secure information storage methods for QR codes, leveraging cryptographic algorithms for robust integrity and confidentiality. 6. Development of algorithms and software for determining individuals' familiarity with facial image databases (IF), taking into account ethical considerations and potential biases. 7. Investigating the potential of facial image de-identification utilizing the "Fawkes" technology and analyzing countermeasures against it. The objects of the study are multimedia files containing graphic, biometric and documentary information about persons, and intended for secure storage, transmission and further use of this information in registration systems and biometric databases. The subject of the research is methods and algorithms for obtaining facial biometric and documentary information, its recording, storage and presentation in multimedia files intended for creating biometric databases and digital biometric identification. Research methods: Theoretical analysis: A review and analysis of modern methods for biometric registration using fingerprints, facial recognition, and iris scanning was conducted. Additionally, the study explored methods for obtaining and utilizing biometric data in border control scenarios. The review also investigated the potential of bar codes and QR codes for storing and transmitting biometric information. Comparative analysis: Selection of secure, efficient, and applicable information and algorithmic models for facial recognition systems in border control settings, focusing on deep learning-based approaches and homomorphic encryption. Experimental study: Development of methods and algorithms for collecting, presenting, storing and transmitting complete facial biometric information within the framework of the information-algorithmic model of the biometric data registration system. Development of algorithms for checking databases and determining loyalty, including the development of algorithms for recognizing images that have undergone the de-identification procedure. Programming and Simulation: Software has been developed and tested for collecting and processing biometric and documentary information, for checking against databases and determining loyalty, for recognizing images that have undergone the de-identification procedure. The simulation was performed in the Python programming language using the OPENCV and DLIB libraries. Integration and Application: The developed methods and models were implemented and tested to demonstrate their practical application in real systems. Principal Provisions (proven scientific hypotheses and other conclusions that are new knowledge) presented for defense. Secure Information Exchange: Barcode technology within color QR codes, due to its protection against direct access and cryptographic algorithms, creates conditions for the secure exchange of facial biometric and documentary information when transmitted over international, including open communication channels. Efficient data storage and transmission: A fast method has been developed for forming a multimedia file in the form of a filled container containing complete facial biometric and documentary data, providing a compact representation of information for efficient storage and transmission. Correction of facial position: A method for correcting the position of the facial region has been developed, based on the control of key points along the eye line. This allows facial biometric characteristics to be represented, transmitted, and compared in a standard coordinate grid, improving the accuracy and consistency of data in the creation of biometric databases. Layered structure of QR codes: Colored QR codes generated with biometric and documentary data within the framework of the biometric registration system have a layered structure consisting of three QR codes, which allows them to contain 3 times more information. This provides wide opportunities and prospects for the development of barcode technologies in the field of biometric registration. "Fawkes" de-identification threat: The face de-identification technology "Fawkes" that has emerged in recent years poses a threat to existing face recognition systems. Deterministic face recognition algorithms and their integration with deep learning methods are one way to solve this problem, as these approaches allow extracting the most important image features without resorting to deep processing. Description of the main results of the study: Result 1. The study confirmed that the barcode technology based on colored QR codes and cryptographic algorithms creates secure conditions for the exchange of facial biometric and documentary information via international communication channels, ensuring protection against direct access to data. Result 2. Development of a new fast method for generating a multimedia file - as a filled container, with full facial biometric and documentary information using barcode technology based on QR codes. The method is based on one-step embedding of colored QR codes into the container, which distinguishes it from the method in which embedding is performed on separate layers of colored QR codes in three independent steps. Result 3. Development of a method and implementation of an algorithm for correcting the position of the face area on a standard-sized field, with control of key points along the eye line, which allows representing, transmitting, and comparing facial biometric characteristics within a common coordinate grid and using them in the creation of biometric databases. Result 4. The study confirmed that the generated colored QR codes with biometric and documentary data have a layered structure of three QR codes, which allows them to contain 3 times more information. This opens up prospects for the wide application of barcode technologies in biometric registration. Result 5. The face de-identification technology "Fawkes" has been proven to pose a threat to existing face recognition systems. Deterministic face recognition algorithms integrated with deep learning methods represent an effective way to counteract this threat by extracting key features without deep image processing. Justification for the novelty and importance of the results obtained Result 1. Secure Information Exchange. Novelty: The integration of barcode technology based on colored QR codes and cryptographic algorithms is an innovation that provides secure conditions for the exchange of facial biometric and documentary information. Importance: This is a critical step in ensuring the confidentiality of data transmitted over international communication channels, enhancing the security of the biometric registration system. Result 2. Efficient data storage and transmission. Novelty: The development of a new method for forming a multimedia file as a filled container using colored QR codes is an effective and secure means of storing and transmitting complete facial biometric and documentary information. The innovative one-step method of embedding colored QR codes gives it unique characteristics. Importance: This method provides efficient storage and transmission of complete facial biometric and documentary information, ensuring convenience and security. Result 3. Correction of facial position. Novelty: The developed method of face position correction using key points along the eye line ensures the accuracy of presentation and comparison of facial biometric characteristics. This makes it possible to use these data in the creation of biometric databases, improving their quality. Importance: The accuracy of data representation improves the quality of biometric databases and recognition systems. Result 4. Layered structure of QR codes. Novelty: The study confirms that the layered structure of colored QR codes allows for 3 times more information to be stored, opening up prospects for more capacious biometric registration systems. Importance: This significantly expands the functionality and capacity of systems, which is especially important in the context of the growing volume of biometric data. Result 5. Addressing the threat of de-identification. Novelty: The study emphasizes the importance of countering the threat of face image de-identification posed by the "Fawkes" technology. The proposed methods, which include deterministic algorithms and deep learning, demonstrate an effective approach to extracting key features without deep image processing, ensuring the reliability of face recognition systems. Importance: The proposed methods represent an effective way to maintain the integrity of face recognition systems, countering the threat of face image de-identification. The overall rationale for the novelty and importance of these results lies in their potential to improve the security, efficiency, and functionality of biometric registration systems, which is critical in today's digital society. Practical significance of the obtained results. The results of this research work are information models, methods and algorithms for recognition, image processing, as well as software systems and work scenarios based on QR-code barcode technology, which have high practical significance. These results can be widely used to solve practical problems of facial biometrics and its applications in various fields, including: access control systems, video surveillance systems and forensics, interactive human-computer systems, medicine and biology, and much more. Correspondence to scientific development directions or state programs 4. Information, Communication, and Space Technologies. 4.1 Artificial Intelligence and Information Technologies. 4.1.3 Image Recognition and Processing; 4.1.5 Machine Learning; 4.5 Methods and Systems of Information Security and Data Protection. 4.5.1 Methods and Algorithms for Ensuring Information Security of Complex Systems and Data. 4.5.2 Information Protection Technologies and Hardware and Software Tools. 9. National Security and Defense. 9.2 Applied Scientific Research. 9.2.1 Information Security Personal contribution of the applicant to the preparation of each publication. The research work presented in the dissertation was carried out with the direct participation of the author. The results obtained were published in the form of scientific articles and scientific reports. The author of the dissertation is either the first or the corresponding author in all of these articles, which confirms his direct and full participation in the research conducted. These publications, covering various aspects and research methods, fully disclose the essence of the research conducted, including theoretical and practical aspects. Publications: Articles published in Web of Science and Scopus indexed journals: 1. Maulenov, K., Kudubayeva, S., Razakhova, B. (2023). Modern problems of face recognition systems and ways of solving them. Revue d'Intelligence Artificielle, Vol. 37, No. 1, pp. 209-214. https://doi.org/10.18280/ria.370126 International Conference Proceedings Published in Scopus: 1. Maulenov K. S., Kudubayeva S. A., and Uvaliyeva A. A. "Studying a Face Search Method Based On the Idea of Sparse Data Representation by Generating Random Points," 2021 IEEE International Conference on Smart Information Systems and Technologies (SIST), 2021, pp. 1-6, doi: 10.1109/SIST50301.2021.9465986./2021.2518-1726.87 2. Kukharev G. A., Maulenov K., Shchegoleva N. L. CAN I PROTECT MY FACE IMAGE FROM RECOGNITION? Proceedings of the 9th International Conference "Distributed Computing and Grid Technologies in Science and Education" (GRID'2021), Dubna, Russia, July 5-9, 2021 Publications in the MES RK CSA List: 1. Maulenov K. S., Kudubayeva S. A. Comparative analysis of the characteristics of existing systems for searching and recognizing facial images. [Сравнительный анализ характеристик существующих систем поиска и распознавания изображений лиц]. [Text] / Maulenov K. S., Kudubayeva S. A.// Vestnik KazNRTU. — 2020. —No. 4. — P. 155-160. 2. Maulenov K.S., Kudubayeva S.A. Face image recognition problems using the Monte Carlo method [Монте-Карло әдісі арқылы бет бейнелерін тану мәселелері]. [Text] / Maulenov K. S., Kudubayeva S.A. // Vestnik KazNRTU. — 2020. — No. 4. — P. 151-155. 3. Maulenov K.S., Kudubayeva S.A. Comparative analysis of face detectors HAAR, HOG, CNN [Сравнительный анализ детекторов лиц HAAR, HOG, CNN]. [Text] / Maulenov K.S., Kudubayeva S.A. // The Bulletin of the Academy of Science of the Republic of Kazakhstan. — 2021. — No. 5. — P. 74-82. 4. Maulenov, K., Kaziyeva, N., Shuren, J., & Kudubayeva, S. (2023). Methods of Face De-identification and Solutions to Address Them [МЕТОДЫ ДЕ-ИДЕНТИФИКАЦИИ ИЗОБРАЖЕНИЙ ЛИЦ И ПУТИ ИХ РЕШЕНИИ]. Bulletin of KazATC, 127(4), 196–206. https://doi.org/10.52167/1609-1817-2023-127-4-196-206 Publications in the List of HAC of the Russian Federation: 1. Kukharev G.A., Maulenov K., Shchegoleva N.L. Protecting Facial Images from Recognition on Social Networks: Solutions and Their Prospects [Защита изображений лиц от распознавания в социальных сетях: способы решения и их перспективы]. Scientific and Technical Journal of Information Technologies, Mechanics and Optics – 2021. – P. 755-766. Author's certificates, patents: 1. Kukharev G.A., Maulenov K., Shchegoleva N.L. “Method for Embedding Biometric Information into a Color Face Image and Device for Its Implementation” [Способ встраивания биометрической информации в цветное изображение лица и устройство для его осуществления] - Patent for Invention, No. 2771789 dated May 12, 2022. 2. Shuren Zh.B., Maulenov K.S., Kaziyeva N.M. “Online QR Code Generation Program with Biometric Document Information” [Программа онлайн генерации QR-кода с биометрической документальной информации] (09/08/2023) - Computer Program, No. 38330 dated August 9, 2023, Republic of Kazakhstan; 3. Kazyeva N.M., Maulevov K.S., Kaliyev A.K. “Program for Checking the Correct Position of the Face before Creating a Photo for Registration/Authentication” [Программа проверки правильного положения лица перед созданием фотоснимка для регистрации/аутентификации] - computer program, No. 39851 dated November 29, 2023, Republic of Kazakhstan Implementation of work results: 1. The act of introducing research results into the educational process in the form of a set of programs that can be used for laboratory work, independent work of students and as an illustration for lecture material. 2. The results of the dissertation were used in the applied research project “Development of Methods and Algorithms for Secure Use of QR Codes for Biometric Tasks and Its Applications, Including Blockchain Technology” [Разработка методов и алгоритмов безопасного использования QR-кодов для задач биометрии и ее приложений в том числе технологии блокчейн], AP19678000. 3. Confirmation of the successful implementation of the research results in the development and research of the automated access control and management system of the “Kostanay Expert” company. Structure and scope of the dissertation. The dissertation follows the traditional structure of scientific research, encompassing symbols and abbreviations, an introduction, four distinct sections, a concluding section, and a comprehensive list of utilized sources. The research itself employed a combination of printed methodologies and enhanced data visualization through the incorporation of computer-generated illustrations, diagrams, and tables. In total, the dissertation spans 165 pages, featuring 80 illustrations and diagrams along with 12 tables.
'/%3e%3cpath%20fill-rule='evenodd'%20clip-rule='evenodd'%20d='M17.5427%2012.684C17.1383%2011.7889%2016.7126%2011.771%2016.3278%2011.7553C16.0132%2011.7419%2015.6529%2011.7427%2015.2932%2011.7427C14.9333%2011.7427%2014.3483%2011.8774%2013.8537%2012.4152C13.3587%2012.9531%2011.9639%2014.2533%2011.9639%2016.8978C11.9639%2019.5426%2013.8987%2022.098%2014.1684%2022.4569C14.4383%2022.8153%2017.9033%2028.4156%2023.3907%2030.5701C27.9513%2032.3606%2028.8793%2032.0045%2029.8692%2031.9149C30.8591%2031.8253%2033.0633%2030.6149%2033.5133%2029.3597C33.9632%2028.1048%2033.9632%2027.0291%2033.8282%2026.8043C33.6932%2026.5803%2033.3333%2026.4459%2032.7933%2026.1771C32.2534%2025.9083%2029.5992%2024.6078%2029.1043%2024.4286C28.6093%2024.2494%2028.2494%2024.1598%2027.8894%2024.6979C27.5295%2025.2355%2026.4954%2026.4459%2026.1804%2026.8043C25.8655%2027.1635%2025.5505%2027.2083%2025.0105%2026.9395C24.4706%2026.6699%2022.7318%2026.1029%2020.6691%2024.2717C19.0641%2022.8472%2017.9806%2021.0877%2017.6656%2020.5496C17.3506%2020.012%2017.6318%2019.7209%2017.9027%2019.4529C18.1451%2019.2121%2018.4426%2018.8254%2018.7126%2018.5116C18.982%2018.1977%2019.072%2017.9737%2019.252%2017.6153C19.4319%2017.2564%2019.342%2016.9425%2019.207%2016.6737C19.072%2016.405%2018.0228%2013.7468%2017.5427%2012.684Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_181_11959'%20x1='22.8525'%20y1='42.8177'%20x2='22.8525'%20y2='0.921701'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2320B038'/%3e%3cstop%20offset='1'%20stop-color='%2360D66A'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e)
