본문 바로가기
KERD
논문KCI 등재

국방획득체계와 연계한 국방 인공지능(AI) 체계 시험평가 방안

Test and Evaluation Procedures of Defense AI System linked to the ROK Defense Acquisition System

이민호|최민우|이용복
한국산업경영시스템학회 (KOREAN SOCIETY OF INDUSTRIAL AND SYSTEMS ENGINEERING)|2023.12|한국산업경영시스템학회지 (Journal of Korean Society of Industrial and Systems Engineering)|제46권 제4호|pp. 229-237

0

조회

0

다운로드

3

인용

0

참고문헌

키워드

Defense Test and Evaluation(T&E)Data-based Performance Evaluation ProcedureDefense AI System
주제분류산업공학

초록

In this research, a new Test and Evaluation (T&E) procedure for defense AI systems is proposed to fill the existing gap in established methodologies. This proposed concept incorporates a data-based performance evaluation, allowing for independent assessment of AI model efficacy. It then follows with an on-site T&E using the actual AI system. The performance evaluation approach adopts the project promotion framework from the defense acquisition system, outlining 10 steps for R&D projects and 9 steps for procurement projects. This procedure was crafted after examining AI system testing standards and guidelines from both domestic and international civilian sectors. The validity of each step in the procedure was confirmed using real-world data. This study's findings aim to offer insightful guidance in defense T&E, particularly in developing robust T&E procedures for defense AI systems.

Abstract

In this research, a new Test and Evaluation (T&E) procedure for defense AI systems is proposed to fill the existing gap in established methodologies. This proposed concept incorporates a data-based performance evaluation, allowing for independent assessment of AI model efficacy. It then follows with an on-site T&E using the actual AI system. The performance evaluation approach adopts the project promotion framework from the defense acquisition system, outlining 10 steps for R&D projects and 9 steps for procurement projects. This procedure was crafted after examining AI system testing standards and guidelines from both domestic and international civilian sectors. The validity of each step in the procedure was confirmed using real-world data. This study's findings aim to offer insightful guidance in defense T&E, particularly in developing robust T&E procedures for defense AI systems.

참고문헌 (30)

  1. Calabrese, J., "Framework for Data Quality Evaluation Based on ISO", IEC 25012and ISO, VIII Conference on Cloud Computing, Big Data & Emerging Topics (Modalidad virtual, 8 al 10 de septiembre de 2020), pp.2020[DOI]
  2. Chang, W., "ISO", IEC JTC 1, WG 2 (data)data quality for analytics and machine learning (ML), pp.Information Technology Laboratory, 2022[DOI]
  3. 조규태, "M&S 기반 시험평가 장비 개발의 효율성 및 신뢰성 강화 방안", 한국시뮬레이션학회 논문지, 21(1), pp.89~96 (2012), 한국시뮬레이션학회
  4. 조규표, "국방분야 인공지능 영상감시체계의 시험평가 발전방안 연구", 신뢰성 응용연구, 23(1), pp.97~104 (2023), 한국신뢰성학회[DOI]
  5. 변재현, "자율무기체계 지능화를 위한 시뮬레이션 기반 강화학습", 한국시뮬레이션학회 논문지, 32(2), pp.91~111 (2023), 한국시뮬레이션학회[DOI]
  6. Clegg, A., "Sim2real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?", IEEE Robotics and Automation Letters, 5(4), pp.6670~6677 (2020)
  7. Géron, A., "Hands-On Machine Learning with Scikit-Learn", pp.61~ (2020), Hanbit Publishing Network
  10. "ISO", IEC JTC 1, WG2, 《ISO, pp.IEC WD –5259-5:202X(E)(Artificial intelligence-Data quality for analytics and ma-chine learning(ML)-Part 5: Data quality governance framework≫[DOI]
  11. "ISO", IEC JTC 1, WG2, 《ISO, pp.IEC WD 5259-1:202X(E)(Artificial intelligence-Data quality for analytics and ma-chine learning(ML)-Part 1: Overview, terminology, and example≫[DOI]
  12. "ISO", IEC JTC 1, WG2, 《ISO, pp.IEC WD 5259-2:202X(X)(Artificial intelligence-Data quality for analytics and ma-chine learning(ML)-Part 2: Data quality measures≫[DOI]
  13. "ISO", IEC JTC 1, WG2, 《ISO, pp.IEC WD 5259-3:20##(X)(Artificial intelligence-Data quality for analytics and ma-chine learning(ML)-Part 3: Data quality management requirements and guidelines≫[DOI]
  14. "ISO", IEC JTC 1, WG2, 《ISO, pp.IEC WD 5259-4:202#(E)(Artificial intelligence-Data quality for analytics and ma-chine learning(ML)-Part 4: Data quality process frame-work≫[DOI]
  15. "ISO", IEC JTC 1, IEC 25024:2015 Systems and software engineering: Systems and software Quality Requirements and Evaluation (SQuaRE) - Measurement of data quality≫, 2015[DOI]
  16. Kim, B. -S., "A study on the test evaluation of automated-vehicles based on artificial intelligence", Spring Conf, pp.722~722 (2019)
  17. Kim, B. S., "A Study on the Test Evaluation of Automated-Vehicles Based on Artificial Intelligence", The Korean Society of Automotive Engineers, Annual Spring Conference, pp.722~722 (2019)
  18. Kim, M. Y., "Acquisition Process of Military Weapon System Using Artificial Intelligence", Proceed-ings of Symposium of the Korean Institute of Communica-tions and Information Sciences (2022)
  19. Kohavi, R., "A study of cross-validation and bootstrap for accuracy estimation and model selection", Proc. 14th Int. Joint Conf. on Artif. Intell., pp.1137~1143 (1995)
  20. Kurrek, P., "Ai motion control–a generic approach to develop control policies for robotic manipulation tasks", Proceed-ings of the Design Society: International Conference on Engineering Design, 1(1) (2019), Cambridge University Press
  21. 이용복, "인공지능(AI) 무기체계 시험평가방법 발전방안 연구: 분류모델 성능평가를 중심으로", 신뢰성 응용연구, 22(1), pp.1~9 (2022), 한국신뢰성학회[DOI]
  22. Trustworthy Artificial Intelligence Development Guide (Plan), "Ministry of Science and ICT", ~ (2023)
  23. National Information Society Agency, "Republic of Korea: Guide for Constructing Artificial Intelligence Training Data", ~ (2023)
  24. 박주혜, "시험평가용 M&S에 대한 V&V 프로세스 연구", 한국산학기술학회논문지, 20(9), pp.397~404 (2019), 한국산학기술학회[DOI]
  25. Test & evaluation of ai-en-abled and autonomous systems: A literature review, "Porter, D. J.", Institute for Defense Analyses, ~ (2020)
  26. Priestley, M., "A survey of data quality requirements that matter in ML develop-ment pipelines", ACM Journal of Data and Information Quality, 15(2), pp.1~39 (2023)
  27. Rangineni, S., "An Analysis of Data Quality Requirements for Machine Learning Development Pipelines Frame-works", International Journal of Computer Trends and Technology, 71(9), pp.16~27 (2023)
  28. Schnelle, S., "ADS Standardization Landscape: Making Sense of its Status and of the Associated Research Questions", arXiv preprint arXiv, 2023, 2306.17682, ~ (2023)
  29. Shin, J. H., "Data quality verification method for artificial intelligence learning", Journal of Electronic Engineering, 48(7), pp.28~34 (2021)
  30. Torchiano, M., "Preserving the benefits of Open Government Data by measuring and improving their quality: An Empirical Study", IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), 1 (2017), IEEE

이 간행물의 다른 논문 (32건)

제품 및 서비스 개선을 위한 기술기회 발굴: 특허와 상표 데이터 활용윤장혁, 박서인, 손창호, 이승현, 이지호pp. 1-14토픽 모형과 ChatGPT를 활용한 스마트팩토리 연관 특허 빅데이터 분석에 관한 연구김상국, 임정선, 권태훈, 윤민영pp. 15-31미세먼지 농도 및 대중의 인식도가 천식질환 발생빈도에 미치는 영향 분석이기광pp. 32-38정부출연연구기관 식품연구개발사업의 기술이전 성과동인 분석김승운, 정미림pp. 39-52A Network-based Optimization Model for Effective Target Selection이기현, 이진호pp. 53-62
원문 보기