본문 바로가기
KERD
논문KCI 등재

종이에서 컴퓨터로의 매체 전환에 따른 평가 결과의 변화 탐색: 고등학교 수학 국가수준 학업성취도 평가를 중심으로

Exploring the change in achievement by the transition of the test mode from paper to computer: Focusing on the national assessment of educational achievement of high school mathematics

이경화|김영준|송창근|정혜윤
한국수학교육학회|2022.11|수학교육 (The Mathematical Education)|제61권 제4호|pp. 597-612|DOI https://doi.org/10.7468/mathedu.2022.61.4.597

1

조회

0

다운로드

1

인용

0

참고문헌

키워드

대규모 평가mode effectslarge-scale assessment컴퓨터 기반 수학 평가모드 효과혁신 문항computer-based test of mathematicsinnovative item

초록

최근 국내외 대규모 평가를 중심으로, 수학 평가가 전통적인 지필 평가에서 컴퓨터 기반 평가로 전환되고 있다. 본 연구는 평가 문항의 유형, 문항에반영된 기술공학적 기능, 학생의 컴퓨터 사용 특성 및 컴퓨터 평가 환경에 따라 평가 매체 전환에 따른 학생의 성취도 차이가 어떻게 변화하는지탐색하였다. 이를 위해 지필 환경과 컴퓨터 환경에서 실행된 2020 고등학교 수학의 국가수준 학업성취도 평가 결과를 분석하였다. 연구결과, 첫째, 단순 매체만 전환된 경우 모드 효과는 대체로 미미하였지만, 서술형 문항의 경우 단답형이나 선택형 문항에 비해 모드 효과가 더 크게 나타났다. 둘째, 매체 전환과 기술공학적 기능의 반영이 함께 이루어진 경우, 문항에 반영된 기술공학적 기능에 따라 모드 효과에 차이가 있었다. 셋째, 컴퓨터 기반 평가 시행환경, 학생의 컴퓨터 활용 빈도와는 달리, 학생의 컴퓨터 활용 효능감에 따라 컴퓨터 기반 평가에서 통계적으로 유의한점수 차이가 나타났다. 이와 같은 연구 결과는 향후 평가 내용과 역량에 맞춘 적절한 기술공학적 기능의 도입이 필요하며, 학생의 수학 학습 능력외의 부수적인 능력이나 상황이 평가 결과에 영향을 미치지 않도록 세밀한 평가 설계와 환경 구비가 필요함을 시사한다.

Abstract

Recently, large-scale mathematics assessments are shifting from traditional paper-based tests to computer-based tests, nationally and internationally. This study explored the mode effect (the difference in student achievement by the change of test mode) according to the types of test items, the technological function reflected in the items, the characteristics of students' computer use, and the computer-based test environment. To this end, we analyzed the results of the 2020 national assessment of educational achievement of high school mathematics conducted on a paper and computer basis. As a result, firstly, the mode effect induced by the mode transition was generally insignificant, but the mode effect was larger in the extended response type than other types. Secondly, there were differences in the mode effect according to the transition to test with computer mode where innovative items were added. Thirdly, the difference between mode effects was statistically significant according to the student’s sense of efficacy in computer use. The results of this study suggest that innovative items should be introduced deliberately according to the targeted content and competency in evaluation, and that assessment design and environment preparation need to be carefully developed so that nonessential abilities other than students' mathematical ability or incidental situation do not distort the assessment results.

참고문헌 (48)

  1. Arslan, B., "The Effect of Drag-and-Drop Item Features on Test-Taker Performance and Response Strategies", Educational Measurement : Issues and Practice, 39(2), pp.96~106 (2020)[DOI]
  2. Backes, B., "Is the pen mightier than the keyboard? The effect of online testing on measured student achievement", Economics of Education Review, 68(1), pp.89~103 (2019)[DOI]
  3. Bennett, R. E., "Does it Matter if I Take My Mathematics Test on Computer? A Second Empirical Study of Mode Effects in NAEP", Journal of Technology, Learning, and Assessment, 6(9), pp.1~38 (2008)
  4. Bridgeman, B., "A Comparison of quantitative questions in open-ended and multiple-choice formats", Journal of Educational Measurement, 29(3), pp.253~271 (1992)
  5. Buerger, S., "What makes the difference? The impact of item properties on mode effects in reading assessments", Studies in Educational Evaluation, 62(3), pp.1~9 (2019)[DOI]
  6. Clements, D. H., "From exercises and tasks to problems and projects : Unique contributions of computers to innovative mathematics education", The Journal of Mathematical Behavior, 19(1), pp.9~47 (2020)[DOI]
  7. Csapó, B., "Assessment and teaching of 21st century skills" (2012), Springer
  8. Drasgow, F., "Computer-based Testing and the Internet: Issues and Advances", pp.59~76 (2006), John Wiley and Sons Ltd
  9. Ebrahimi, M. R., "Score equivalence, gender difference, and testing mode preference in a comparative study between computer-based testing and paper-based testing", International Journal of Emerging Technologies in Learning, 14(7), pp.128~143 (2019)[DOI]
  10. Eid, G. K., "An investigation into the effects and factors influencing computer-based online math problem-solving in primary schools", Journal of Educational Technology Systems, 33(3), pp.223~240 (2005)[DOI]
  11. Fishbein, B., "The TIMSS 2019 item equivalence study : Examining mode effects for computer-based assessment and implications for measuring trends", Large-scale Assessments in Education, 6(1), pp.1~23 (2018)[DOI]
  12. Geraniou, E., "Towards a definition of"mathematical digital competency"", Educational Studies in Mathematics, 102(1), pp.29~45 (2019)[DOI]
  13. Haladyna, T., "A review of multiple-choice item-writing guidelines for classroom assessment", Applied Measurement in Education, 15(3), pp.309~333 (2002)[DOI]
  14. Heyd-Metzuyanim, E., "Mathematical media literacy in the COVID-19 pandemic and its relation to school mathematics education", Educational Studies in Mathematics, 108(1), pp.201~225 (2021)[DOI]
  15. Hoogland, K., "Computer-based assessment of mathematics into the twenty-first century : pressures and tensions", ZDM, 50(4), pp.675~686 (2018)[DOI]
  16. Horkay, N., "Does it matter if I take my writing test on computer? An empirical study of mode effects in NAEP", The Journal of Technology, Learning, and Assessment, 5(2) (2006)[DOI]
  17. Hu, L., "Multimedia effect in problem solving : A meta-analysis", Educational Psychology Review, 33(4), pp.1717~1747 (2021)[DOI]
  18. Jeong, H., "A comparative study of scores on computer-based tests and paper-based tests", Behaviour & Information Technology, 33(4), pp.410~422 (2014)[DOI]
  19. NAEP transition to digitally based assessments in mathematics and reading at grades 4 and 8: Mode evaluation study, "Jewsbury, P.", National Center for Education Statistics, ~ (2017)
  20. 정혜윤, "컴퓨터 기반 국가수준 학업성취도 평가에 대한 학생과 교사의 만족도 및 문제점 인식 : 고등학교 수학 과목을 중심으로", 학교수학, 24(1), pp.119~145 (2022), 대한수학교육학회[DOI]
  21. Karay, Y., "Computer versus paper : Does it make any difference in test performance", Teaching and Learning in Medicine, 27(1), pp.57~62 (2015)[DOI]
  22. Keng, L., "Item-level comparative analysis of online and paper administrations of the texas assessment of knowledge and skills", Applied Measurement in Education, 21(3), pp.207~226 (2008)[DOI]
  23. Kingston, N., "Comparability of computer-and paper-administered multiple-choice tests for K-12 populations : A synthesis", Applied Measurement in Education, 22(1), pp.22~37 (2008)[DOI]
  24. Advancing the test tools for Electronic National Assessment of Educational Achievement (eNAEA), "Lee, J. B.", KICE, ~ (2021)
  25. 이상구, "언택트(Untact) 디지털 전환(DT) 시대의 대학수학교육 사례", 수학교육 논문집, 34(3), pp.201~214 (2020), 한국수학교육학회[DOI]
  26. Methods and Procedures: TIMSS 2019 Technical Report, "Martin, M. O.", TIMSS & PIRLS International Study Center, ~ (2020)
  27. Martin, R., "Towards a research agenda on computer-based assessment", pp.6~9 (2008), Office for Official Publications of the European Communities
  28. Mcclelland, T., "A comparison of computer based testing and paper and pencil testing in mathematics assessment", The Online Journal of New Horizons in Education, 10(2), pp.78~89 (2020)
  29. McDonald, A., "The impact of individual differences on the equivalence of computer-based and paper-and-pencil educational assessments", Computers and Education, 39(3), pp.299~312 (2002)[DOI]
  30. Noyes, J., "Computer- vs. paper-based tasks: Are they equivalent?", Ergonomics, 51(9), pp.1352~1375 (2008)[DOI]
  31. 박만구, "수학교육에서 인공지능 활용 가능성", 수학교육 논문집, 34(4), pp.545~561 (2020), 한국수학교육학회
  32. Parshall, C. G., "Elements of adaptive testing", pp.215~230 (2009), Springer
  33. Poggio, J., "A comparative evaluation of score results from computerized and paper & pencil mathematics testing in a large scale state assessment program", The Journal of Technology, Learning, and Assessment, 3(6) (2005)
  34. Pommerich, M., "Developing computerized versions of paper-and-pencil tests: mode effects for passage-based tests", The Journal of Technology, Learning, and Assessment, 2(6), pp.1~44 (2004)
  35. Rose, J., "NAPLAN discourses : A systematic review after the first decade", Discourse : Studies in the Cultural Politics of Education, 41(6), pp.871~886 (2020)[DOI]
  36. Russell, M., "Testing on computers: A follow-up study comparing performance on computer and on paper" (1999), Boston College
  37. Russell, M., "The influence of computer-print on rater scores", Practical Assessment, Research, and Evaluation, 9(10), pp.1~13 (2004)
  38. Sandene, B., "Online assessment in mathematics and writing:Reports from the NAEP technology-based assessment project, research and development series" (2005), National Center for Education Statistics
  39. Scalise, K., "Computer-based assessment in E-Learning : A framework for constructing"Internet Constraint"questions and tasks for technology platforms", The Journal of Technology, Learning, and Assessment, 4(6) (2006)
  40. Schleppegrell, M. J., "The linguistic challenges of mathematics teaching and learning : A research review", Reading & Writing Quarterly, 23(2), pp.139~159 (2007)[DOI]
  41. Schoenfeld, A. H., "On learning and assessment. Assessment in Education : Principles", Policy & Practice, 24(3), pp.369~378 (2017)[DOI]
  42. Shacham, M., "Computer-based exams in undergraduate engineering courses", Computer Applications in Engineering Education, 6(3), pp.201~209 (1998)
  43. Sireci, S. G., "Handbook of test development", pp.329~347 (2006), Lawrence Erlbaum Associates Publishers
  44. Spires, H. A., "Advanced methodologies and technologies in library science, information management, and scholarly inquiry", pp.12~21 (2019), IGI Global
  45. Taylor, C., "The Relationship between computer familiarity and performance on computer-based TOEFL test tasks", Educational Testing Service, ~ (1998)
  46. Yoo, S. M., "SPSS statistical analysis for writing a thesis" (2013), Slow& Steady
  47. Wang, S., "A meta-analysis of testing mode effects in grade K-12 mathematics tests", Educational and Psychological Measurement, 67(2), pp.219~238 (2007)[DOI]
  48. Zilles, C., "Making testing less trying : Lessons learned from operating a Computer-Based Testing Facility", 2018 IEEE Frontiers in Education (FIE) Conference (2018), IEEE

관련 논문AI추천

국가수준 학업성취도 평가의 시행 매체 변화가 학업성취도에 미치는 영향구남욱, 김용남, 이소라한국교육과정평가원 · 교육과정평가연구2022 · 제25권 제1호컴퓨터 기반 평가와 지필평가 간 학생 응답 특성 탐색 -컴퓨터 기반 국가수준 학업성취도 평가 병행 시행 결과를 중심으로-구자옥, 백종호, 이재봉한국과학교육학회 · 한국과학교육학회지2023 · 제43권 제1호TIMSS 2019 과학 문항에서 컴퓨터 기반 평가와 지필 평가 간의 검사 매체 차이 탐색김현경대한화학회 · 대한화학회지2021 · 제65권 제1호차별적 검사 기능 분석을 통한 컴퓨터 기반 검사의 양식 효과 연구김선미, 박찬호교육문제연구소 · 교육연구2025 · 제92권 컴퓨터 기반 평가에서 디지털 기기 친숙도에 따른 차별기능문항 분석: eTIMSS 2019 4학년 수학을 중심으로서경숙, 안해연, 최윤정한국교육과정평가원 · 교육과정평가연구2024 · 제27권 제1호컴퓨터 기반 평가와 지필 기반 평가 간 점수 비교: 성별과 학습능력에 따른 차이분석을 중심으로정한호한국교육공학회 · 교육공학연구2010 · 제26권 제3호

이 간행물의 다른 논문 (6건)

Beyond adaptation: Transforming pedagogies of teaching elementary mathematics methods course in the online environment여승현, 권민성pp. 521-537교육적 지도서 개발 사례에 적용한 소설적 글쓰기 기법으로 살펴 본 예술기반 연구의 수학교육연구 활용 가능성 논의서희주pp. 539-557수학 교수학적 어휘의 변화: 텍스트 마이닝 기법을 이용한 교실수업 어휘 연구의 확장김희정, 이기마pp. 559-579수학교과에서 자동문항생성 기반의 디지털 평가 활용 방안: CAFA 시스템을 중심으로김성연pp. 581-595고등학생이 지각한 수학 수업방식이 수학자기효능감에 미치는 영향력에 대한 유형탐색: REBUS-PLS를 적용하여송효섭, 정희선pp. 613-629
원문 보기