본문 바로가기
KERD
논문KCI 등재

채점 정확도와 채점자 편향을 통해 본 대학생의 평가 역량

Exploring the evaluation capacity of university students through rating accuracy and rater bias

0

조회

0

다운로드

0

인용

0

참고문헌

키워드

주제분류심리과학

초록

채점자의 역량을 개인 수준에서 측정하는 대표적인 지표는, 채점 정확도와 채점자 편향(채점 엄격성, 중앙값 채점 경향성, 채점 무작위성)이다. 두 지표는 주로 전문 채점자를 대상으로 연구되었지만, 영역이 다른 둘 이상의 과제를 통해 평가 역량의 일반성이 검증된 바 없다. 본 연구에서는 대학생들로 하여금 언어영역 과제와 시공간영역의 과제를 채점하게 한 결과로, 채점 정확도와 채점자 편향을 탐색하였다. 분석 결과 한 영역에서 채점 정확도가 높고 채점척도를 잘 사용하는 학생들은 다른 평가영역에서도 그럴 가능성이 높고, 채점 정확도가 높은 학생들보다 낮은 학생들이 평가영역 차이에 영향을 받는 정도가 유의하게 더 컸다. 전체 논의에서는 대학생 평가역량 측정 연구에 시사하는 점과 한계점이 다루어졌다.

Abstract

Traditionally, researchers have used rating accuracy and rater bias (severity, centrality, and randomness) as individual-level indicators of rating quality. While these have been studied mostly for expert raters, research on whether evaluation capacity is domain-general over two or more different tasks is lacking. Thus, we investigated the two indicators in the context of undergraduate raters. In two studies, undergraduates scored outputs from a verbal-linguistic task and a visual-spatial task. The results showed that proficient students in one domain are also likely to be proficient in the other in terms of rating accuracy and the use of rating scale. In addition, students with lower rating accuracy were more significantly affected by the difference between domains compared to more accurate students. We also discuss the implications and limitations of our findings on measuring student evaluation capacity.

참고문헌 (43)

  1. Barron, F., "The disposition toward originality", The Journal of Abnormal and Social Psychology, 51, pp.478~485 (1955)[DOI]
  2. Becker, B. E., "Influence of halo error on appraisal effectiveness : A conceptual and empirical reconsideration", Journal of Applied Psychology, 71, pp.662~671 (1986)[DOI]
  3. Bejar, I. I., "Rater cognition : Implications for validity", Educational Measurement : Issues and Practice, 31(3), pp.2~9 (2012)[DOI]
  4. 조재한, "평가정확성에 대한 평가자 훈련 유형의 영향: 책임성과 평가자 기분(mood)의 상호작용 효과", 한국심리학회지: 산업 및 조직, 21(4), pp.745~768 (2008), 한국산업및조직심리학회[DOI]
  5. Cho, K., "Student revision with peer and expert reviewing", Learning and Instruction, 20, pp.328~338 (2010)[DOI]
  6. Cronbach, L. J., "Processes affecting scores on “understanding of others” and “assumed similarity"", Psychological Bulletin, 52, pp.177~193 (1955)[DOI]
  7. Diedrich, J., "Are creative ideas novel and useful?", Psychology of Aesthetics, Creativity, and the Arts, 9, pp.35~40 (2015)[DOI]
  8. Engelhard Jr, G., "Examining rater errors in the assessment of written composition with a many-faceted Rasch model", Journal of Educational Measurement, 31, pp.93~112 (1994)[DOI]
  9. Engelhard, G. Jr., "Evaluating rater accuracy in performance assessments", Journal of Educational Measurement, 33, pp.56~70 (1996)[DOI]
  10. Engelhard, J. G., "Invariant measurement: Using rasch models in the social, behavioral, and health sciences" (2012), Taylor & Francis Group
  11. Finke, R. A., "Creative Imagery: Discoveries and inventions in Visualization" (1990), Psychology Press
  12. Finke, R. A., "Creative Cognition : Theory, research, and applications, Cambridge" (1992), MIT Press
  13. Furr, R., "A framework for profile similarity : Integrating similarity, normativeness, and distinctiveness", Journal of Personality, 76, pp.1267~1316 (2008)[DOI]
  14. Ji-Ping Jiang, "Fostering college students’ critical thinking skills through peer assessment in the knowledge building community", Interactive Learning Environments, 31(10), pp.6480~6496 (2022), Informa UK Limited[DOI]
  15. Jin, K. Y., "Detecting rater centrality effects in performance assessments : A model-based comparison of centrality indices", Measurement : Interdisciplinary Research and Perspectives, 20, pp.228~247 (2022)[DOI]
  16. Jin, K. Y., "A new facets model for rater's centrality", extremity response style, 2018, pp.55, 543~563
  17. 진미석, "대학생 핵심역량 진단체제 구축 방안 연구", 교육행정학연구, 29(4), pp.461~486 (2011), 한국교육행정학회
  18. Johnson, R. L., "Assessing performance: Designing, scoring, and validating performance tasks" (2008), Guilford Press
  19. Kahneman, D., "Noise : a flaw in human judgment" (2021), Hachette UK
  20. Dave S. Kerby, "The Simple Difference Formula: An Approach to Teaching Nonparametric Correlation", Comprehensive Psychology, 3, pp.11.IT.3.1~ (2014), Ammons Scientific[DOI]
  21. Koo, T. K., "A guideline of selecting and reporting intraclass correlation coefficients for reliability research", Journal of Chiropractic Medicine, 15, pp.155~163 (2016)[DOI]
  22. Lee, J. -w., "How quantity judgment changes as the number of cues increases : An analytical framework and review", Psychological Bulletin, 112, pp.363~377 (1992)[DOI]
  23. Letzring, T. D., "Information quantity and quality affect the realistic accuracy of personality judgment", Journal of Personality and Social Psychology, 91, pp.111~ (2006)[DOI]
  24. Hongli Li, "Does peer assessment promote student learning? A meta-analysis", Assessment & Evaluation in Higher Education, 45(2), pp.193~211 (2019), Informa UK Limited[DOI]
  25. Linacre, J. M., "Many-facet Rasch measurement" (1989), MESA Press
  26. McGraw, K. O., "Forming inferences about some intraclass correlation coefficients", Psychological Methods, 1, pp.30~46 (1996)[DOI]
  27. McNamara, T., "The Rasch wars : The emergence of Rasch measurement in language testing", Language Testing, 29, pp.555~576 (2012)[DOI]
  28. Myford, C. M., "Detecting and measuring rater effects using many-facet Rasch measurement : Part I", Journal of Applied Measurement, 4, pp.386~422 (2003)
  29. Myford, C. M., "Detecting and measuring rater effects using many-facet Rasch measurement : Part Ⅱ", Journal of Applied Measurement, 5, pp.189~227 (2004)
  30. 박주용, "동료평가의 현황과 전망", 인지과학, 29(2), pp.85~104 (2018), 한국인지과학회[DOI]
  31. Pulakos, E. D., "A comparison of rater training programs : Error training and accuracy training", Journal of Applied Psychology, 69, pp.581~588 (1984)[DOI]
  32. Runco, M. A., "The standard definition of creativity", Creativity Research Journal, 24, pp.92~96 (2012)[DOI]
  33. Schleicher, D. J., "A new frame for frame-of-reference training : Enhancing the construct validity of assessment centers", Journal of Applied Psychology, 87, pp.735~746 (2002)[DOI]
  34. Snow, R. E., "Advances in the psychology of human intelligence (Vol. 2)", pp.47~104 (1984), Psychology Press
  35. Stein, M. I., "Creativity and culture", The Journal of Psychology : Interdisciplinary and Applied, 36, pp.311~322 (1953)[DOI]
  36. Topping, K., "Peer Assessment between students in colleges and universities", Review of Educational Research, 68, pp.249~276 (1998)[DOI]
  37. Topping, K. J., "Peer assessment", Theory into practice, 48, pp.20~27 (2009)[DOI]
  38. Uto, M., "Empirical comparison of item response theory models with rater's parameters", Heliyon, 4(5), pp.e00622~ (2018)[DOI]
  39. Wind, S. A., "Examining rating quality in writing assessment : rater agreement, error, and accuracy", Journal of Applied Measurement, 13, pp.321~335 (2012)
  40. Wind, S. A., "How invariant and accurate are domain ratings in writing assessment", Assessing Writing, 18, pp.278~299 (2013)[DOI]
  41. Wobbrock, J. O., "The aligned rank transform for nonparametric factorial analyses using only ANOVA procedures", Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), pp.143~146 (2011)
  42. Wolfe, E. W., "Application of latent trait models to identifying substantively interesting raters", Educational Measurement : Issues and Practice, 31(3), pp.31~37 (2012)[DOI]

관련 논문AI추천

이 간행물의 다른 논문 (7건)