한국어 말하기 평가의 최적의 채점자 배정 방식 연구 - 일반화가능도 이론을 중심으로 -
A Study on the Optimal Rater Assigning Method of Korean Speaking Proficiency Test
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
This study intended to propose the method of assigning raters for Korean speaking proficiency test that considered both reliability and practicality of test tool by applying the generalizability theory. Prior to the discussion of the relevant research problem, G-Study was conducted to analyze the influence of the error source of the speaking test tool, and as a result, errors related to the task effect and the interaction effect of the task with the examinee acted as a major error source that interfered with the generalization of the test results. Relatively speaking, the interaction effect between the rater and the interaction effect of the rater with examinee was very small, confirming that the raters consistently participated in the grading without showing any difference in severity or generosity. The G-Study of multi-facet fully crossed design was linked to a D-Study with an embedded rater in the task, comparing the reliability of the test tool with changes in the rater assigning method. The analysis confirmed that the reliability of the test tool when the rater divided the tasks one by one was higher than that of the current test tool in which the two raters scored all the tasks, and that higher reliability of the test tool could be expected compared to the current measurement design, even if only one rater was assigned to each task. In addition, the reliability of the test tool was analysed by adjusting the number of raters assigned to each task from one to three. As a result, assigning two raters to each task was an optimal method of allocating raters, which resulted in a significant drop in error variance and a increase in reliability compared to the measurement design in which one rater was assigned to each task.
Abstract
This study intended to propose the method of assigning raters for Korean speaking proficiency test that considered both reliability and practicality of test tool by applying the generalizability theory. Prior to the discussion of the relevant research problem, G-Study was conducted to analyze the influence of the error source of the speaking test tool, and as a result, errors related to the task effect and the interaction effect of the task with the examinee acted as a major error source that interfered with the generalization of the test results. Relatively speaking, the interaction effect between the rater and the interaction effect of the rater with examinee was very small, confirming that the raters consistently participated in the grading without showing any difference in severity or generosity. The G-Study of multi-facet fully crossed design was linked to a D-Study with an embedded rater in the task, comparing the reliability of the test tool with changes in the rater assigning method. The analysis confirmed that the reliability of the test tool when the rater divided the tasks one by one was higher than that of the current test tool in which the two raters scored all the tasks, and that higher reliability of the test tool could be expected compared to the current measurement design, even if only one rater was assigned to each task. In addition, the reliability of the test tool was analysed by adjusting the number of raters assigned to each task from one to three. As a result, assigning two raters to each task was an optimal method of allocating raters, which resulted in a significant drop in error variance and a increase in reliability compared to the measurement design in which one rater was assigned to each task.
참고문헌 (36)
- 강승혜, "한국어 고급 말하기 평가 도구 개발 기초 연구 - 고급 말하기 토론 활동을 중심으로 -", 외국어로서의 한국어교육, 30, pp.1~21 (2005), 언어연구교육원 한국어학당
- 김상경, "한국어 CBT 말하기 평가의 개선 방안-CBT 와 인터뷰시험의 수험자 담화 비교를 중심으로", 한국언어문화교육학회 학술대회, pp.65~77 (2010)
- 김상경, "학문 목적 한국어 학습자의 말하기 능력 평가 방안 연구", 박사, pp.2015
- 김정숙, "한국어 표준 말하기 시험 측정 도구를 위한 기초 연구", 한민족어문학(구 영남어문학), (51), pp.227~258 (2007), 한민족어문학회
- 김경선, "일반화가능도 이론을 적용한 한국어 말하기 성취도 평가의 신뢰도와 오차요인 분석", 한국어교육, 21(4), pp.51~75 (2010), 국제한국어교육학회[DOI]
- 김경선, "한국어 말하기 평가의 면대면 인터뷰 방식과 컴퓨터 기반 평가 방식 간 측정학적 특성 비교 연구", 박사, 교육학과, pp.2017
- 김지영, "한국어 말하기 평가 채점자의 채점 경향 연구", 박사, pp.2018
- 노대규, "해외 자녀 교육의 현황과 과제: 외국어로서의 한국어 시험과 평가", 이중언어학, 1, pp.139~170 (1983)
- 제1차 한국어능력시험 말하기 평가 개발 연구 보고서, "박동호", 국립국제교육원, ~ (2012)
- 박순옥, "대규모 주관식 검사 상황에서 일반화가능도이론에 의한 신뢰도 추정에 관한 연구", 박사, 교육학, pp.2001
- 성태제, "현대교육평가" (2004), 학지사
- 신동일, "일반화가능도 이론 적용을 중심으로 한 말하기 평가도구 타당도검증 연구", 응용언어학, 17(1), pp.199~221 (2001)
- 양지승, "일반화가능도 이론을 적용한단위검사 구성 검사점수의 신뢰도 추정", 교육평가연구, 20(1), pp.119~139 (2007), 한국교육평가학회
- 원미진, "언어능력 평가로서의 말하기 평가 내용 연구- 한국어 말하기 평가 개발을 위한 국내외 말하기 평가의 비교 분석을 중심으로", 어문론총, (75), pp.45~78 (2018), 한국문학언어학회[DOI]
- 원진숙, "한국어 말하기 능력 평가 기준 설정을 위한 연구", 한국어문교육, 6, pp.101~133 (1992)
- 이미혜, "컴퓨터 기반 한국어 초급 말하기 평가의 타당도 검증 연구", 국어교육, (159), pp.311~338 (2017), 한국어교육학회
- 이향, "말하기 수행 평가에서 발음 범주 채점의 최적화 방안 연구 -일반화가능도 이론을 활용하여-", 한국어교육, 23(2), pp.301~329 (2012), 국제한국어교육학회[DOI]
- 전나영, "한국어 말하기 능력 평가 도구 개발 연구", 외국어로서의 한국어교육, 32, pp.259~338 (2007), 언어연구교육원 한국어학당
- 황지유, "일반화가능도 이론을 적용한 한국어 말하기 숙달도 평가의 신뢰도 및 타당도 분석", 국제어문, (81), pp.321~349 (2019), 국제어문학회[DOI]
- Bachman, L. F., "Language testing in practice: Designing and developing useful language tests (Vol. 1)" (1996), Oxford University Press
- Brennan, R. L., "Generalizability theory: Educational Measurement: Issues and Practice", 11(4), pp.27~34 (1992)
- Brennan, R. L., "Performance assessments from the perspective of generalizability theory", Applied Psychological Measurement, 24(4), pp.339~353 (2000)
- Brennan, R. L., "Generalizability Theory" (2001), Springer-Verlag
- Brown, J. D., "What do the L2 Generalizability Studies Tell Us? 1", Asian Journal of Assessment in Teaching and Learning, 1, pp.1~37 (2011)
- Canale, M., "Theoretical bases of communicative approaches to second language teaching and testing", Applied linguistics, 1(1), pp.1~47 (1980)
- Cronbach, L. J., "The dependability of behavioral measurements: Theory of generalizability of scores and profiles" (1972), John Wiley
- Dunbar, S. B., "Quality control in the development and use of performance assessments", Applied measurement in education, 4(4), pp.289~303 (1991)
- In’nami, Y., "Task and rater effects in L2 speaking and writing:A synthesis of generalizability studies", Language testing, 33(3), pp.341~366 (2016)
- Lee, Y. W., "Dependability of scores for a new ESL speaking test: Evaluating prototype tasks" (2005), Educational Testing Service
- Lin, C. K., "Working with sparse data in rated language tests: Generalizability theory applications", Language Testing, 34(2), pp.271~289 (2017)
- Lynch, B. K., "Using G-theory and manay-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants", Language Testing, 15, pp.158~180 (1998)
- Schoonen, R., "Measuring writing: Recent insights into theory, methodology and practice", pp.1~22 (2012), Brill
- Shavelson, R. J., "Generalizability theory: A primer" (1991), Sage
- Shavelson, R. J., "Sampling variability of performance assessments", Journal of Educational Measurement, 30(3), pp.215~232 (1993)
- Investigating the utility of analytic scoring for the TOEFL Academic Speaking Test (TAST), "Xi, X.", ETS Research, 1~71, pp.ETS Research Report Series, 2006(1) (2006)
- "GENOVA(GENeralized analysis Of VAriance) 3.0 프로그램", ~