논설문 평가에 나타난 국어교사의 평가 특성 및 편향 분석
The Rater Characteristics and Raters Bias in Korean Language Teacher's Persuasive Writing Assessment
0
조회
0
다운로드
25
인용
0
참고문헌
키워드
초록
The aims of this paper is to analyze the features of Korean language teacher's writing assessment based on Many-Facets Rasch model. For that, the 68 Korean language teachers assessed 35 persuasive writing of middle school students randomly. the rating data was analyzed by the computer program FACETS (ver 3.66.1, Linacre, 2004), with FACETS analyses run on the persuasive writing. The results are following. (1) the 14 teachers(20.58%) out of 68 were founded as overfiting raters. (2) the 10 teachers(14.70%) out of 68 were founded as underfiting or misfiting raters. (3) In the rater's gender, male and female teachers showed acceptable fit that means stable rating. But the female teachers had higher severity than that of the male teacher. (4) In the rater's career, teachers who above 20 years of teaching career reported the lowest severity level, and teachers who 1∼5 years of teaching career reported the highest severity level within teacher groups. (5) MFRM revealed several recurring bias patterns among rater subgroups. In rater–category and rater-examinee bias interactions, Some raters also rated more severely and more leniently than expected.
Abstract
The aims of this paper is to analyze the features of Korean language teacher's writing assessment based on Many-Facets Rasch model. For that, the 68 Korean language teachers assessed 35 persuasive writing of middle school students randomly. the rating data was analyzed by the computer program FACETS (ver 3.66.1, Linacre, 2004), with FACETS analyses run on the persuasive writing. The results are following. (1) the 14 teachers(20.58%) out of 68 were founded as overfiting raters. (2) the 10 teachers(14.70%) out of 68 were founded as underfiting or misfiting raters. (3) In the rater's gender, male and female teachers showed acceptable fit that means stable rating. But the female teachers had higher severity than that of the male teacher. (4) In the rater's career, teachers who above 20 years of teaching career reported the lowest severity level, and teachers who 1∼5 years of teaching career reported the highest severity level within teacher groups. (5) MFRM revealed several recurring bias patterns among rater subgroups. In rater–category and rater-examinee bias interactions, Some raters also rated more severely and more leniently than expected.
참고문헌 (39)
- 김성숙, "논술 문항 채점의 변동 요인 분석과 일반화 가능도 계수의 최적화 조건", 교육 평가연구, 8(1), pp.35~57 (1995)
- 김성숙, "채점의 변동 요인 분석 방법에 대한 고찰: 일반화 가능도 이론과 다국면 라쉬 모형의 적용과 재해석", 교육평가연구, 14(1), pp.303~325 (2001)
- 박영민, "현직 국어교사와 예비 국어교사의 쓰기 평가 비교 연구", 교육과정평가연구, 12(1), pp.123~143 (2009), 한국교육과정평가원
- 박영민, "중학생 논설문 평가의 모평균 추정과 평가 예시문 선정", 국어교육, (131), pp.437~461 (2010), 한국어교육학회
- 박영민, "Rasch 모형을 활용한 국어교사의 쓰기 평가 특성 분석－중학생 설명문 쓰기 평가를 중심으로－", 국어교육학연구, (37), pp.367~391 (2010), 국어교육학회
- 성태제, "현대 교육평가" (2002), 학지사
- 이현숙, "다국면 라쉬모형을 적용한 논술 채점 상황에서 채점 설계 및 채점자 특성이 채점의 정확성에 미치는 효과", 교육평가연구, 21(4), pp.129~152 (2008), 한국교육평가학회
- 장소영, "언어교육평가 연구를 위한 FACETS 프로그램 : 기초과정편" (2009), 글로벌 콘텐츠
- 지은림, "RASCH 모형의 이론과 실제" (2000), 교육과학사
- Barrett, S, "The impact of training on rater variability", International Education Journal, 2(1), pp.49~58 (2001)
- Barnes, L. L., "Gender bias in teachers’written comments, in Gender in the classroom：Power and pedagogy", pp.140~159 (1990), University of Illinois Press
- Bond, T.G, "Applying the rasch model: Fundamental measurement in the human sciences" (2001), Lawrence Erlbaum Associates
- Cason G.J, "A deterministic theory of clinical performance rating", Evaluation and the Health Professions, 7, pp.221~247 (1984)
- Cantor, N. K., "The reliability and validity of writing assessment: An investigation of rater, prompt within mode, and prompt between mode sources of error", Paper presented at the annual meeting of the American Educational Research Association (1986)
- Cronbach, L. J, "Essentials of psychological testing (5th ed.)" (1990), New York HarperCollins
- Engelhard, G., Jr, "The measurement of writing ability with a many-faceted Rasch model", Applied Measurement in Education, 5, pp.171~191 (1992)
- Engelhard, G., Jr, "Examining rater errors in the assessment of written composition with a many-faceted Rasch model", Journal of Educational Measurement, 31, pp.93~112 (1994)
- Engelhard, G, "Large-scale assessment programs for all students: Validity, technical adequacy, and implementation", pp.261~287 (2002), Erlbaum
- Engelhard, G., Jr., "The influences of mode of discourse, experiential demand, and gender on the quality of student writing", into the Teaching of English, 26, pp.315~336 (1991)
- Engelhard, G, "Monitoring faculty consultant performance in the Advanced Placement English Literature and Composition Program with a many-faceted Rasch model" (2003), College Entrance Examination Board
- Gabrielson, S., "The effects of task choice on the quality of writing obtained in a statewide assessment", Applied Measurement in Education, 8, pp.273~290 (1995)
- Etaugh, C. B., "Evaluating competence of women and men: Effects of experimenter gender and group gender composition", Psychology of Women Quarterly, 12, pp.191~200 (1988)
- Haswell, J, "Gendership and the miswriting of students", College Composition & Communication, 46(2), pp.223~254 (1995)
- Haswell, R. H, "Gender bias and critique of student writing", Assessing Writing, 3, pp.31~83 (1996)
- Hayes, J. R, "Understanding and reducing the knowledge effect: Implication for writers", Written Communication, 25(1), pp.104~118 (2008)
- Linacre, J. M.., "Many-facet Rasch measurement" (1989), MESA Press
- Lumley, T., "Rater characteristics and rater bias: Implications for training", Language Testing, 12(1), pp.54~71 (1995)
- Lunz, M. E., "Impact of examiners on candidate scores: An introduction to the use of multifacet Rasch model analysis for oral examinations", Teaching and Learning in Medicine, 5(3), pp.174~181 (1993)
- McNamara, T. F, "Measuring second language performance" (1996), Addison Wesley Longman Ltd
- Myford, C. M, "Judging acting ability: The transition from notice to expert", Paper presented at the American Educational Research Association (1991)
- Myford, C. M., "Detecting and measuring rater effects using many-facet Rasch measurement: Part I", Journal of Applied Measurement, 4, pp.386~422 (2003)
- Ruth, L., "Designing writing tasks for the assessment of writing" (1988), Ablex Publishing
- Peterson, S. S, "Sixth-grade teachers’ written comments on student writing: genre and gender influences", Written Communication, 23(1), pp.36~62 (2006)
- Peterson, S., "Evaluation and teachers’ perceptions of gender in sixth-grade student writing", Research in the Teaching of English, 33(2), pp.181~208 (1998)
- Roulis, E., "Gendered voice in composing, gendered voice in evaluating：Gender and the assessment of writing quality, in Composing social identity in written language", pp.151~183 (1995), Lawrence Erlbaum Associates
- Spandel, V, "Writing Assessment, in A Handbook for Student Performance Assessment in an Era of Restructuring" (1996), ASCD
- Weigle, S. C., "Using FACETS to model rater training effects", Language Testing, 15(2), pp.263~287 (1998)
- Wright, B. D, "Measuring the impact of judge severity on examination scores", Applied Measurement in Education, 3, pp.331~345 (1990)
- Wright, B.D, "Rating Scale Analysis" (1982), MESA Press