A Study of the Criteria Design and Application to the Criterion-Referenced Test of English
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The paper investigates whether teacher raters differentiating L1 nativeness affect their rating patterns in the English speaking assessment and, if so, what criteria cause the difference. More specifically, this study examines (1) the particular characteristics in setting up the criteria design between the two groups (native English teacher raters vs. non-native Korean English teacher raters), (2) the rating patterns of both groups’ actual assessment as the criteria application, and (3) test-takers’ response to the teacher raters’ evaluation of both groups. The results show that teacher raters of the two groups approach the criteria design and application differently in a subtle way, which results in different assessment patterns, and finally has an influence on the test-takers’ satisfaction levels. There are some similarities and differences between both groups. In the survey of the criteria design, both groups set up the topic performance and expression abilities as the most important constructs in the assessment, whereas the accuracy of grammar is considered less important. However, the two groups present different sets of opinions on interactions, expression ability, and relevant knowledge. In the survey for the test-takers, the respondents show no statistically significant difference between the two groups. However, differences have been found in some constructs. Finally, the criteria design by the effect of raters’ L1 nativeness affect the raters’ application patterns, in which these characteristics cause various responses of the test-takers involved.
Abstract
The paper investigates whether teacher raters differentiating L1 nativeness affect their rating patterns in the English speaking assessment and, if so, what criteria cause the difference. More specifically, this study examines (1) the particular characteristics in setting up the criteria design between the two groups (native English teacher raters vs. non-native Korean English teacher raters), (2) the rating patterns of both groups’ actual assessment as the criteria application, and (3) test-takers’ response to the teacher raters’ evaluation of both groups. The results show that teacher raters of the two groups approach the criteria design and application differently in a subtle way, which results in different assessment patterns, and finally has an influence on the test-takers’ satisfaction levels. There are some similarities and differences between both groups. In the survey of the criteria design, both groups set up the topic performance and expression abilities as the most important constructs in the assessment, whereas the accuracy of grammar is considered less important. However, the two groups present different sets of opinions on interactions, expression ability, and relevant knowledge. In the survey for the test-takers, the respondents show no statistically significant difference between the two groups. However, differences have been found in some constructs. Finally, the criteria design by the effect of raters’ L1 nativeness affect the raters’ application patterns, in which these characteristics cause various responses of the test-takers involved.
참고문헌 (20)
- Bachman, L., "Investigating variability in tasks and rater judgments in a performance test of foreign language speaking", Language Testing, 12(2), pp.238~256 (1995)
- Barnwell, D., "Naive native speakers and judgments of oral proficiency in Spanish", Language Testing, 6, pp.152~163 (1989)
- Bonk, W., "A many-facet Rasch analysis of the second language group oral discussion task", Language Testing, 20(1), pp.89~110 (2003)
- Brown, A., "The effect of rater variables in the development of an occupation-specific language performance test", Language Testing, 12, pp.1~15 (1995)
- Brown, A., "Getting the balance right: Criteria in integrated speaking tasks", Paper presented at the 24th Language Testing Research Colloquium (2002)
- Elder, C., "Language testing and English as an international language", Australian Review of Applied Linguistics, 31(3), pp.34.1~34.11 (2008)
- Hinkel, E., "Native and non-native speakers’ pragmatic interpretations of English texts", TESOL Quarterly, 28(2), pp.353~376 (1994)
- Hudson, T., "The companion to language assessment", pp.561~577 (2014), John Wiley & Sons
- 강석한, "A Study of the Korean English Teachers’ Rating Tendency to the Oral English Proficiency Tests", 영어영문학, 21(4), pp.237~264 (2016), 미래영어영문학회
- 강석한, "A Comparative Study on Criteria and Tasks in Korean English Speaking Assessment by Native and Non-native Raters", 어학연구, 48(2), pp.241~262 (2012), 언어교육원
- Kobayashi, T., "Native and nonnative reactions to ESL compositions", TESOL Quarterly, 26(1), pp.81~112 (1992)
- 이춘희, "Improving inter-rater reliability in oral proficiency test at college level", 현대영어영문학, 54(1), pp.367~387 (2010), 한국현대영어영문학회
- Lincare, M., "Many-facet Rasch measurement" (1989), MESA Press
- Linacre, M., "Facets Rasch measurement computer program" (2014), Winsteps.com
- Lumley, T., "Rater characteristics and rater bias: Implication for training", Language Testing, 12, pp.54~71 (1995)
- Luoma, S., "Assessing speaking" (2004), Cambridge University Press
- Popham, J., "Criterion-referenced measurement" (1978), Prentice Hall
- Shohamy, E., "The effect of raters’ background and training on the reliability of direct writing tests", The Modern Language Journal, 76(1), pp.27~33 (1992)
- Weigle, C., "Using FACETS to model rater training effects", Language Testing, 15, pp.263~287 (1998)
- Zhang, Y., "Judgements of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs?", Language Testing, 28(1), pp.31~50 (2011)