Meta-Analysis of Inter-Rater Agreement and Discrepancy Between Human and Automated English Essay Scoring

키워드

참고문헌 (60)

1. Construct validity of e-rater in scoring TOEFL essays, "Attali, Y.", ETS, ~, pp.ETS RR-07-21 (2007)
2. Attali, Y., "Automated essay scoring with e-rater® V.2", Journal of Technology, Learning, and Assessment, 4(3), pp.1~31 (2006)
3. Attali, Y., "Performance of a generic approach in automated essay scoring", The Journal of Technology, Learning, and Assessment, 10(3), pp.1~15 (2010)
4. Ben-Simon, A., "Toward more substantively meaningful automated essay scoring", The Journal of Technology, Learning, and Assessment, 6(1), pp.1~17 (2007)
5. Bishara, A. J., "Testing the significance of a correlation with nonnormal data : Comparison of Pearson, Spearman, transformation, and resampling approaches", Psychological Methods, 17(3), pp.399~417 (2012)
6. Blood, I., "Automated essay scoring : A literature review", Studies in Applied Linguistics and TESOL, 11(2), pp.40~64 (2011)
7. Borenstein, M., "Introduction to Meta-Analysis" (2009), John Wiley & Sons
8. Borenstein, M., "A basic introduction to fixed-effect and random-effectsmodels for meta-analysis", Research Synthesis Methods, 1(2), pp.97~111 (2010)
9. A study of the use of the e-rater scoring engine for the analytical writing measure of the GRE revised general test, "Breyer, F. J.", ETS, ~, pp.ETS RR-14-24 (2014)
10. Bridgeman, B., "Comparison of human and machine scoring of essays : Differences by gender, ethnicity, and country", Applied Measurement in Education, 25(1), pp.27~40 (2012)
11. Brown, G. T. L., "Accuracy in the scoring of writing : Studies of reliability and validity using a New Zealand writing assessment system", Assessing Writing, 9, pp.105~121 (2004)
12. Burry-Stock, J. A., "Rater agreement indexes for performance assessment", Educational and Psychological Measurement, 56(2), pp.251~262 (1996)
13. Burstein, J., "Automated Essay Scoring for nonnative English speakers", Proceedings of the ACL99 Workshop on Computer-Mediated Language Assessment and Evaluation of Natural Language Processing, pp.68~75 (1999), Association for Computational Linguistics
14. Burstein, J., "Computer analysis of essays", Proceedings of the NCME Symposium on Automated Scoring (1998)
15. Beyond essay length: Evaluating e-rater®’s performance on TOEFL® essays, "Chodorow, M.", ETS, ~, pp.TOEFL Report 73, RR-04-04 (2004)
16. Cohen, J., "A power primer", Psychological Bulletin, 112(1), pp.155~159 (1992)
17. Cohen, Y., "The effect of specific language features on the complexity of systems for automated essay scoring", Paper presented at the 29th Annual Conference of the International Association for Educational Assessment (2003)
18. Deane, P., "On the relation between automated essay scoring and modern views of the writing construct", Assessing Writing, 18, pp.7~24 (2013)
19. Deane, P., "Automated essay scoring in innovative assessments of writing from sources", The Journal of Writing Assessment, 6(1), pp.1~15 (2013)
20. Dikli, S., "An overview of automated scoring of essays", Journal of Technology, Learning, and Assessment, 5(1), pp.1~36 (2006)
21. Doğan, A., "Linguistics, culture and identity in foreign language education", pp.277~288 (2014), IBU Publications
22. "Robumeta: Robust Variance Meta-Regression (version 1.6)" (2015)[URL]
23. Foltz, P. W., "The intelligent essay assessor : Applications to educational technology", Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 1(2), pp.939~944 (1999)
24. Hearst, M. A., "The debate on automated essay grading", IEEE Intelligent Systems, 15(5), pp.22~37 (2000)
25. Higgins, J. P. T., "Measuring inconsistency in meta-analyses", British Medical Journal, 327, pp.557~560 (2003)
26. Hussein, M., "Automated language essay scoring systems:A literature review", PeerJ Computer Science, 5, pp.e208~ (2019)
27. Ifenthaler, D., "Hanboock of open, distance and digital education", pp.1~15 (2022), Springer Nature Singapore
28. Jeong, H., "The level of collapse we are allowed : Comparison of different response scales in Safety Attitudes Questionnaire", Biometrics & Biostatistics International Journal, 4(4), pp.128~134 (2016)
29. 진경애, "Exploring Automated Scoring of Korean High School Students’ English Composition", 멀티미디어 언어교육, 15(1), pp.11~29 (2012), 한국멀티미디어언어교육학회[DOI]
30. Ke, Z., "Automated essay scoring: A survey of the state of the art", Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pp.6300~6308 (2019)
31. Klebanov, B. B., "Automated evaluation of writing–50 years and counting", Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.7796~7810 (2020), Association for Computational Linguistics
32. Koul, R., "Comparing several human and computerbased methods for scoring concept maps and essays", Journal of Educational Computing Research, 32(3), pp.227~239 (2005)
33. Landauer, T. K., "Proceedings of the 19th annual meeting of the Cognitive Science Society", pp.412~417 (1997), Erlbaum
34. LeBreton, J. M., "Answers to 20 questions about interrater reliability and interrater agreement", Organizational Research Methods, 11(4), pp.815~852 (2008)
35. Li, Z., "The role of automated writing evaluation holistic scores in the ESL classroom", System, 44, pp.66~78 (2014)
36. Page, E. B., "The imminence of grading essays by computer", The Phi Delta Kappan, 47(5), pp.238~243 (1966)
37. Page, E. B., "The computer moves into essay grading : Updating the ancient test", The Phi Delta Kappan, 76, pp.561~565 (1995)
38. Powers, D. E., "Stumping e-rater : Challenging the validity of automated essay scoring", Computers in Human Behavior, 18, pp.103~134 (2002)
39. (2016)[URL]
40. Ramesh, D., "An automated essay scoring system : A systematic literature review", Artificial Intelligence Review, 55, pp.2495~2527 (2022)
41. Ramineni, C., "Validating automated essay scoring for online writing placement", Assessing Writing, 18, pp.40~61 (2013)
42. Ramineni, C., "Automated essay scoring : Psychometric guidelines and practices", Assessing Writing, 18(1), pp.25~39 (2013)
43. Evaluations of automated scoring systems in practice, "Rotou, O.", ETS, ~, pp.Research Report No. RR-20-10 (2020)
44. Rudner, L. M., "An evaluation of the IntelliMetric essay scoring system", Journal of Technology, Learning, and Assessment, 4(4), pp.1~22 (2006)
45. Scammacca, N., "Meta-analysis with complex research designs : Dealing with dependence from multiple measures and multiple group comparisons", Review of Educational Research, 84(3), pp.328~364 (2014)
46. Schmidt, F. L., "Fixed-versus random-effects models in metaanalysis : Model properties and an empirical comparison of differences in results", British Journal of Mathematical and Statistical Psychology, 62, pp.97~128 (2009)
47. Shermis, M. D., "State-of-the-art automated essay scoring : Competition, results, and future directions from a United States demonstration", Assessing Writing, 20, pp.53~76 (2014)
48. Shermis, M. D., "Automated essay scoring: A cross disciplinary perspective" (2003), Lawrence Erlbaum
49. Shermis, M. D., "Trait ratings for automated essay grading", Educational and Psychological Measurement, 62(1), pp.5~18 (2002)
50. Comparing computerized and human scoring of students'essays, "Sireci, S. G.", School of Education, University of Massachusetts, ~, pp.Laboratory of Psychometric and Evaluative Research Report (No. 354) (2000)
51. Streeter, L., "The credible grading machine:Automated essay scoring in the DOD", Proceedings of the Interservice, 2002, ~[DOI]
52. Uzun, K., "Home-grown automated essay scoring in the literature classroom : A solution for managing the crowd?", Contemporary Educational Technology, 9(4), pp.423~436 (2018)
53. Valenti, S., "An overview of current research on automated essay grading", Journal of Information Technology Education, 2, pp.319~330 (2003)
54. Viechtbauer, W., "Conducting meta-analyses in R with the metafor package", Journal of Statistical Software, 36(3), pp.1~48 (2010)
55. Wang, J., "Automated essay scoring versus human scoring: A comparative study", The Journal of Technology, Learning, and Assessment, 6(2), pp.1~29 (2007)
56. Warschauer, M., "Automated writing evaluation : Defining the classroom research agenda", Language Teaching Research, 10(2), pp.157~180 (2006)
57. Validation of Automated Scores of TOEFL iBT Tasks against Nontest Indicators of Writing Ability, "Weigle, S. C.", ETS, ~, pp.TOEFL iBT Research Report -15, RR-1-24 (2011)
58. Williamson, D. M., "A framework for evaluation and use of automated scoring", Educational Measurement : Issues and Practice, 31(1), pp.2~13 (2012)
59. Yang, Y., "A review of strategies for validating computer-automated scoring", Applied Measurement in Education, 15(4), pp.391~412 (2002)
60. Yun, J., "The impact of rater variability on relationships among different effect-size indices for inter-rater agreement between human and automated essay scoring", doctor, pp.2017

관련 논문AI추천