수용성 판단 실험의 설계 및 통계 분석 처리의 몇 문제- 서열형·연속형 데이터를 중심으로 -
Some issues in designing and analyzing Korean ordinal and continuous acceptability judgment experiments
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Acceptability judgment experiments, often conducted as surveys, offer the advantage of obtaining a large amount of data with minimal time, effort, or cost. However, conducting acceptability judgment experiments involves several considerations. First is the issue related to the sensitivity of result analysis, second is related to the identification of careless responses, and third pertains to the statistical analysis of result data. This study empirically investigates these issues, focusing on ordinal data and continuous data. Sensitivity, represented by statistical power, was assessed across various effect sizes with sample sizes ranging from 5 to 100, each simulated with 1,000 resamplings. Similar to Sprouse & Almeida 2017, for the Likert scale(LS), it was found that a minimum of 9 participants is required for an extra-large effect size, 19 for a large effect size, and 51 for a medium effect size. For the magnitude estimation(ME), in the case of an extra-large effect size, at least 11 participants are needed, 25 for a large effect size, and 75 for a medium effect size. In both methods, for a small effect size, statistical power did not reach 80%. Additionally, relying on crowd-sourced large-scale acceptability judgment experiments, 30 standard sentences (gold standard) for identifying careless responses were provided, consisting of 30 grammatical and 30 ungrammatical sentences. These could be useful for developing practice items and items for identifying careless responders. Moreover, 20 sentences are presented for designing practice items, for participants to grasp a possible range of of acceptability judgments. This is expected to be very helpful in designing acceptability judgment experiments. Finally, we examined various possible statistical analyses for ordinal and continuous data. For ordinal data, there are methods such as standardization for statistical analysis or multivariate cumulative logit analysis using raw values. For continuous data, there are methods such as standardization or natural logarithm transformation for statistical analysis. As an example, for each of the four syntactic phenomena, 200 samples were randomly selected, and statistical analyses were compared. Overall, we suggest that standardizing ordinal data for statistical analysis is preferable.
Abstract
Acceptability judgment experiments, often conducted as surveys, offer the advantage of obtaining a large amount of data with minimal time, effort, or cost. However, conducting acceptability judgment experiments involves several considerations. First is the issue related to the sensitivity of result analysis, second is related to the identification of careless responses, and third pertains to the statistical analysis of result data. This study empirically investigates these issues, focusing on ordinal data and continuous data. Sensitivity, represented by statistical power, was assessed across various effect sizes with sample sizes ranging from 5 to 100, each simulated with 1,000 resamplings. Similar to Sprouse & Almeida 2017, for the Likert scale(LS), it was found that a minimum of 9 participants is required for an extra-large effect size, 19 for a large effect size, and 51 for a medium effect size. For the magnitude estimation(ME), in the case of an extra-large effect size, at least 11 participants are needed, 25 for a large effect size, and 75 for a medium effect size. In both methods, for a small effect size, statistical power did not reach 80%. Additionally, relying on crowd-sourced large-scale acceptability judgment experiments, 30 standard sentences (gold standard) for identifying careless responses were provided, consisting of 30 grammatical and 30 ungrammatical sentences. These could be useful for developing practice items and items for identifying careless responders. Moreover, 20 sentences are presented for designing practice items, for participants to grasp a possible range of of acceptability judgments. This is expected to be very helpful in designing acceptability judgment experiments. Finally, we examined various possible statistical analyses for ordinal and continuous data. For ordinal data, there are methods such as standardization for statistical analysis or multivariate cumulative logit analysis using raw values. For continuous data, there are methods such as standardization or natural logarithm transformation for statistical analysis. As an example, for each of the four syntactic phenomena, 200 samples were randomly selected, and statistical analyses were compared. Overall, we suggest that standardizing ordinal data for statistical analysis is preferable.
참고문헌 (31)
- 김영희, "부정 극성어의 허가 양상", 한글, 240-241, pp.263~297 (1998), 한글학회
- 남길임, "'이다' 구문의 한 유형―양태 구문으로서의 '이다' 구문 연구―", 한글, (259), pp.171~197 (2003), 한글학회
- 박재연, "한국어 양태 어미 연구" (2006), 태학사
- 안희돈, "문장 문법성 판단을 위한 기초 자료 구축" (2020), 국립국어원
- 조용준, "한국어 실험 통사론의 모색", 박사, pp.2015
- 한정한, "‘공부를 하다’ 구문의 정보구조", 한국어학, 11, pp.325~338 (2000), 한국어학회
- Baer, R. A., "Detection of random responding on the MMPI-A", Journal of Personality Assessment, 68, pp.139~151 (1997)
- Beach, D. A., "Identifying the random responder", The Journal of Psychology, 123, pp.101~103 (1989)
- Berry, D. T. R., "Detection of malingering on the MMPI: A meta-analysis", Clinical Psychology Review, 11, pp.585~598 (1991)
- Bošković, Ž., "On the distribution of null complementizers", Linguistic Inquiry, 34(4), pp.527~546 (2003)
- Cohen, J., "Statistical power analysis for the behavioral sciences" (1988), Erlbaum
- Curran, P., "Careless responding in surveys: Applying traditional techniques to organizational settings", Poster presented at the 25th Annual Conference of the Society for Industrial and Organizational Psychology (2010)
- DeRight, J., "I just want my research credit: Frequency of suboptimal effort in a non-clinical healthy undergraduate sample", The Clinical Neuropsychologist, 29, pp.101~117 (2015)
- Garrod, S., "Encyclopedia of Language & Linguistics", pp.251~257 (2006)
- Goodall, G., "The Cambridge handbook of experimental syntax", pp.1~3 (2021), Cambridge University Press
- Gough, H. G., "CPI manual" (1996), Consulting Psychologists Press
- Johnson, J. A., "Ascertaining the validity of individual protocols from web-based personality inventories", Journal of Research in Personality, 39, pp.103~129 (2005)
- Lewis, S., "Aligning grammatical theories and language processing models", Journal of Psycholinguistic Research, 44(1), pp.27~46 (2015)
- Maniaci, M. R., "Caring about carelessness: Participant inattention and its effects on research", Journal of Research in Personality, 48, pp.61~83 (2014)
- Meade, A. W., "Identifying careless responses in survey data", Psychological Methods, 17, pp.437~455 (2012)
- Myers, J., "The design and analysis of small-scale syntactic judgment experiments", Lingua, 119, pp.425~444 (2009)
- Nichols, D. S., "Criteria for assessing inconsistent patterns of item endorsement on the MMPI: Rationale, development, and empirical trials", Journal of Clinical Psychology, 45, pp.239~250 (1989)
- Phillips. C., "Oxford handbook of psycholinguistics", pp.739~756 (2007), Oxford University Press
- Schwarz, N., "Self-reports: How the questions shape the answers", American Psychologist, 54, pp.93~105 (1999)
- Sprouse, J, "The Oxford handbook of experimental syntax", pp.3~28 (2023), Oxford University Press
- Sprouse, J., "A program for experimental syntax", Doctor, pp.2007
- Sprouse, J., "A test of the cognitive assumptions of magnitude estimation: Commutativity does not hold for acceptability judgments", Language, 87(2), pp.274~288 (2011)
- Sprouse, J., "Experimental Syntax", Mini-course taught at the Linguistic Society of Korea Linguistic School (2018), Seoul National University
- Sprouse, J., "Design sensitivity and statistical power in acceptability judgment experiments", Glossa, 2(1), pp.14~ (2017)
- Sprouse, J., "A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001-2010", Lingua, 134, pp.219~248 (2013)
- Stevens, S. S., "On the psychophysical law", Psychological Review, 64, pp.153~181 (1957)