5 or 7: Is the Choice So Important in Acceptability Judgment Testing?
1
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
Song, Sanghoun and Oh, Eunjeong. 2016. 5 or 7: Is the Choice So Important in Acceptability Judgment Testing? Korean Journal of Linguistics, 41-3, 449-480. This article concerns whether two different numbers of points on a Likert scale task produces different results in acceptability judgment testing. The most popular numbers of points on response scales are 5 and 7, and there seems to be no clear consensus about which of the two is better and why. As the same goes for experimental syntax studies, the choice of the numbers of points on the scales still remains questionable, though the Likert scale task has been widely employed in acceptability judgment testing. The present study compares two experimental data sets using the same stimuli sentences but with different point scales (5 and 7). It includes 46,356 data points and 506 Korean native speakers participated in the study. The comparison between the 5- and 7-point scale data is made in terms of (a) variance of data points, (b) convergence between the linguists’ judgments and the participants’ judgments, and (c) response time. The comparative analysis reveals that the two different point scales do not yield significantly different results. Yet, it is also observed that there exist pros and cons to both sides. The 7-point scale is more demanding than the 5-point scale to the extent that the middle point is rather scarcely used. On the other hand, the 5-point scale is sloppier than the 7-point scale to the extent that the latter captures the variation in acceptability judgments across the participants slightly better. (Incheon National University & Sangmyung University)
Abstract
Song, Sanghoun and Oh, Eunjeong. 2016. 5 or 7: Is the Choice So Important in Acceptability Judgment Testing? Korean Journal of Linguistics, 41-3, 449-480. This article concerns whether two different numbers of points on a Likert scale task produces different results in acceptability judgment testing. The most popular numbers of points on response scales are 5 and 7, and there seems to be no clear consensus about which of the two is better and why. As the same goes for experimental syntax studies, the choice of the numbers of points on the scales still remains questionable, though the Likert scale task has been widely employed in acceptability judgment testing. The present study compares two experimental data sets using the same stimuli sentences but with different point scales (5 and 7). It includes 46,356 data points and 506 Korean native speakers participated in the study. The comparison between the 5- and 7-point scale data is made in terms of (a) variance of data points, (b) convergence between the linguists’ judgments and the participants’ judgments, and (c) response time. The comparative analysis reveals that the two different point scales do not yield significantly different results. Yet, it is also observed that there exist pros and cons to both sides. The 7-point scale is more demanding than the 5-point scale to the extent that the middle point is rather scarcely used. On the other hand, the 5-point scale is sloppier than the 7-point scale to the extent that the latter captures the variation in acceptability judgments across the participants slightly better. (Incheon National University & Sangmyung University)
참고문헌 (29)
- Alwin, D. F., "Feeling Thermometers Versus 7-Point Scales Which are Better?", Sociological Methods & Research, 25(3), pp.318~340 (1997)
- Chomsky, N., "Language and Mind" (1968), Harcourt Brace Jovanovich
- Cohen, J., "A Coefficient of Agreement for Nominal Scales", Educational and Psychological Measurement, 20(1), pp.37~46 (1960)
- Colman, A. M., "Comparing Rating Scales of Different Lengths: Equivalence of Scores from 5-point and 7-point Scales", Psychological Reports, 80(2), pp.355~362 (1997)
- Cox, E. P. III, "The Optimal Number of Response Alternatives for a Scale: A Review", Journal of Marketing Research, 17(4), pp.407~422 (1980)
- Dawes, J., "Do Data Characteristics Change According to the Number of Scale Points Used? An Experiment Using 5 Point, 7 Point and 10 Point Scales", International Journal of Market Research, 51(1), pp.61~104 (2008)
- Finstad, K., "Response Interpolation and Scale Sensitivity: Evidence against 5-point Scales", Journal of Usability Studies, 5(3), pp.104~110 (2010)
- Green, P. E., "Rating Scales and Information Recovery. How Many Scales and Response Categories to Use?", The Journal of Marketing, 34(3), pp.33~39 (1970)
- Krosnick, J. A., "Handbook of Survey Research", pp.263~314 (2010), Emerald
- Krosnick, J. A., "Survey Measurement and Process Quality", pp.141~164 (1997), John Wiley & Sons
- Krosnick, J. A., "The Optimal Length of Rating Scales to Maximize Reliability and Validity", Stanford University, ~ (2008)
- Krosnick, J. A., "Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys", Applied Cognitive Psychology, 5(3), pp.213~236 (1991)
- Likert, R., "A Technique for the Measurement of Attitudes", Archives of Psychology, 140, pp.1~55 (1932)
- Mathôt, S., "OpenSesame: An Open-Source, Graphical Experiment Builder for the Social Sciences", Behavior Research Methods, 44(2), pp.314~324 (2012)
- Miller, G. A., "The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information", Psychological Review, 63(2), pp.81~97 (1956)
- Neumann, L., "Comparison of Six Lengths of Rating Scales: Students’ Attitude Toward Instruction", Psychological Reports, 48(2), pp.399~404 (1981)
- Nunnally, J. C., "Psychometric Theory" (1978), McGraw-Hill
- Osgood, C. E., "The Measurement of Meaning" (1957), University of Illinois Press
- Peirce, J. W., "PsychoPy – Psychophysics Software in Python", Journal of Neuroscience Methods, 162(1), pp.8~13 (2007)
- Preston, C. C., "Optimal Number of Response Categories in Rating Scales: Reliability, Validity, Discriminating Power, and Respondent Preferences", Acta Psychologica, 104(1), pp.1~15 (2000)
- Revilla, M. A., "Choosing the Number of Categories in Agree-Disagree Scales", Sociological Methods & Research, 43(1), pp.73~97 (2013)
- Royeen, C. B., "Adaptation of Likert Scaling for Use with Children", Occupational Therapy Journal of Research, 5(1), pp.59~69 (1985)
- Rundquist, E. A., "Personality in the Depression" (1936), University of Minnesota Press
- Sauro, J., "Comparison of Three One-Question, Post-Task Usability Questionnaires", Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp.1599~1608 (2009)
- Shaw, M. E., "Scales for the Measurement of Attitudes" (1967)
- Sprouse, J., "A Comparison of Informal and Formal Acceptability Judgments Using a Random Sample from Linguistic Inquiry 2001–2010", Lingua, 134, pp.219~248 (2013)
- Symonds, P. M., "On the Loss of Reliability in Ratings Due to Coarseness of the Scale", Journal of Experimental Psychology, 7(6), pp.456~461 (1924)
- Thurstone, L. L., "Attitudes Can Be Measured", American Journal of Sociology, 33(4), pp.529~554 (1928)
- Viera, A. J., "Understanding Interobserver Agreement: the Kappa Statistic", Family Medicine, 37(5), pp.360~363 (2005)