본문 바로가기
KERD
논문KCI 등재

상황판단검사의 채점용 답 결정방식과 채점방식이 타당도에 미치는 영향: 왜곡여부에 따른 비교 연구

Comparison of validities for scoring keys and scoring algorithms in situational judgment test: the influence of faking

한영석|김명소|김의수
한국산업및조직심리학회 (Korean Society for Industrial and Organizational Psychology)|2011.02|한국심리학회지: 산업 및 조직 (Korean Journal of Industrial and Organizational Psychology)|제24권 제1호|pp. 231-255|DOI 10.24230/ksiop.24.1.201102.231

0

조회

0

다운로드

4

인용

0

참고문헌

키워드

상황판단검사채점방식응답왜곡faking채점용 답 결정Situational judgment testsscoring keyscoring algorith
주제분류심리과학

초록

본 연구에서는 상황판단검사(SJT)에서 피험자가 의도적으로 자신의 점수를 높일 수 있는지 즉, 긍정적 응답왜곡(faking good)이 가능한지를 확인하고, 만약 응답왜곡이 가능하다면 검사의 준거관련 타당도와 증분 타당도에 어떠한 영향을 미치는지 알아보고자 하였다. 이와 더불어 상이한 채점용 답 결정방식(전문가 합의, 응답자 평균, 경험적 방법)과 채점방식(Pick most 방식, 시나리오 채점방식, Best-Worst 채점방식)을 조합하여 사용하였을 경우, 응답왜곡의 정도와 검사의 준거관련 타당도 및 증분 타당도에 미치는 영향이 달라 질 수 있는지를 탐색적으로 살펴보고자 하였다. 이를 위해 국내 A대학의 리더선발 프로그램에 지원한 2학년 학생 110명과 B학과의 재학생 129명을 대상으로 적성검사, 성격검사, 상황판단 검사를 순차적으로 실시하였다. 적성검사와 성격검사는 두 집단 모두 동일한 절차와 방법으로 시행하였으나, 상황판단검사의 경우 제한된 수의 리더를 선발하기 위한 리더선발 프로그램에 지원한 학생들은 자연적으로 긍정적 응답왜곡이 발생하는 왜곡집단으로 간주하였다. B학과의 재학생들의 경우 검사의 목적이 자기진단임을 강조하고 별도의 지시문으로 솔직한 응답을 요구하였기 때문에 솔직집단으로 간주하였다. 또한 A대학의 재학생 78명을 사용하여 응답자 평균, 경험적 채점용 답을 설정하였으며, B학과의 재학생 중 성적 우수자, 리더 경험자, 대학원 1학년생을 전문가로 설정하여 전문가 기반 채점용 답으로 활용하였다. 수집된 자료들을 바탕으로 각각의 채점용 답 결정방식과 채점방식을 조합하여 솔직집단과 왜곡집단 모두에서 9개의 독립된 상황판단검사점수를 산출하였으며, 이를 바탕으로 각각의 준거관련 타당도와 증분 타당도를 비교하였다. 그 결과, 독립적으로 산출된 9개의 상황판단검사 점수 모두에서 왜곡집단이 솔직집단에 비하여 유의미하게 높은 검사점수를 보이는 것으로 나타났으며, 준거관련 타당도 또한 솔직집단이 왜곡집단보다 더 큰 것으로 나타났다. 증분 타당도의 경우 과업수행준거와 맥락수행 준거에 대한 결과가 상이하게 나타났으며, 맥락수행 준거에서만 9개의 상황판단검사 모두 솔직집단이 왜곡집단 보다 더 큰 증분적 설명량을 보였다. 이러한 결과를 토대로 본 연구의 의의와 제한점에 대하여 논의하였다.

Abstract

The purpose of the present study was to examine the fakability of the situational judgment test. Specifically, the study was focused on the following questions; (1) whether participants are able to fake their answers on the situational judgment test in the real situation of selection, (2) whether faking influences the criterion-related validity of the situational judgment test and its incremental validity over cognitive and personality tests, and (3) whether the combination of different scoring key(SME consensus, average in response, and empirical keying) and different scoring algorithm(scenario, Best-Worst, and Pick most) has influence on the degree of fakability as well as both criterion-related validity and incremental validity of the situational judgment test. 110 students who applied to the leadership program were considered the faking group, while 129 students of B department at A university were considered the honest group. The members of both groups completed a cognitive test, a personality questionnaire and a situational judgment test. Only for the situational judgment tests, each group was asked to respond as instructed. Another group of 78 students of A university participated in the survey to develop two scoring key(empirical, average in response keying). SME consensus key was developed by 9 SMEs(5 undergraduate students with leadership and good GPA, 4 graduate students). And then 9 situational judgment scores were produced independently. Results indicated that the all scores of students in the faking group were significantly higher than those of students in the honest group. Furthermore, criterion-related validity of the situational judgement test in the honest group was higher than that of the faking group for both task performance and contextual performance. While faking had negative effects on the criterion-related validity for both criteria of performance, incremental validity of the situational judgement test in the honest group was higher than that of the faking group only for the contextual criteria. Finally, the limitation and future direction of the present study were discussed.

참고문헌 (56)

  1. 강민우, "지시문과 채점방식에 따른 상황판단검사의 타당도 비교", 한국심리학회지: 산업 및 조직, 18(3), pp.547~565 (2005), 한국산업및조직심리학회
  2. 김 명 소, "성격검사의 형식이 응답왜곡(faking)에 미치는 효과:Normative 형식과 ipsative 형식의 비교", 한국심리학회지: 산업 및 조직, 19(3), pp.371~393 (2006), 한국산업및조직심리학회
  3. 박동건, "전기자료(biodata) 문항의 가중치 부여 체계간의 타당도 연구: 분석집단 크기에 따른 비교연구", 한국심리학회지: 산업 및 조직, 14(1), pp.101~113 (2001)
  4. 연합뉴스, "신입사원 채용 시 인, 적성 검사 강화", ~ (2005)
  5. 이상철, "지필형 상황판단검사에 대한 비평적 고찰", 한국심리학회지: 산업 및 조직, 16(3), pp.129~154 (2003), 한국산업및조직심리학회
  6. 유태용, "한국판 HEXACO 성격검사의 구성타당화 연구", 한국심리학회지: 사회및성격, 18(3), pp.61~75 (2004), 한국사회및성격심리학회
  7. Arthur, W., "The importance of distinguishing between constructs and methods when comparing predictors in personnel selection research and practice", Journal of Applied Psychology, 93, pp.435~442 (2008)
  8. Bergman, M. E., "Scoring situational judgement tests: Once you get the data, Your trouble begin", International Journal of Selection and assessment, 14, pp.223~235 (2006)
  9. Bess, T. L., "Exploring a dimensionality of situational judgment: task and contextual knowledge", Paper presented at the 17th Annual Conference of the Society for Industrial and Organizational Psychology (2002)
  10. Chan, d., "Situatioanl judgment Tests, in The blackwell handbook of Personnel selection", pp.219~242 (2005), The blackwell Publisher
  11. Clevenger, J., "Incremental validity of situational judgment tests", Journal of Applied Psychology, 86, pp.410~417 (2001)
  12. Clevenger, J. p, "Examining the relationship beetween job knowledge and situational judgment performance", Paper presented at the 15th Annual Conference of the Society of Industrial and Organizational Psychology (2000)
  13. Cureton, E. E, "Validity, reliability, and baloney", Educational Measurement, 5, pp.115~124 (1950)
  14. Douglas, E. F, "The validity of non-cognitive measure decays when applicants fake", Paper presented at the annual conference of the Academy of Management (1996)
  15. Dwight, S. A, "An assessment of the difference of warning applicants not to fake", 박사, pp.1999
  16. Ellingson, J. E., "Investigating the Influence of social desirability on personal factor structure", Journal of Applied Psychology, 86, pp.122~133 (2001)
  17. Hogan, J. B, "Empirical keying of background data measures, in Biodata handbook: Theory, research, and use of biographical information in selection and performance predictions", pp.69~). (1994), Consulting Psychology Press
  18. Hooper, A. C., "Operational threat to the use of SJTs: faking, coaching, and retesting issue, in Situational Judgment Tests: Theory, Measurement and Application", pp.205~232 (2006), Lawrence Erlbaum Associates
  19. Hough, L. M, "Effects of intentional distortion in personality measurement and evaluation of suggested palliatives", Human Performance, 11, pp.209~244 (1998)
  20. Hough, L. M., "Detrerminats, detection, and amelioration of adverse impact in personnel selection procedure: Issue, evidence, and lessons learned", Internal Journal of Selection and assessment, 9, pp.152~194 (2001)
  21. Juraska, S. E., "Faking situational judgment: A tests of Conflicts Resolution Skills Assessment", Paper presented at the 16th annual conference of the Society for Industrial and Organizational Psychology (2001)
  22. Knapp, D. J., "Performance assessment for a population of jobs, in Exploring the limits in personnel selection and classfication" (2001), Lawrence Erlbaum Associates
  23. Latham., G. P., "Do people do what they say? Further studies of the situational interview", Journal of Applied Psychology, 69, pp.569~573 (1984)
  24. Latham., G. P, "The situational interview", Journal of applied psychology, 65, pp.422~427 (1980)
  25. Lievens, F., "Situational judgment tests: a review of recent research", Personnel Review, 37, pp.426~411 (2008)
  26. McDaniel, L. A, "Use of situational judgement test to predict job performance: A clarification of the literature", Journal of Applied psychology, 86, pp.812~821 (2001)
  27. McDaniel, M. A., "Situational Judgments, response instructions, and validity: A meta analysis", Personnel psychology, 60, pp.63~91 (2003)
  28. McDaniel., M. A, "Situational judgment tests: a review of practice and construct assessed", International Journal of Selection and assessement, 9, pp.103~113 (2001)
  29. McFarland, L. A., "Variance in faking across non-cognitive measure: Effects on faking behavior and test measurement properties", Journal of Personality Assessment, 78l, pp.348~369 (2000)
  30. Motowidlo, S. J, "Further studies of low-fidelity simulation in the form of a situational inventory", Journal of Occupational and Organizational Psychology, 66, pp.337~344 (1993)
  31. Motowidlo. S. J., "An alternative selection procedure: The low-fidelity simulation", Journal of Applied psychology, 75, pp.640~647 (1990)
  32. Motowidlo. S. J., "Applied measurement method in industrial Psychology, in Applied measurement method in industrial psychology", Davies-Black, pp.241~260 (1997)
  33. Mumford, T. V., "Methodology review: Principles, procedures, and findings in the application of background data measures", Applied Psychological Measurement, 11, pp.1~31 (1987)
  34. Ones, D. S., "The effects of social desirability and faking on personality and integrity assessment for personnel selection", Human Performance, 11, pp.245~269 (1998)
  35. Oswald, F. L., "Developing a biodata measure and situational judgment inventory as predictors of college student performance", Journal of Applied Psychology, 89, pp.187~208 (2004)
  36. Paullin, C., "Comparing the validity of rationally-derived and empirically-derived scoring keys for situational judgment inventory", Paper presented at the 16th Annual Conference of the Society for Industrial and Organizational Psychology (2001)
  37. Peeters, H., "Situational judgment tests and their predictiveness of college student's success: the influence of faking", Educational and psychological Measurement, 65, pp.70~89 (2005)
  38. Ployhart, R. E., "Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests", International Journal of Selection and Assessment, 11, pp.1~16 (2003)
  39. Ployhart, R. E., "The relative importance of procedure and distributive justice in determining applicant's reactions", Journal of applied psychology, 83, pp.3~16 (1998)
  40. Ployhart, R. E., "Organizational staffing: Contemporary practice and theory" (2005), Lawrence Erlbaum Associates
  41. Ployhart, R. E, "Web-based and paper-and-pencil testing of applicants in proctored setting: Are personality, biodata, and situational judgment tests comparable", Personnel Psychology, 56, pp.733~752 (2003)
  42. Pulakos, E. D., "An evaluation of two strategies for reducing adverse impact and their effects on criterion-related validity", Human Performance, 9, pp.241~258 (1996)
  43. Reynolds, D. H., "Development, validation and translation of a Professional-level situational judgment Inventory items", Invited presentation to College Board (1999)
  44. Schmidt, D. B, "Susceptibility of SJTs to applicant faking: An examination of applicant and incumbent samples", Paper presented at the 18th Annual Conference of the Society for Industrial and Organizational Psychology (2003)
  45. Shuang-Yueh Pui, "Situational Judgement Tests: A Measurement of Judgement", 박사, Industrial and Organizational Psychology, pp.2007
  46. Smith, K. C., "Criterion and construct validity evidence for a situational judgment measure", Paper presented at the 13th annual conference of the Society for Industrial and Organizational Psychology (1998)
  47. Smith, C. A.,, "Organizational Citizenship Behavior: Its Nature and Antecedents", Journal of Applied Psychology, 68, pp.653~663 (1983)
  48. Wall Street Journal, "Adding Personality to the College Admissions Mix", ~ (2009)
  49. Waugh, G, "Selecting response options and item for situational judgment test. Paper presented as part of following symposium - Understanding and predicting Performance in future jobs", 17th Annual conference of society for industrial and organizational psychology (2002)
  50. Weekley, J. A., "Wideo-based situational testing", Personnel psychology, 50, pp.24~49 (1997)
  51. Weekley, J. A., "Situational judgment: Antecedents and relation ship with performance", Human Performance, 18, pp.81~104 (2005)
  52. Weekley, J. A., "An introduction to situational judgment testing, in Situational judgment tests" (2006), Lawrence Erlbaum Associates
  53. Weekley, J. A., "Personality and situational judgment tests across applicant and incumbent setting: An examination of validity, measurement, and subgrouping differences", Paper presented at the 18th Annual Conference of the Society for Industrial and Organizational Psychology (2003)
  54. Weekley., J. A, "Reliability and validity of situational interview for a sales position", Journal of applied Psychology, 72, pp.484~487 (1987)
  55. Zickar, M. J, "Computer simulation of faking on a personality test", Paper Presented at the annual conference of the Society for Industrial and Organizational Psychology (1997)
  56. Zickar, M. J., "Modeling faking good on personality items: An item-level analysis", Journal of Applied Psychology, 84, pp.551~563 (1999)

관련 논문AI추천

지식형 상황판단검사의 채점용 답(Scoring key) 결정과 채점 방식에 따른 준거관련 타당도 비교 연구박동건, 정현선한국산업및조직심리학회 · 한국심리학회지: 산업 및 조직2009 · 제22권 제1호지시문과 채점방식에 따른 상황판단검사의 타당도 비교강민우, 윤창영, 이순묵한국산업및조직심리학회 · 한국심리학회지: 산업 및 조직2005 · 제18권 제3호상황판단검사 형식을 이용한 정직성검사의 타당화권은아, 이종구한국산업및조직심리학회 · 한국심리학회지: 산업 및 조직2020 · 제33권 제4호성격검사의 형식이 응답왜곡(faking)에 미치는 효과:Normative 형식과 ipsative 형식의 비교김명소, 이헌주한국산업및조직심리학회 · 한국심리학회지: 산업 및 조직2006 · 제19권 제3호지필형 상황판단검사에 대한 비평적 고찰이상철, 이순묵, 조영일한국산업및조직심리학회 · 한국심리학회지: 산업 및 조직2003 · 제16권 제3호업무 시뮬레이션을 활용한 상호평가 기반 논술형 SJT와 직무적성검사의 타당도 비교김송의, 박주용한국산업및조직심리학회 · 한국심리학회지: 산업 및 조직2025 · 제38권 제3호

이 간행물의 다른 논문 (9건)

생산적 노화에 대한 세대간 인식차 및 생산적 노화 척도의 타당성 분석김지현, 강연욱, 이주일, 유경pp. 1-27운전부하등급에 따른 교통사고 위험도의 차이와 자아탄력성의 조절효과이순철, 송훈화pp. 29-50조직몰입이 안전행동에 미치는 효과: 안전 분위기의 조절효과문광수, 오세진, 이재희pp. 51-73집단 내 갈등과 이직의도의 관계에서 직무만족의 매개효과: 한국과 중국 종업원들의 공통점과 차이점이준호, 박지환pp. 75-102자기평가의 정확도와 수행 및 개인특성 간의 관계손영우, 노연희pp. 103-126
원문 보기