국고보조사업 지원 대상 선정평가 신뢰성 개선: 급내상관계수와 표준점수의 활용
Improving Reliability of the Section Evaluation of Government Subsidy Program Support Targets Using Intraclass Correlation Coefficient(ICC) and Standard Score
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
(Purpose) Subsidy targets for a government subsidized project are mostly selected based on the results of written evaluation and on-site evaluation by external experts. However, there is a risk that the priorities of this evaluation change depending on the subjective propensity of evaluators (e.g. generous, strict) and the reliability of the evaluation get lowered. Measuring reliability of evaluation and correcting evaluators’ subjective propensity are, therefore, needed. This study suggests the intraclass correlation coefficient (ICC) as a method to measure the reliability of the selection evaluation of subsidy targets in government subsidized projects, and proposes the use of standardized scores to increase the evaluation reliability. (Design/methodology/approach) ICC and standardized scores are identified in theoretical analysis, and then the reliability of the evaluation scores of the 2022 support targets of the Fishing Village New Deal Project, a government subsidized project, is analyzed using ICC. Lastly the rankings are recalculated using standardized score. (Findings) Written evaluation reliability was low, with the ICC of less than 0.5, and on-site evaluation reliability was also not good with the ICC of less than 0.75. When using standardized scores, 14% of the ports selected by raw scores was changed. As a result of ICC analysis on the standardized score, it was found that ICC increased. This means that when the standardized score is used, the reliability of the evaluation score increase by reducing the statistical error. (Research implications or Originality) This study suggests methods to verify and increase the reliability of the selection evaluation of the support target of government subsidized projects to improve the performance.
Abstract
(Purpose) Subsidy targets for a government subsidized project are mostly selected based on the results of written evaluation and on-site evaluation by external experts. However, there is a risk that the priorities of this evaluation change depending on the subjective propensity of evaluators (e.g. generous, strict) and the reliability of the evaluation get lowered. Measuring reliability of evaluation and correcting evaluators’ subjective propensity are, therefore, needed. This study suggests the intraclass correlation coefficient (ICC) as a method to measure the reliability of the selection evaluation of subsidy targets in government subsidized projects, and proposes the use of standardized scores to increase the evaluation reliability. (Design/methodology/approach) ICC and standardized scores are identified in theoretical analysis, and then the reliability of the evaluation scores of the 2022 support targets of the Fishing Village New Deal Project, a government subsidized project, is analyzed using ICC. Lastly the rankings are recalculated using standardized score. (Findings) Written evaluation reliability was low, with the ICC of less than 0.5, and on-site evaluation reliability was also not good with the ICC of less than 0.75. When using standardized scores, 14% of the ports selected by raw scores was changed. As a result of ICC analysis on the standardized score, it was found that ICC increased. This means that when the standardized score is used, the reliability of the evaluation score increase by reducing the statistical error. (Research implications or Originality) This study suggests methods to verify and increase the reliability of the selection evaluation of the support target of government subsidized projects to improve the performance.
참고문헌 (30)
- Capano, G., "Do policy instruments matter? Governments’ choice of policy mix and higher education performance" (2019), Western Europe Cambridge University Press
- Capano, Giliberto, "The Knowns and Unknowns of Policy Instrument Analysis: Policy Tools and the Current Research Agenda on Policy Mixes", pp.1~13 (2020), SAGE Open
- Cha, J. S., "An Analitical Review of Interrater Reliability &Agreement", Business Research, 23, pp.75~102 (1994)
- Chun, Y., "Diversity of Policy Tools: Issues and Assessment of Tool Type Classification", Government Study, 3(4), pp.259~295 (2007)
- Fleiss JL., "Statistical Methods for Rates and Proportions" (2003), John Wiley and Sons
- Results-oriented Government Practices that can help Enhance and Sustain Collaboration among Federal Agencies, "Government Acountability Office", ~ (2005)
- Helgøy, Ingrid, "Policy Tools and Institutional Change:Comparing Education Policies in Norway, Sweden and England", Journal of Public Policy, 26(2), pp.141~165 (2006)
- Jeong, Jung-Gil, "Policy Theory" (2020), Daemyung Publishing House
- Kong, Kyoung Ae, "Statistical Methods: Reliability Assessment and Method Comparison", Ewha Medical Journal, 40(1), pp.9~16 (2017)
- 김난영, "Determinants and Implications for Creativity Education in Korea : Policy Tools, Street-level Administration, and Implementation Conditions", Journal of Policy Studies, 36(2), pp.71~88 (2021), 서울대학교행정대학원
- Koo, Terry K., "A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research", Journal of Chiropractic Medicine, pp.1~67 (2016)
- 이재창, "조현병 환자가 시행한 주의력 네트워크 검사 점수의 검사-재검사 신뢰도", 정신신체의학, 25(2), pp.210~217 (2017), 한국정신신체의학회
- Lee, Wonhee, "Financial Administration" (2020), Daeyoung Cultural History
- 임재훈, "국고보조금이 기초자치단체 국고보조사업 예산편성에 미치는 영향 : Fungibility 가설을 중심으로", 지방행정연구, 34(3), pp.059~088 (2020), (재) 한국지방행정연구원[DOI]
- McGraw KO, "Forming inferences about some intraclass correlation coefficients", Psychol Methods, pp.30~46 (1996), National Treasury Subsidy Integrated Management Guidelines
- 민대기, "원점수 순위와 표준점수 순위 비교를 통한 표준화 방법비교", 한국데이터정보과학회지, 23(1), pp.113~120 (2012), 한국데이터정보과학회
- Ministry of Economy and Finance, "2022 Budget and Fund Management Plan Implementation Guidelines", ~ (2022)
- Ministry of Education, "Public operation management manual for improving fairness and transparency of financial support projects", ~ (2020)
- Ministry of Environment, "Smart Green City Business Competition Guidelines", ~ (2020)
- Ministry of Oceans and Fisheries, "2022 Fishing Village New Deal 300Project Contest Guidelines", ~ (2021)
- Ministry of the Interior and Safety, "In 2023, the state and local governments jointly promote a government subsidy project worth 122.1trillion won, press release. National Treasury Subsidy Integrated Management Guidelines", ~ (2023)
- 박철용, "면접점수 표준화 방법 모의실험 비교", 한국데이터정보과학회지, 22(2), pp.189~196 (2011), 한국데이터정보과학회
- Salamon, L. M., "Rethinking Public Management: Third-party Government and the Changing Forms of Government Action", Public Policy, 29(3), pp.256~275 (1981)
- Salamon, L. M., "The Tools of Government" (2002), Oxford University Press
- Sohn, Hosung, "Experimental and quasi-experimental policy evaluation I" (2022), ParkYoungsa
- 송방원, "우선순위를 이용한 점수 표준화 방법 연구", 한국정보통신학회논문지, 19(10), pp.2500~2506 (2015), 한국정보통신학회[DOI]
- "eNara help"[URL]
- "Fishing Village Regeneration Management System"[URL]
- Ministry of Oceans and Fisheries, "Regulations on government subsidy management", ~
- "Subsidy Management Act", ~