팀기반 온라인 토론의 내용분석을 지원하는 대시보드 개발 및 사용성 평가
Dashboard Development and Usability Test to Support Contents Analysis of Team-based Online Discussion
1
조회
0
다운로드
9
인용
0
참고문헌
키워드
초록
학습자의 고차원적 사고능력을 향상시킬 수 있는 토론활동에서 산출되는 학습데이터를 분석하여 그 결과를 시각화하여 제시하는 연구들이 이루어지고 있다. 대시보드 개발 관련 선행 연구에서는 토론활동에 대한 참여도 및 상호작용 중심으로 이루어져 학생들에게 토론내용에 대한 피드백을 제공하는 데 제한점이 있어 왔다. 이에 이 연구에서는 온라인 토론학습환경에서 학습자가 토론내용에 대한 성찰활동을 원활히 할 수 있도록 토론내용분석을 지원하는 대시보드를 개발하고, 이에 대한 사용성 평가와 사용자의 경험 평가를 수행함으로써 대시보드 활용 가능성을 확인해 보고자 하였다. 이 연구는 산출물 및 도구를 설계하고 개발하는 설계․개발 연구 방법의 유형 1을 적용하여 선행연구에서 제안된 시각화 가이드라인을 기반으로 토론활동의 내용분석을 지원하는 ‘중심단어 대시보드’와 ‘메시지유형 대시보드’를 개발하였고, 대시보드의 주요 사용자인 학습자와 교수자를 대상으로 사용성 및 사용자 경험 평가를 실시하였다. 중심단어에 대한 사용성 평가 결과 학습자와 교수자 모두 만족감 항목에 가장 점수가 높은 것으로 나타났다. 사용자 경험 평가 결과 학습자와 교수자 모두 ‘참신성’ 항목에 대한 점수가 가장 높은 것으로 나타났고, 학습자는 교수자에 비해 상대적으로 ‘의존성’ 항목에 대한 사용자 경험 평균 점수가 낮은 것으로 분석되었다. 메시지유형에 대한 사용성 평가 결과 학습자는 만족감, 유용성, 학습용이성, 사용용이성 순으로 평균 점수가 높게 나타났고, 교수자는 학습용이성, 사용용이성, 만족감, 유용성 순으로 나타났다. 사용자 경험 평가 결과 학습자와 교수자 모두 ‘효율성’, ‘자극성’ 항목의 등급이 높은 것으로 나타났고, 상대적으로 학습자는 ‘매력성’, ‘명확성’, ‘의존성’ 항목에 대해 평균 점수가 낮게 도출되었다. 두 대시보드에 대한 사용성 평가와 사용자 경험 평가 결과를 기반으로 온라인 토론 내용분석을 지원하는 대시보드의 활용 가능성과 토론내용 대시보드의 교수설계 관련 시사점에 대해 논의하였다.
Abstract
The studies of analyzing the learning data generated from discussion activities that can enhance the learners’ higher order thinking abilities and visualizing the results of learning analysis are conducted by many researchers. However, previous studies related to dashboard development have been limited in providing feedback on the contents of the discussion because they are focused on discussion participation and interaction among learners. In this study, we adopted Rickey & Klein(2007)’s design and development research method type 1 and developed two types1) keyword, 2) message typesof dashboard to support the analysis of the discussion contents so that the learners can reflect on the discussion contents in the online discussion learning environment. Also we evaluated user experience by conducting the usability test and the user experience evaluation. As a result of usability test for keyword dashboard, both learners and instructors showed the highest scores in satisfaction. As a result of the user experience evaluation, both the learners and the instructors showed the highest score on the ‘novelty’ and the learners’ user experience score on the ‘dependability’ item was relatively low compared to the instructors. As a result of usability test for message types dashboard, learners showed higher average score in order of satisfaction, usability, ease of learning, and ease of use. Instructors showed higher average score in ease of learning, ease of use, satisfaction, and usability. As a result of the user experience evaluation, both the learners’ and the instructors’ ‘efficiency’ and ‘stimulation’ showed high scores and relatively, the learners had low scores on ‘attractiveness’, ‘perspicuity’, and ‘dependability’. Based on the results of usability test and user experience evaluation of two dashboards, the possibility of using dashboard to support online discussion contents analysis and implications if instructional design of discussion activities.
참고문헌 (45)
- 김태홍, "시각화 속성을 고려한 정보 서비스 평가 및 제안", 한국콘텐츠학회 논문지, 11(5), pp.489~499 (2011), 한국콘텐츠학회
- 유미나, "학습분석 기반 온라인 토론활동의 시각화 원리 개발", 교육공학연구, 33(4), pp.901~941 (2017), 한국교육공학회[DOI]
- 유미나, "온라인 토론활동에 대한 학습분석기반 대시보드 사례 분석 연구", 교육정보미디어연구, 23(3), pp.465~497 (2017), 한국교육정보미디어학회[DOI]
- 이지현, "학습분석 기반 대시보드 구성에 대한 학습자 인식 분석", 석사, pp.2016
- 이창환, "언어적 특성을 이용한 '심리학적 한국어 글분석 프로그램(KLIWC)'의 개발과정에 대한 고찰", 인지과학, 16(2), pp.93~121 (2005), 한국인지과학회
- 조영환, "온라인 토론 촉진을 위한 피드백의 역할 탐색", 아시아교육연구, 16(2), pp.289~313 (2015), 교육연구소[DOI]
- 진성희, "이러닝 학습환경에서 학습분석기반 대시보드 연구동향 분석", 교육정보미디어연구, 21(2), pp.185~213 (2015), 한국교육정보미디어학회[DOI]
- 진성희, "이러닝 학습참여활동 및 상호작용에 대한 대시보드 설계 연구", 교육공학연구, 31(2), pp.191~221 (2015), 한국교육공학회[DOI]
- 진성희, "온라인 토론활동에 대한 참여도 대표변인 및 AHP 기법에 의한 가중치 결정", 교육정보미디어연구, 23(4), pp.945~971 (2017), 한국교육정보미디어학회[DOI]
- Andresen, M. A., "Asynchronous discussion forums : Success factors, outcomes, assessments, and limitations", Journal of Educational Technology & Society, 12(1), pp.249~257 (2009)
- Awasthi, P., "INSIGHT: A semantic visual analytics for programming discussion forums", Paper presented at the 5th International Learning Analytics and Knowledge Conference (2015)
- Bueckle, M. G. N. S. A., "Empowering instructors in learning management systems: interactive heat map analytics dashboard", Paper presented at the 7th International Learning Analytics and Knowledge Conference (2017)
- Chatti, M. A., "A reference model for learning analytics", International Journal of Technology Enhanced Learning, 4(5), pp.318~331 (2012)
- Cho, Y., "Exploring adaptive instructional support for collaborative learning: Application of learning analytics", Paper presented at 2018 International Conference of Educational Technology (2018)
- Desmet, P., "Framework for product experience", International Journal of Design, 1(1), pp.57~66 (2007)
- Erickson, T., "Social translucence : an approach to designing systems that support social processes", ACM Transactions on Computer-Human Interaction, 7(1), pp.59~83 (2000)
- Erkens, G., "Automatic coding of dialogue acts in collaboration protocols", International Journal of Computer-Supported Collaborative Learning, 3(4), pp.447~470 (2008)
- Ferguson, R., "Social learning analytics: five approaches", Proceedings of LAK 2012 (2012), ACM
- Garrison, D. R., "Critical inquiry in a text-based environment: Computer conferencing in higher education", The Internet & Higher Education, 2(2-3), pp.87~105 (1999)
- Gonda, D. E. V., "Assessing an online discussion forum using a conceptual framework guided by learning analytics", Paper presented at the 7th International Learning Analytics and Knowledge Conference (2017)
- Hassenzahl, M., "User experience : A research agenda", Behaviour & Information Technology, 25(2), pp.91~97 (2006)
- Hatala, M., "Associations between students' approaches to learning and learning analytics visualizations", Paper presented at the 6th International Learning Analytics and Knowledge Conference (2016)
- Iandoli, L., "A debate dashboard to enhance online knowledge sharing", VINE, 42(1), pp.67~93 (2012)
- Jin, S., "Using visualization to motivate student participation in collaborative online learning environments", Educational Technology & Society, 20(2), pp.51~62 (2017)
- Laugwitz, B., "Construction and evaluation of a user experience questionnaire", Symposium of the Austrian HCI and Usability Engineering Group, pp.63~76 (2008), Springer
- Lund, A. M., "Measuring usability with the use questionnaire", Usability Interface, 8(2), pp.3~6 (2001)
- Mayhew, D. J., "The usability engineering lifecycle", Proceedings of CHI'99 Extended Abstracts on Human Factors in Computing Systems, pp.147~148 (1999), ACM
- Mazzolini, M., "Sage, guide or ghost? The effect of instructor intervention on student participation in online discussion forums", Computers & Education, 40(3), pp.237~253 (2003)
- McKenney, S., "Computer-based support for curriculum designers : A case of developmental research", Educational Technology Research & Development, 53(2), pp.41~66 (2005)
- Mochizuki, T., "User-centered design of online learning communities", pp.365~386 (2007), Hershey, Information Science Publishing
- Murray, T., "A prototype facilitators dashboard: assessing and visualizing dialogue quality in online deliberations for education and work", Paper presented at the International Conference on e-Learning, e-Business, Enterprise Information Systems, and e-Government (2013)
- Nielsen, J., "Designing web usability: The practice of simplicity" (1999), New Rider Publishing
- Nokelainen, P., "An empirical assessment of pedagogical usability criteria for digital learning material with elementary school students", Journal of Educational Technology & Society, 9(2), pp.178~197 (2006)
- Pallotta, V., "Automatic argumentative analysis for interaction mining", Argument & Computation, 2(2-3), pp.77~106 (2011)
- Preece, J., "A taxonomy for combining software engineering and human-computer interaction measurement approaches : towards a common framework", International Journal of Human-Computer Studies, 41(4), pp.553~583 (1994)
- Richey, R. C., "Design and development research: Methods, strategies, and issues" (2007), Routledge
- Rutter, R., "Web accessibility : Web standards and regulatory compliance" (2007), Apress
- Sack, W., "Conversation map : An interface for very large-scale conversations", Journal of Management Information Systems, 17(3), pp.73~92 (2000)
- Schrepp M., "Design, User Experience, and Usability. Theories, Methods, and Tools for Designing the User Experience", pp.383~392 (2014), Springer
- Siemens, G., "Learning analytics : The emergence of a discipline", American Behavioral Scientist, 57(10), pp.1380~1400 (2013)
- Siemens, G., "Special issue on learning and knowledge analytics", Educational Technology & Society, 15(3), pp.1~163 (2012)
- Tan, J. P. L., "Learner dashboards a double-edged sword? Students’ sense-making of a collaborative critical reading and learning analytics environment for fostering 21st century literacies", Journal of Learning Analytics, 4(1), pp.117~140 (2017)
- Teplovs, C., "The knowledge space visualizer: A tool for visualizing online discourse", Paper presented at the International Conference of the Learning Sciences (2008)
- Wickersham, L. E., "A content analysis of critical thinking skills as an indicator of quality of online discussion in virtual learning communities", Quarterly Review of Distance Education, 7(2), pp.185~ (2006)
- Wise, A. F., "Learning analytics for online discussions: A pedagogical model for intervention with embedded and extracted analytics", Paper presented at the 3rd International Learning Analytics and Knowledge Conference (2013)