Interface Application of a Virtual Assistant Agent in an Immersive Virtual Environment
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
In immersive virtual environments including mixed reality (MR) and virtual reality (VR), avatars or agents, which are virtual humans, are being studied and applied in various ways as factors that increase users' social presence. Recently, studies are being conducted to apply generative AI as an agent to improve user learning effects or suggest a collaborative environment in an immersive virtual environment. This study proposes a novel method for interface application of a virtual assistant agent (VAA) using OpenAI’s ChatGPT in an immersive virtual environment including VR and MR. The proposed method consists of an information agent that responds to user queries and a control agent that controls virtual objects and environments according to user needs. We set up a development environment that integrates the Unity 3D engine, OpenAI, and packages and development tools for user participation in MR and VR. Additionally, we set up a workflow that leads from voice input to the creation of a question query to an answer query, or a control request query to a control script. Based on this, MR and VR experience environments were produced, and experiments to confirm the performance of VAA were divided into response time of information agent and accuracy of control agent. It was confirmed that the interface application of the proposed VAA can increase efficiency in simple and repetitive tasks along with user-friendly features. We present a novel direction for the interface application of an immersive virtual environment through the proposed VAA and clarify the discovered problems and limitations so far.
Abstract
In immersive virtual environments including mixed reality (MR) and virtual reality (VR), avatars or agents, which are virtual humans, are being studied and applied in various ways as factors that increase users' social presence. Recently, studies are being conducted to apply generative AI as an agent to improve user learning effects or suggest a collaborative environment in an immersive virtual environment. This study proposes a novel method for interface application of a virtual assistant agent (VAA) using OpenAI’s ChatGPT in an immersive virtual environment including VR and MR. The proposed method consists of an information agent that responds to user queries and a control agent that controls virtual objects and environments according to user needs. We set up a development environment that integrates the Unity 3D engine, OpenAI, and packages and development tools for user participation in MR and VR. Additionally, we set up a workflow that leads from voice input to the creation of a question query to an answer query, or a control request query to a control script. Based on this, MR and VR experience environments were produced, and experiments to confirm the performance of VAA were divided into response time of information agent and accuracy of control agent. It was confirmed that the interface application of the proposed VAA can increase efficiency in simple and repetitive tasks along with user-friendly features. We present a novel direction for the interface application of an immersive virtual environment through the proposed VAA and clarify the discovered problems and limitations so far.
참고문헌 (22)
- Maloney D, "Falling asleep together:What makes activities in social virtual reality meaningful to users", Proceedings of the Annual Symposium on Computer-Human Interaction in Play, pp.510~521 (2020)
- Kim J, "VIVR : Presence of immersive interaction for visual impairment virtual reality", IEEE Access, 8, pp.196151~196159 (2020)
- Cho Y, "X-person asymmetric interaction in virtual and augmented realities", Computer Animation and Virtual Worlds, 32(5), pp.e1985~ (2021)
- S. Hong, "A Study on Movement Interface in Mobile Virtual Reality", Journal of the Korea Computer Graphics Society, 27(3), pp.55~63 (2021)
- Huang A, "Proxemics for human-agent interaction in augmented reality", Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp.1~13 (2022)
- A. von der Pütten, ""It doesn’t matter what you are!"Explaining social effects of agents and avatars", Computers in Human Behavior, 26(6), pp.1641~1650 (2010)
- Kyrlitsias C, "Social interaction with agents and avatars in immersive virtual environments: A survey", Frontiers in Virtual Reality, 2 (2022)
- Oh C S, "A systematic review of social presence: Definition, antecedents, and implications", Frontiers in Robotics and AI, 5 (2018)
- Pelechano N, "Being a part of the crowd : Towards validating vr crowds using presence", 7th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2008) (2008)
- Park M, "Application of virtual avatar using motion capture in immersive virtual environment", International Journal of Human-Computer Interaction (2023)
- Park M, "Application of Immersive Virtual Environment Through Virtual Avatar Based On Rigid-body Tracking", Journal of the Korea Computer Graphics Society, 29(3), pp.69~77 (2023)
- Narang S, "Simulating movement interactions between avatars & agents in virtual worlds using human motion constraints", 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp.9~16 (2018)
- Holz T, "Mixed reality agents as museum guides", 2nd Agent-Based Systems for Human Learning (2006)
- Topsakal O, "Framework for a foreign language teaching software for children utilizing ar, voicebots and chatgpt(large language models)", The Journal of Cognitive Systems, 7(2), pp.33~38 (2022)
- Jones B, "Using generative ai to produce situated action recommendations in augmented reality for high-level goals", GenAICHI (2023)
- Numan N, "Ubiq-genie : Leveraging external frameworks for enhanced social vr experiences", 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp.497~501 (2023)
- "OPENAI: Chatgpt" (2023)[URL]
- "Openai unity package" (2023)[URL]
- "Aicommand-chatgpt integration with unity editor" (2023)[URL]
- "MICROSOFT: Mixed reality toolkit" (2022)[URL]
- "Spiritual chat with chatgpt npcs in virtual reality" (2023)[URL]
- Lund A., "Measuring usability with the use questionnaire", Usability Interface, 8(2), pp.3~6 (2001)