Enhancing Context-Aware Interaction in Collaborative VR Using Retrieval-Augmented Generation

Collaborative Virtual Reality (VR) interactions often lack context awareness, leading to disjointed communication and inefficient task coordination. While Zhou et al. (2024a, 2024b) demonstrated the value of shared control in VR collaboration, existing systems struggle to adapt to dynamic task contexts (e.g., changing user roles, unexpected obstacles). This study addresses this gap by integrating Su et al. (2025)’s dynamic and parametric retrieval-augmented generation (RAG) into a collaborative VR framework. The RAG module retrieves context-relevant information (e.g., task guidelines, user preference history) in real time and generates personalized interaction cues (verbal, visual). We tested the system with 28 participants (14 pairs) across two collaborative tasks (virtual training, problem-solving). Results show that the RAG-enhanced system increases task completion accuracy by 25.4% (p<0.01) and reduces communication errors by 31.2% (p<0.001) compared to a non-RAG baseline (based on PairPlayVR; Zhou et al., 2024b). Subjective feedback highlights improved context relevance (SUS score: 84.6 vs. 70.2) and reduced cognitive load. This work advances collaborative VR by merging RAG-driven context awareness with shared control, building on prior research in both VR interaction and retrieval generation.