C01: Designing the Next-Level User Experience of Integrated XR-AI Worlds

Abstract

The physical world is undergoing two main forms of integration. At a physical level, computing technologies are increasingly embedded within the material fabric of everyday objects, turning them into smart appliances and assistants and enabling embodiment, anthropomorphism, and emotional engagement. At an informational level, digital content is increasingly overlaid onto the real world, augmenting the functionality and affordances of physical objects with a digital layer that encompasses telepresence, virtual bodies, and the superposition of perceptual cues. At this intersection of integrated artificial intelligence and overlaid extended reality, what we know as the physical world is bound to change significantly. This course addresses the topic of user experience in future worlds where the physical integration of computing and the superposition of digital information layers become mainstream to deliver sensory and cognitive augmentation. It focuses on technological aspects, interaction design, and user experience guidelines. The course also proposes a thought-provoking perspective where the visions contoured within the disciplines of ambient intelligence and extended reality represent two faces of the same coin.

Course Content

The goal of the course is to inspire creative exploration of interaction design that, in the context of deep physical and informational integration of computing technology and virtual content in the physical world, evokes new experiences for users.

1. Introduction (20 min.): Establish context, provide a glimpse into existing visions of computing and interaction, highlight the importance of natural interaction among humans, digital devices, and environments in these visions.
2. Artificial Intelligence embedded in the physical world (45 min.): An exposition of ambient intelligence, smart environments, and tangible computing with a focus on conceptual grounding, design principles, and examples of representative systems with insights from user research. This part of the course will highlight natural interaction, such as proximity, movement, and gesture-based interactions, as key means of engaging intuitively with the computational and informational layer of future intelligent environments.
3. Extended Reality as the evolving informational substrate of the physical world (45 min.): an exposition of augmented, mixed, mediated, and extended reality concepts, taxonomies, and design principles accompanied by examples of representative systems and findings from user research. This part of the course will highlight natural interaction, such as gesture-based interaction and novel feedback modalities, as fundamental for engaging uniformly and consistently with both virtual content and physical objects.
4. Coffee break (30 min.)
5. Exploring the conceptual and practical implications of AI-XR (60 min.): a joint examination of AI integrated into the fabric of physical objects and XR, where virtual content is overlaid onto the surrounding space. This part highlights the user experience of living in such future worlds, illustrated with practical examples from the research community and the Super-XR project.
6. Hands-on session (30 min.): Attendees will be invited to apply the discussed principles to their own projects.
7. Wrap-up (10 min.): Conclusions will be drawn on the benefits of thinking beyond the mainstream interaction design paradigms to enable next-level user experiences emerging from the deep integration of intelligence and virtual content into the physical environment.