$ cat portals.json
title: Portals
category: ar-vr
client: Stanford's Immerse the Bay 2024 XR Hackathon
date: November 2023
stack:ARSnap SpectaclesSpatial AudioVoice UIAccessibility
Project Overview
An immersive AR experience for Snap Spectacles designed to bring music, culture, and climate awareness to life through interactive and accessible features. Utilizing spatial audio and computer vision, Portals allows users to engage with cultural and musical experiences in a unique way.
Situation
• Overview
As AR technologies become more widespread, there's a growing need for experiences that prioritize accessibility, cultural education, and climate awareness. Many existing AR applications focus primarily on visual elements, often overlooking audio-based interactions and accessibility features.
• Challenge
How might we create an immersive AR experience that makes diverse musical traditions accessible to everyone, including individuals with limited mobility, while promoting cultural education and climate awareness?
• Context
Snap Spectacles provide a unique platform for AR development with spatial audio capabilities, offering new possibilities for creating inclusive and educational experiences.
Task
• Overview
The goal was to develop an AR experience for Snap Spectacles that would achieve four main objectives:
• Accessibility
Design inclusive, voice-controlled features and audio cues for individuals with limited hand mobility or speech difficulties.
• Cultural Education
Enable auditory exploration of music from various cultures, focusing on historical context and lesser-known instruments.
• Joy
Create fun, collaborative AR jam sessions with friends using spatial audio.
• Climate Change Awareness
Highlight music that connects to natural environments or climate themes to increase awareness through audio immersion.
Action
1. Co-Creation Features
Developed spatial audio circles that position friends and collaborators in a virtual jam session, each playing an instrument or adding to the ambiance, allowing friends to interact audibly to avoid visual clutter.
2. Interactive Auditory Timeline
Implemented cultural markers recognition using Spectacles' computer vision to identify cultural artifacts or historical sites and launch a timeline with music, stories, and significant figures from that culture.
3. Accessibility Implementation
Created voice command controls for the experience, providing accessibility for individuals with limited mobility, along with a custom 'safe word' exit feature to prioritize user comfort and safety.
4. Educational Layers
Integrated optional deep dives into the history and significance of various music elements for educational purposes.
Result
• Inclusive AR Experience
Successfully created an AR experience that prioritizes accessibility through voice controls and audio-based interactions.
• Cultural Education Platform
Developed a platform for exploring diverse musical traditions and their historical contexts, focusing on underrepresented communities.
• Collaborative Music Creation
Implemented spatial audio features that enable users to create collaborative, immersive musical experiences with friends.
• Climate Awareness Integration
Incorporated elements that highlight the connection between music, natural environments, and climate themes to raise awareness through immersive audio experiences.