XR experiences often reset between devices, users, sessions, and task phases. That makes the interface feel unaware of what the person is trying to do.
01 Harmony framework
Adaptive XR + AI for human-centered 3D interaction.
Harmony is a research framework for interfaces that perceive context, preserve spatial continuity, and adapt with transparency so technology can support people without taking control away from them.
Why now
Spatial interfaces need continuity, not just more features.
Most digital systems still assume static screens and fixed layouts, even when the work is spatial, collaborative, and changing in real time. Harmony starts from that gap.
Interface complexity rarely changes with cognitive load, attention, or task state. The user carries the burden of adaptation.
AI-mediated interfaces need readable reasons for what changed, why it changed, and how the user can override it.
Research vision
Human judgment stays central.
Harmony investigates how AI and XR can co-adapt with people by interpreting spatial, behavioral, and collaborative cues without replacing human agency.
Design for well-being, accessibility, meaningful control, and calm interaction.
Use environmental and behavioral cues to inform adaptation without turning every signal into surveillance.
Support multi-user sensemaking, shared state, and human-AI co-creation.
Keep adaptation fast and privacy-conscious by making careful choices about local and cloud processing.
Make adaptation explainable, opt-in, auditable, and reversible.
Evaluate usefulness, cognitive load, user trust, and long-term fit instead of only measuring novelty.
Framework
Three layers working together.
Harmony is a conceptual framework for adaptive XR experiences. It connects perception, mediation, and collaboration into one research direction.
Signals about space, task, and behavior are summarized into context descriptors: just enough awareness to support adaptation.
Interface complexity, layout, and guidance tune to moment-to-moment needs so the experience can reduce friction.
Shared attention and state are aligned across people, devices, and AI assistance for smoother multi-user workflows.
01Signals
Gaze, pose, hands, task phase, spatial state.
02Context model
Compact descriptors for the current situation.
03Spatial memory
Continuity across sessions and shared spaces.
04Adaptive response
Layout, complexity, guidance, and feedback.
05User control
Explanation, override, tuning, and auditability.
Research axes
The questions guiding the work.
Each axis is intentionally practical: it connects a research question to an evaluation path and a system behavior.
- Context-aware interfaces
How can systems interpret task and behavioral cues to adapt layout, timing, and feedback?
- Perceptual adaptation
When should complexity expand or collapse to match attention and cognitive load?
- Human-AI collaboration
How do we support shared agency, transparency, and trust in co-creation?
- Edge-aware responsiveness
What should run locally, what can use cloud support, and how do latency and privacy shape that split?
- Evaluation and ethics
Which metrics capture usefulness, trust, well-being, and long-term fit?
Current explorations
Prototype questions, not product claims.
Current work scopes pilot studies and low-friction prototypes around adaptation mechanisms that can be evaluated in real scenarios.
UI elements scale, relocate, or simplify based on task phase and user focus.
Subtle cues for wayfinding and feedback without unnecessary interruption.
Lightweight mechanisms to align group attention in collaborative XR.
Micro-disclosures that show what changed, why it changed, and how to adjust it.
Where it helps
Domains where context matters.
Harmony is most relevant when the interface needs to understand task state, shared attention, user ability, or environmental context.
Adaptive simulations and skill training that respond to learner progress.
Context-aware support that adapts to individual needs and preferences.
Guidance for field service, AEC, healthcare training, and operational procedures.
Multi-user spatial sketching, review, and synchronized attention.
Progressive disclosure to manage cognitive load in high-stakes scenarios.
Spatial sensemaking where data, environment, and human attention interact.
Roadmap
A phased research path.
The roadmap keeps the work grounded: synthesize, prototype, evaluate, and share only what is responsible to share.
Literature synthesis, study design, low-fidelity prototypes, and evaluation planning.
Iterative prototypes, evaluation methods, architecture notes, and early publications.
Toolkit patterns, reference implementations, and carefully scoped community pilots.
Open materials, partnerships, and longitudinal studies where appropriate.
Ethics first: adaptation must be opt-in, reversible, and user-tunable. Transparency and well-being metrics guide every phase.
Background
Grounded in XR research and delivery.
Harmony sits inside Varun Siddaraju's broader body of XR, spatial AI, and applied systems work across research, publications, prototypes, and enterprise delivery.
50+
XR projects and prototypes across research and applied work.
2
Public books spanning mixed reality engineering and systems thinking.
3
Peer-reviewed publications plus research receipts across XR and spatial systems.
- In-Situ Wireless Channel Visualization Using AR and Ray Tracing
Sensors, 2020.
- X-Reality: AR meets IoT
IEEE INFOCOM Demos, 2018.
- AR Facet Mapping Technique for Ray Tracing
ICDT Best Paper Award, 2018.
- Beginning Windows Mixed Reality Programming
Apress, 2021.
Ecosystem
Part of Varun Innovates Lab.
This is the owned, indexable Harmony home inside the lab ecosystem.
Get involved
For research collaboration and serious applied work.
Harmony is a research-driven framework. Collaboration is best when it connects HCI, XR systems, responsible AI, spatial computing, or applied evaluation.
- Is Harmony a product?
No. It is a research framework. Any tools or patterns released from it should prioritize evaluation, openness, and responsible use.
- Will code be open-sourced?
Where feasible and ethical, reference designs and patterns can be shared. IP-sensitive or study-sensitive work follows normal research and client boundaries.
- How is adaptive success measured?
Through mixed-methods HCI evaluation: task outcomes, cognitive load, user trust, accessibility, and longitudinal engagement.