NUILab
NUILab · Research
Research
How do we make interaction with 3D and immersive systems natural, low-effort, and available to everyone?
Our lab researches human-centered computing problems, including human-computer interaction using augmented and virtual reality (also referred to as extended reality) and human-AI interaction. We started building natural user interaction (hence the name of the lab), looking originally only at gestures, and later at other unimodal (e.g., gesture, speech, gaze, and touch) and multimodal interaction paradigms. Over time, we have added multiple problems in augmented and virtual reality, and now we are also adding work in human-centered AI. One of the new areas of particular interest is how to apply neuromorphic computing to some of the problems we have been working on. Below are the different aspects of the lab, organized by cluster. Some projects may sit in more than one area.
Where we’re heading. The lab has always asked how people naturally interact with computers. As AI becomes part of that interface, we’re extending the question to human-centered AI: how people work alongside intelligent systems naturally, effectively, and safely — in the real world and in immersive ones. As mentioned earlier, a big emphasis is using neuromorphic computing for extended reality and other human-centered computing research problems.
Our north star: making the computer invisible“The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.”
— Mark WeiserEverything below serves that goal: letting people concentrate on the task, not the technology.
XR Input Devices and Interaction Techniques: From Unimodal/Multimodal User Interaction to Intelligent User Interfaces
Given a headset and no instructions, what do people actually do? They reach, point, speak, and look, often at the same time. The hard part is that there is no single natural vocabulary waiting to be discovered. What people propose depends on how you ask them, what you show them, and what they have used before.
XR Visual Search and Notifications
Augmented reality (AR) can put an arrow on the object you are looking for. The questions are where to place that arrow and whether the cue is correct — and what information-access effort costs. Related to visual search cues, we have notifications and timers: how do they compete for attention with the world they exist to explain, and where do we place them?
XR Training & Education
We have been looking at computer science education, wind and fluids engineering, and US Navy training systems to improve learning. We have used different approaches, including cognitive load theory.
XR for Health & Wellbeing
Does a virtual forest do what a real one does? Nature contact lowers stress and restores attention, and extended reality makes it deliverable to a hospital room, a dorm, or a clinic. But nature is not one thing, and neither is the benefit. We ask which properties of a restorative environment survive being simulated in virtual reality, for whom, and on which measure.
XR Accessibility
Accessible XR is usually framed as adding a feature. Our evidence points somewhere else: what people need is control over the configuration. No single layout or interaction pattern serves everyone, and the arrangement that helps one person is often the one that gets in another’s way. The community is what matters, and in our research we always follow Charlton’s principle: “Nothing About Us Without Us.”
Human-Centered AI: Neuromorphic Computing, Human-AI Decision Making, and Agentic-AI
There are four high-level primary questions. (1) How can neuromorphic computing aid user interaction and decision making? (2) How is decision making affected as a user goes from a frontier LLM to a smaller model, and everywhere within that spectrum? (3) Agentic AI is set to become pervasive in many jobs. How does its use affect our cognitive load, and how can we improve the interfaces? And (4) how can neuromorphic computing and other AI paradigms help the user have a personalized interaction?
XR Systems: Networking, Cybersecurity, and Privacy
XR interaction depends on more than the headset and the interface. We have conducted work in networking and cybersecurity, and we are currently looking at privacy. For example, we studied the path from sensing and rendering to network transport and edge computation. Our goal is to support real-time collaboration, understand possible threats, and protect data.
Funding
The lab's work is supported by more than $5.5M in external funding at CSU since 2018 — from NSF, ONR, DARPA, NIH, and foundation partners.
Active grants
- NSF I-USE · co-PI Collaborative Research: Improving Conceptual Understanding of Invisible Physics by Translating the Science of Learning into Virtual Reality Environments
- ONR DURIP · PI Training Optimization for US Navy and Marine Radio Operations to Assess Cognitive Load and Manage Extraneous Load
- Office of Naval Research · PI Assessing Cognitive Load and Managing Extraneous Load to Optimize Training
- NSF CAREER · PI CAREER: Microgesture and Multimodal Interaction Techniques for Augmented Reality
Completed grants (14)
- Office of Naval Research · PI Perceptual/Cognitive Aspects of Augmented Reality: Experimental Research and a Computational Model
- DARPA · PI Ego-Centric Emotion Recognition using Augmented Reality Headsets (I2O Postdoctoral Fellowship)
- Dan Marino Foundation · Gift Improving User Interfaces for Young Adults with Autism
- Dan Marino Foundation · Gift Improving User Interfaces for Young Adults with Autism
- ONR DURIP · PI WARFighting Performance: Augmented Reality Multi-Modal Interaction Techniques for JTAC and Battlefield Readiness
- NSF CRII · PI CRII: CHS: Understanding Gesture User Behavior in Augmented Reality Headsets
- NSF CCRI · PI CCRI Planning: Collaborative Research: Low-Latency for Augmented Reality Interactive Systems (LLARIS)
- ONR sub-award via VR Rehab, Inc. · PI Fused Augmented Realities with Synthetic Vision (FAR/SV) Systems for Ground Forces
- ONR via VR Rehab, Inc. · PI SAIPAN: Single Amphibious Integrated Precision Augmented Reality Navigation System (SBIR Phase I)
- NIH challenge award to BioMagic VR, Inc. · PI BioBraceVR: Bio-Interactive Device with Personalized Avatar Therapy for SUD (NIH-NIDA SUD Challenge)
- NSF SBIR via Polymer Braille Inc. · PI NSF SBIR Phase IIA: 2.5D Extensions to Braille-based User Interaction (sub-award)
- NSF Future of Work · co-PI (PI: Benjamin Clegg) FW-HTF-P: Optimizing Long-Term Human Performance in Future Work
- DARPA via DOD-ARMY · co-PI (PI: J. Ross Beveridge) Communication through Gestures, Expression, and Shared Perception (CWC)
- Florida Center for Cybersecurity (FC2) · co-PI (PI: Peter Clarke, FIU) Using a Cyberlearning Environment to Enhance Critical Cybersecurity Education