NUILab
Did I Do It Differently? Comparing Gesture Elicitation in Virtual and Augmented Reality
Jieqiong Li, Zahra Borhani, Matthew Buckman, Adam Sinclair Williams, Yanlin Guo, and Francisco Raul Ortega (Colorado State University); Mayra Donaji Barrera Machuca (University of Calgary); Andreea Danielescu (Wordly, Inc.)
Accepted to IEEE ISMAR 2026, the 25th International Symposium on Mixed and Augmented Reality, October 4–9 in Bari, Italy.
The paper is not out yet. Until IEEE publishes it, you can read the accepted version here: Preprint (PDF) (opens in new tab). This is a preprint, not the version of record. The final paper will appear in the IEEE ISMAR 2026 proceedings, and we will link it here when it does.
When we asked 16 people to “Move the Rightmost Building Close,” they saw the same buildings, emitters, and smoke animation both times. In augmented reality, they also saw their real hands and the physical desk. In virtual reality, they saw virtual hands and a digital copy of the room. That difference mattered, but in a narrower way than we expected.
The same wind simulation, four days apart
We built matching AR and VR versions of a wind-simulation environment in Unreal Engine and ran both on a Meta Quest 3, using video passthrough for the AR condition. The scene contained virtual buildings, smoke, and 15 emitters. Moving a building changed the simulated airflow around it, which gave participants a real visual result instead of an isolated cube sliding through empty space.
Each participant worked through 33 referents: commands such as moving a building closer, yawing it clockwise, scaling the rightmost building down, selecting three emitters, changing an emitter’s color, and opening a chart. Some involved one object, some involved several, and some had no obvious physical counterpart at all.
Participants did one modality on Day 1 and the other on Day 4, with the order counterbalanced. The two days in between were there to blunt recall. In each session a command and its resulting animation played on a loop, and the researcher stayed out of it. No gesture was ever demonstrated.
What participants actually saw. Left: AR through video passthrough, with the real room and real hands. Right: VR, with a modeled room and virtual hands.
That procedure is called gesture elicitation. Instead of handing people a gesture set the researchers already designed and asking them to rate it, we show the intended result (the referent) and ask each person to propose the gesture they would use for it. Common mappings emerge from what participants propose rather than from what we guessed.
Turning 1,056 performances into agreement rates
Sixteen participants performing 33 referents in two modalities produced 1,056 primary gestures. Two researchers coded them independently while a third ran the consensus process and the codebook revisions. The codebook ended up with 29 hand-gesture types and tracked features like one or two active hands, movement direction, and rotation. Across the whole dataset we identified 178 unique gesture combinations.
For each referent we then computed an agreement rate: the fraction of all possible participant pairs that produced equivalent gestures. With 16 participants that is 120 pairs per referent per modality. Above 0.30 counts as high agreement, 0.10 to 0.30 as medium, below 0.10 as low.
This measure is easy to misread. An agreement rate of 0.30 does not mean 30% of participants chose one gesture. It means 30% of participant pairs agreed with each other once gestures were coded for equivalence.
Most of the gesture vocabulary carried across
Overall agreement was 0.20 in AR and 0.18 in VR, a small and non-significant difference, F(1,64) = 0.640, p = 0.43. Rotation was similarly close, 0.26 in AR against 0.24 in VR. Scale ran 0.09 against 0.10, abstract actions 0.07 against 0.06.
The winning gestures tell the same story from another angle. For 73% of the 33 referents, the most frequently proposed gesture matched across AR and VR. The second-most-frequent gesture matched for 52%. Averaged across referents, 68.2% of participants personally used the same gesture in both conditions.
People kept landing on direct manipulation. Clockwise yaw was hold and rotate in yaw. Scaling down was usually a pinch in. Across all tasks, “hold” accounted for 32% of gestures in AR and 28% in VR, “grab” for 14.4% in both, and “push” for 8.5% in AR and 9.5% in VR. Single-handed, whole-hand interaction stayed dominant in both.
Agreement rate per referent in both conditions. The two modalities track each other closely almost everywhere. Yaw is the high point; the abstract commands on the right are the floor. Click to open full size.
Abstract commands were the predictable troublemakers. Turning emitters on or off, destroying the rightmost building, and opening a chart all produced low agreement, and often had no matching top gesture at all. “Yaw clockwise” carries a physical motion in its name. “Change color” tells your hand nothing.
Translation was the exception
Translation was the only category with a significant difference between modalities. Its agreement rate was 0.34 in AR and 0.25 in VR, t(14) = 2.26, p < 0.05. Selection had only two referents and was left out of the category-level tests.
The individual proposals show what that looked like. To move the rightmost building left or right, the leading VR gestures were push left and push right. In AR, participants more often held the building and moved it. For moving a building closer, hold-and-move-in won in both conditions, but agreement in AR was far tighter: 0.508, against 0.158 in VR.
Our best explanation is spatial anchoring. Translation depends on where an object sits relative to your body and the surfaces around it. In AR the physical desk, your real hands, and the room stay visible. In VR the desk is modeled, the hands are virtual, and participants could not see their own trunk. Rotation and scale can be expressed almost entirely through how the hand moves around an object. Translation asks where that object should end up, in a space whose physical cues differ between the two conditions.
That is a plausible account, not a causal result. We did not independently manipulate body visibility or the presence of a physical surface.
Where the winning gesture matched across AR and VR (1) and where it did not (0). The abstract commands on the right fail in both rows. Click to open full size.
Participants felt more difference than the gesture codes showed
Seventy-five percent of participants said the environment shaped the gestures they designed. Several described VR as more like a computer game and easier to sink into. AR pulled their attention back toward the physical world, the real room, and even the other people in it.
And yet 94% said the animation or the task requirement drove their choice, which goes a long way toward explaining why the proposals came out so similar. P7 described deliberately building a consistent vocabulary: “For each different task, I was looking for correlations between gestures and tasks. For example, if you rotate pitch up and down, I would consider using similar gestures.”
Half the participants thought about whether gestures were distinct from one another, and 44% weighed simplicity and how easy a gesture would be to remember. When an action had no physical analogy, people borrowed one. P5: “to change the color, I used a swipe gesture, just like swiping videos on TikTok. In this way, I can view the next color.”
There is a useful distinction buried in those interviews. AR and VR changed how participants thought about their gestures even when the coded result was identical. A hold-and-rotate can feel grounded in a real room or native to a digital one and still arrive at the recognizer as exactly the same input.
What this means if you are building XR interfaces
For rotation and scale, a shared gesture set is a reasonable place to start. Hold, grab, pinch, and direct rotation survived the change in modality well enough that maintaining separate AR and VR vocabularies would add learning cost without much evidence of a benefit.
Translation deserves its own prototype pass. Test hold-and-drag next to push-based movement, and test it with the actual surfaces and body visibility your system gives people. A mapping elicited in passthrough AR can keep its meaning in VR while losing the spatial cue that made it obvious in the first place.
Abstract commands need to be designed and taught, not discovered. Low agreement on things like changing color or turning emitters on says there is no single natural gesture waiting to be found. A familiar metaphor, a visible menu, or a short onboarding cue is more honest than picking one arbitrary motion and calling it intuitive. Gesture sets are also worth evaluating as sets: participants cared about keeping related commands consistent while keeping different ones distinguishable.
This lines up with what the lab cares about generally, Weiser’s invisible computer. Moving between AR and VR should not make someone relearn yaw and scale for no good reason. Translation may be one of the places where the technology genuinely changes what cues are available to think with.
The boundary on this result is sharp, and worth stating plainly: 16 participants, 76% of them between 20 and 29, one wind-simulation application, one video-passthrough headset. Participants had more prior exposure to VR than AR. Current hand tracking still misses subtle finger motion. A two-session design reduces recall but does not erase it.
The next clean experiment is to vary the physical desk and the visible body independently of AR and VR, then run the same translation referents again. That would tell us whether the 0.34-against-0.25 gap follows the modality, or just the spatial cues we happened to hand people.
This work was supported by NSF awards 2439474, 2444132, 2327569, 2238313, and 2223432. Brendan Kelley built several of the virtual objects used in the VR environment.