Applied research project · HSLU Informatik × ABUSIZZ · Innosuisse
ABUSIZZ
Exploring natural hand gestures on projected tabletop interfaces and translating them into interaction and recognition requirements
- Exploratory UX Research
- Wizard of Oz
- Usability Testing
- Prototyping
- Quantitative Analysis
- Context
- Applied research project · HSLU Informatik × ABUSIZZ · Innosuisse
- My role
- Research contributor · study design, facilitation, video analysis, interaction prototyping and quantitative evaluation
- Timeline
- 2020–2022
- Collaboration
- HSLU Algorithmic Business Research Team, ABUSIZZ and project collaborators
The prototype projected the interface onto the table, while a capacitive plate provided a reference signal for physical contact. Colour-coded gloves were used specifically to collect visually distinguishable training data for the machine-learning model. Participants did not wear gloves during the observational user studies, allowing natural gesture behaviour. A separate camera recorded hand and arm movements for post-test analysis.
Research question
How do people naturally perform familiar touch gestures on a projected tabletop, and how can those patterns inform recognition and interface design?
Why this research mattered
Context and challenge
ABUSIZZ's Signature Table Experiences turn an ordinary table into an interactive surface. The unit is a fully functional lamp that also integrates a projector with RGB and depth cameras. It projects digital content directly onto the table, while computer vision and machine-learning models interpret users' hand movements.
The work formed part of the Innosuisse-funded HSLU research project AI Supported MultiTouch for Range Images. The official project record provides further information about its institutional context, participating organisations and wider team.
The central interaction challenge was variability. People could choose different gestures for the same task, and the same gesture could differ in finger choice, hand openness, angle and scale. The recognition system therefore had to accommodate natural behaviour without requiring people to learn rigid, machine-friendly movements.
Decisions and rationale
Research strategy
From 2020 to 2022, I contributed to an extended research programme exploring how people naturally interact with projected table interfaces and how those behaviours could inform both the product's interaction design and its gesture-recognition system.
The work followed an iterative approach rather than a single fixed research design. After each study, we analysed the observed behaviour and participant feedback, identified what remained unclear, and used those insights to refine the next research question, prototype and study design. The programme gradually moved from open exploration of natural gestures towards more focused questions about onboarding, recognition, target selection and hand posture.
Across five research phases, I conducted around 110 user-test and research sessions, including exploratory user tests, comparative studies and machine-learning data collection.
Because interaction design and recognition technology were developing in parallel, I separated the work into two research streams. To understand natural behaviour, I ran glove-free studies on the projected table. Where the system could not yet recognise every gesture, I used Wizard-of-Oz control: participants experienced a responsive interface while I triggered its reactions behind the scenes. With consent, I recorded the sessions and systematically coded gesture choice, fingers used and hand posture.
For the technical development, we organised ongoing data-collection sessions throughout the project. Colour-coded gloves made hands easier to identify in the camera data, while a capacitive plate marked the exact moment of contact. These recordings provided reference data for machine-learning development within the Algorithmic Business Research Team (ABIZ).
One follow-up question was framed through Fitts's Law: selecting a target generally takes longer as the target becomes farther away or smaller. I explored whether this increased demand for precision might also be visible in people's hand posture, for example whether they close unused fingers when reaching for smaller or more distant targets. The comparison reproduced the expected size-and-distance effect for touchscreen movement time, but it did not show a reliable relationship with hand posture. This remained an exploratory hypothesis rather than a product conclusion, and I presented the line of inquiry in the World Usability Day research poster on touch gestures and Fitts's Law.
From question to evidence
Research process
-
Explore natural interaction
Use open-ended minigames to observe how people translated familiar touch gestures to a projected table without prescribing one correct movement.
-
Turn observations into questions
Analyse videos, gesture choices and participant feedback, then identify the most important uncertainty for the next research round.
-
Redesign and test again
Refine the prototype, tasks and comparison conditions around each new question, moving from broad exploration towards focused studies of onboarding, target selection and hand posture.
-
Separate evidence from limitations
Report which patterns were consistently observed, which hypotheses were not supported, and which conclusions were limited by the developing technology or study design.
Research demonstration
This video accompanies our IEEE SDS 2022 paper, Minimal Hand Pose Estimation for Touchable Projector-Depth Systems, and presents the technical approach developed within the ABUSIZZ project.
What the evidence showed
Key findings
Familiar touch patterns transferred across surfaces
Participants largely applied touch conventions they already knew from smartphones and tablets. Although the projected area was much larger, they did not automatically use broader or more exaggerated movements. Whenever a task could be completed comfortably with one hand, they usually chose a one-handed gesture.
Why it matters: Projected-table interfaces can build on established touch conventions instead of requiring people to learn an entirely new gesture language.
Index finger and thumb dominated
In the 15-participant minigame study, 85% of coded interactions could be reduced to index-finger or thumb touches. This result was carried into the 2022 IEEE paper on minimal hand-pose estimation for touchable projector-depth systems.
Why it matters: The evidence supported a six-class model that distinguishes the left and right index fingertip, thumb tip and wrist, reducing the recognition problem before expanding to less common finger configurations.
Guidance helped - within limits
Participants navigated the training application more successfully when tips were available, particularly for unfamiliar gestures. Guidance could influence the broad movement or fingers used, but not reliably prescribe subtle details such as whether unused fingers remained open or closed.
Why it matters: Use lightweight, timely visual cues for unfamiliar actions, while designing recognition to tolerate natural variation in hand posture.
Target difficulty affected timing, not hand posture
On the touchscreen, smaller and more distant targets took longer to select, as Fitts's Law predicts. We did not find consistent evidence that participants also closed their unused fingers for more difficult targets.
Why it matters: The interface and recognition model should not assume that difficult targets produce a more closed hand posture. Target size and distance can inform layout decisions, while the posture relationship remains an open research question.
Contribution
Outcome
Across the research programme, the iterative studies documented how people transferred familiar touch gestures to a projected table, how much natural variation the recognition system needed to accommodate, and where onboarding could support unfamiliar interactions. The gesture studies also showed the importance of index-finger and thumb touches and helped define a focused recognition problem for the technical team.
The work resulted in a gesture catalogue, a series of interactive research prototypes and practical guidance for projected-table interaction. The 15-participant gesture study contributed to the evidence base for Minimal Hand Pose Estimation for Touchable Projector-Depth Systems, published at the 2022 Swiss Conference on Data Science. My contribution centred on the user study and its behavioural finding; the paper's broader contribution was the depth-based machine-learning model developed by the author team.
Research quality
Reflection and limitations
The studies generated valuable exploratory insights and gave me substantial practical experience. At the same time, I personally realised that I wanted a deeper methodological foundation for planning and evaluating human-subject research.
Looking back, I would now formulate narrower hypotheses, plan sample size and statistical power earlier, distinguish exploratory from confirmatory questions more explicitly, and make more deliberate choices between within-subject and between-subject designs. I would also plan counterbalancing and the analysis of repeated measurements before data collection.
Recognising these personal knowledge gaps became an important professional turning point. It motivated me to pursue the Human Factors master's programme at TU Berlin and strengthen my knowledge of experimental psychology, statistics and human-centred research. The ABUSIZZ work was valuable precisely because it allowed me to learn through repeated studies, reflection and iteration.
HF Chapter Berlin
