A multimodal system that uses augmented reality to improve football coaching and communication of gameplay strategies.



how might we leverage augmented reality to help athletes practice gameplays more effectively?



10 weeks
Jan - Mar 2018



Hy Nguyen
Nicolas Germann
Kate Johnson
Daphne Liang
Laura Lopez
Luke McJunkin



involvement highlights

Interaction Flow
Visual Design
High-fidelity Prototyping
Unity Development


holographic interaction experience

Augmented Reality (AR)* has been a game-changer for interaction design due to its 3-dimensional, immersive, yet realistic nature. We were tasked with designing a holographic interaction experience for the Microsoft HoloLens, with a focus on producing a high-fidelity AR prototype.


*also marketed as Mixed Reality, or sometimes encompassed within the broader MR spectrum


opportunities in AR sports training

From our field observation and competitive analysis, we identified sports training as a potential use case for AR since it’s crucial for the interactions to be immersive without being disruptive to or disconnected from the real world. Moreover, HoloLens’ capabilities to track movement and recognize gestures are suitable for enhancing the physical activity experience.


introducing ARena

A multimodal application suite for sports strategy communication and simulation using Augmented Reality and Over-the-Air deployment. The coach will be using a tablet, while the athlete will be using a HoloLens.




choosing and deploying plays

The coach can add, edit, and ultimately choose which strategy (play) to be deployed to the athlete's HoloLens for practice using a flick up motion.



viewing stats

ARena also collects individual athlete's practice statistics (completion time, accuracy, drills completed) for easy viewing, managing and comparison.




visualizing and practicing plays

The play visualization is then procedurally generated and anchored in space by the athlete's headset. The athlete can clearly visualize the intended plays, follow the highlighted trail to practice movement path, and observe virtual teammates' positions.


gestural controls

Using HoloLens' gesture recognition, the athlete can fully control the experience to better comprehend the strategy.






rewind/fast forward



why sports training?

From our competitive analysis, the bulk of AR experience in sports has been focused on the viewer’s experience without much consideration for the athlete’s. Moreover, while multi-angle, speed control capabilities have been essential to the viewing experience, why can’t we have the same capabilities for athletic training?

multimodality for multiple stakeholders

From our secondary research, we decided that multimodal experience would be a good fit for ARena because of its distinct groups of stakeholders (coach and athlete) with varied level of exposure to technology and involvement in the physical activities. Inspired by Apple’s AirDrop, we designed the over-the-air deployment concept for seamless interactions between the coach and athlete.


interaction flow

From the initial concept, we moved forward with fleshing out a comprehensive interaction flow to identify necessary screens or scenes design

main flow@2x.png

AR prototyping in Unity

Our initial introduction to AR prototyping started with building the scene in Unity. Using simple 3D geometries and standard assets pack, we built out the initial setup for our experience.



To calibrate object and UI positioning in 3D space, I then implemented a first-person camera for rapid prototyping before porting to HoloLens.

Our initial athlete's UI implementation featured a static and fixed on screen display for play-related information



shipping to the HoloLens

Shipping to the HoloLens enabled us to fully experience the initial build in augmented reality. But more importantly, the HoloLens build also uncovered various usability issues and technical constraints that needed addressing:

  • Eye-level and fixed on glass interfaces were obstructive to training activities.

  • Due to studio policy, we could not take the HoloLens outdoor for further testing.

  • Developing mainly on MacOS, my workflow was substantially affected by the HoloLens’ Windows environment requirement.


porting to iPhone

As a workaround to technical and policy constraints, I ported the prototype to the iPhone using ARKit and XCode. I then iterated and made necessary changes to the interface, i.e. locking it in 3D space, to minimize distraction while still providing necessary information.

This final prototype enabled us to experience true augmented reality on the field but without gesture recognition capabilities. However, making the necessary trade-off allowed us to move fast and focus on the main experience.



future steps

expand beyond football

The same model of strategy communication and visualization in AR has potentials to be implemented in other team and individual sports, e.g. basketball, baseball, motorsport, etc.

support multiple athletes simultaneously

An essential part of team practice is, well, teamwork. To best support simultaneous team training, we have to think about possible athlete-to-athlete and coach-to-athletes interactions.

investigate hardware capability

ARena was designed for a near future where AR technology is more refined and portable. As we move on to further iterate ARena, we want to investigate deeper into what hardware capabilities and options are being developed.


deal with constraints

Despite the various constraints presented in this project, I was excited to be adaptive and always explore alternatives in tools and platforms to push the concept to its highest-possible fidelity.

understand limitations beforehand

As AR is still a new technology, we had to intentionally make assumptions about the technology to complete the project in time. In retrospect, I would have dedicated more time to fully understand the technical limitations before starting.

2D screen vs. 3D scene

Prototyping for AR presented a variety of elements that I hadn't considered before when designing for 2D scenes. Scale, angle, distance, and line of sight all matter in making the AR experience a delightful one.