A multimodal system that uses augmented reality to improve football coaching and communication of gameplay strategies
how might we leverage augmented reality to help athletes practice gameplays more effectively?
Jan - Mar 2018
holographic interaction experience
Augmented Reality (AR)* has been a game-changer for interaction design due to its 3-dimensional, immersive, yet realistic nature. We were tasked with designing a holographic interaction experience for the Microsoft HoloLens, with a focus on producing a high-fidelity AR prototype.
*also marketed as Mixed Reality, or sometimes encompassed within the broader MR spectrum
opportunities in AR sports training
From our field observation and competitive analysis, sports training is a potential use case for AR since it’s crucial for the interactions to be immersive without being disruptive to or disconnected from the real world. Moreover, HoloLens’ capabilities to track movement and recognize gestures are suitable for enhancing the physical activity experience.
A multimodal application suite for sports strategy communication and simulation using Augmented Reality and Over-the-Air deployment. The coach will be using a tablet, while the athlete will be using a HoloLens.
choosing and deploying plays
The coach can add, edit, and ultimately choose which strategy (play) to be deployed to the athlete's HoloLens for practice using a flick up motion.
ARena also collects individual athlete's practice statistics (completion time, accuracy, drills completed) for easy viewing, managing and comparison.
visualizing and practicing plays
The play visualization is then procedurally generated and anchored in space by the athlete's headset. The athlete can clearly visualize the intended plays, follow the highlighted trail to practice movement path, and observe virtual teammates' positions.
Using HoloLens' gesture recognition, the athlete can fully control the experience to better comprehend the strategy.
why sports training?
From our competitive analysis, the bulk of AR experience in sports has been focused on the viewer’s experience without much consideration for the athlete’s. Moreover, while multi-angle, speed control capabilities have been essential to the viewing experience, why can’t we have the same capabilities for athletic training?
multimodality for multiple stakeholders
From our secondary research, we decided that multimodal experience would be a good fit for ARena because of its distinct groups of stakeholders (coach and athlete) with varied level of exposure to technology and involvement in the physical activities. Inspired by Apple’s AirDrop, we designed the over-the-air deployment concept for seamless interactions between the coach and athlete.
From the initial concept, we moved forward with fleshing out a comprehensive interaction flow to identify necessary screens or scenes design
AR prototyping in Unity
Our initial introduction to AR prototyping started with building the scene in Unity. Using simple 3D geometries and standard assets pack, we built out the initial setup for our experience.
To calibrate object and UI positioning in 3D space, I then implemented a first-person camera for rapid prototyping before porting to HoloLens.
Our initial athlete's UI implementation featured a static and fixed on screen display for play-related information
shipping to the HoloLens
Shipping to the HoloLens enabled us to fully experience the initial build in augmented reality. But more importantly, the HoloLens build also uncovered various usability issues and technical constraints that needed addressing:
Eye-level and fixed on glass interfaces were obstructive to training activities.
Due to studio policy, we could not take the HoloLens outdoor for further testing.
Developing mainly on MacOS, my workflow was substantially affected by the HoloLens’ Windows environment requirement.
porting to iPhone
As a workaround to technical and policy constraints, I ported the prototype to the iPhone using ARKit and XCode. I then iterated and made necessary changes to the interface, i.e. locking it in 3D space, to minimize distraction while still providing necessary information.
This final prototype enabled us to experience true augmented reality on the field but without gesture recognition capabilities. However, making the necessary trade-off allowed us to move fast and focus on the main experience.
expand beyond football
The same model of strategy communication and visualization in AR has potentials to be implemented in other team and individual sports, e.g. basketball, baseball, motorsport, etc.
support multiple athletes simultaneously
An essential part of team practice is, well, teamwork. To best support simultaneous team training, we have to think about possible athlete-to-athlete and coach-to-athletes interactions.
investigate hardware capability
ARena was designed for a near future where AR technology is more refined and portable. As we move on to further iterate ARena, we want to investigate deeper into what hardware capabilities and options are being developed.
deal with constraints
Despite the various constraints presented in this project, I was excited to be adaptive and always explore alternatives in tools and platforms to push the concept to its highest-possible fidelity.
understand limitations beforehand
As AR is still a new technology, we had to intentionally make assumptions about the technology to complete the project in time. In retrospect, I would have dedicated more time to fully understand the technical limitations before starting.
2D screen vs. 3D scene
Prototyping for AR presented a variety of elements that I hadn't considered before when designing for 2D scenes. Scale, angle, distance, and line of sight all matter in making the AR experience a delightful one.