Rapid Authoring

Rapid Authoring of Augmented Reality Task Guidance

vhs repair.jpg — Imagine being able to see instructions for how to perform a difficult task, displayed right on top of the object you are working on. For example, it could show you the location of a switch, or how to place a tool. This is extremely helpful to guide you through a task that you have never done before, such as operating or repairing some complex equipment.

screwdriver tag.jpg — Augmented reality (AR), running on a smart phone, can show instructions and annotations that are displayed right on top of equipment. These annotations (“tags”) can be 3D models of tools, animated arrows, or text. The key idea is that they are continuously aligned with the object, so that as the person moves around, the tag appears to stay on top of the object.

AR task guidance can enable people to do tasks that they have never done before. Research shows that augmented reality can reduce task time (see Henderson, et al) and the number of errors (see Webel, et al).

Before using the system, someone has to create the instructions and labels for the particular task. This could be a “subject matter expert”, a trainer, or anyone that is familiar with the task. It’s important to make it as easy as possible to create guidance for new tasks and new objects. Our method uses a standard smart phone to place tags and instructions, using an intuitive interface. The authoring can be done in a matter of minutes.

Below is an example of authoring a task for repairing a VHS video player. The author selects tags from a menu and places them on the object. At each step, the author can type in instructions, and optionally snap a photo to show additional detail for each step. At the end, data is saved to the Google Cloud Platform. As can be seen by the time counter, the authoring took only about 6 minutes to create.

Authoring of VHS repair task.

Once the task steps have been created, the system can be used by someone else to perform the task. The video below shows a person guided successfully through the maintenance task. Note - this person has never seen this object before, yet she was easily able to perform the task.

Use of the augmented reality system to perform the repair of the VHS player.

Implementation Details

The system was created using Unity and AR Foundation, and is able to run on most Android phones and iPhones, as well as headsets such as HoloLens and Magic Leap. During the authoring process, it attaches tags to AR Cloud Anchors and uploads them to the Google Cloud Platform. During the “show” process, it “resolves”, or re-recognizes these anchors, so that the tags are correctly registered to the objects.

*The covariance of a tag’s position is represented by a 3D ellipsoid.*

Note - since standard smart phones do not typically have a depth sensor, an adjustment step is needed during the authoring process (this step would not be necessary for AR devices that have a depth sensor). Tag adjustment takes into account the uncertainties of the tag position and the camera pose, represented as covariance matrices. It mathematically estimates the new position and uncertainty of the tag, after each adjustment. Details of this method are in Hoff, et al.