An augmented reality guide for museums

The acme of perfection of AR is when you can’t distinguish the virtual from the real. We experiment with AI and computer vision in our AR museum guide ARAO to make the physical experience a natural one.  

Virtual elements in Augmented Reality (AR) need to be embedded in the real world as much as possible. This means that, just like us, they should be able to react to their environment. One example of this kind of integration is adapting to the physical environment as if it were a real object i.e. remaining visible if the virtual element is in front of an object and disappearing if it’s behind, something Apple has clearly understood with their new release of ARkit 3. Dynamic effects can also help improve how realistic the AR is if the virtual element is a mobile one.

We investigated how Artificial Intelligence could help address navigation in a museum where a virtual avatar ‘ARAO’ guides the visitor from artwork to artwork. ARAO was designed to act as a real guide with respect to its autonomy and how it pays attention to the visitor but also how it interacts with the both static and dynamic elements in the environment.  “ARAO” was embedded in reality through a mobile phone app. The video below shows this experiment.

Technology

To make it work, we modeled the indoor environment off-line as a 3D CAD model. Using an accurate localization offered by ARkit 2, we then aligned the rendering online with reality. As shown with the entry door, the AR app manages static occlusion. Specific indoor elements like mirrors, are also modeled, to provide specific rendering.

For the bottle effect, we used a 2D object detector deep network, inspired from the real-time object detection Yolo that we retrained and made run on an iPhone. From the detected 2D bounding box and in conjunction with ARkit, we’re able to guess the object pose and size in the 3D space. It allows us to manage the animation of the avatar and visual occlusion.

The AR app also runs a human pose estimation deep network on the iPhone to detect a human and her/his pose when the phone “sees” someone at a time when ARkit didn’t yet propose such a feature. The pose estimation was inspired from different state of the art solutions like OpenPose or PoseNet and ported on an iPhone. Again, thanks to ARkit 2, the app is able to guess the 3D position of the human and hence the control of avatar animation and visual occlusion.

User Experience

To create a tour for the museum visitor, he/she points ARAO on the piece of artwork of interest for a few seconds until it detects what it is and subsequently selects the corresponding tour. In the case shown in the video, it’s the tour called “Main Contributors”, a subset of the exhibition. ARAO starts the tour by showing you which to go. ARAO stops at each artwork of the tour to indicate that it’s included. Additional content describing the artwork is not included in the video just because we wanted to stay focused on the guidance but it is of course feasible. What we have shown however is that AR can give a new angle when viewing artwork e.g. by adding the depth dimension to a 2D painting.

We wanted to give some freedom to the visitors so that they can easily stray from the tour if attracted to another piece of art or take a rest somewhere without interference from ARAO. When this happens, ARAO adapts by either showing the closest artwork first or by waiting before calling for attention. In other modes, such as the ‘take me straight there’ path (not shown in the video), ARAO is more insistent if you don’t immediately follow.

As in the real world, unexpected things come up like encountering a closed door. In this case, ARAO behaves in the most natural way possible i.e. by requesting help to open the door.

The ARAO team are Nicolas Monet, Claudine Combe, Hadrien Combaluzier and Yves Hoppenot. The project was carried out in collaboration with 3D emotion for the design of ARAO and the animations.

For more information please contact the author, Yves Hoppenot.