Taking Augmented Images for a Spin

Last year at Google IO '18 we saw ARCore for Android take a leap forward with Sceneform. Google introduced improvements to the platform that made augmented reality (AR) more attainable for developers to achieve in their applications. Checkout this blog for a recap of these enhancements!

Skipping forward to 2019, Google introduced further enhancements to ARCore — specifically Augmented Images, that will help developers create even more realistic AR experiences. These Augmented Image enhancements include the ability to track moving images, and once an image has been detected, continue tracking even after the image leaves the camera frame. In this blog, we are going to demonstrate these enhancements via a sample application.

Sample application

This blog includes a sample application that we'll use to demonstrate how ARCore can track moving images along with the ability to continue tracking an image after it leaves the camera frame. To do this we've created three unique images that are placed on top of three toy cars. When an image is detected by the app, we use Sceneform to attach a virtual text label above the corresponding car. It is important to note that we are using QR codes only because they are unique. You can easily swap these images out for actual images to fit your use case. Later, this post will demonstrate how to add and use your own images.

How augmented images work

Augmented Images in ARCore allows developers to build AR applications that detect and respond to 2D images in the physical world. The application simply provides a set of reference images for ARCore to try and detect. Once they are found, ARCore identifies the location of those images in the physical world.

For more information on the capabilities and limitations of ARCore, please see the official documentation.

AR Terminology

Renderable is a 3D model or a 2D Android Layout that can be placed anywhere in the scene and consists of Meshes, Materials, and Textures.

Trackable is something that ARCore can track and that Anchors can be attached to.

Anchor represents a fixed location and orientation in the real world. Anchors stay at a fixed location in physical space, the numerical description of this position will update as ARCore's understanding of the space improves.

Getting started

To get started, we need to provide the reference images that ARCore will detect and track. We do this by adding our images to an AugmentedImageDatabase that we configure to our ARCore session.

Create image database

There are two ways to create your image database: (1) Create a new empty database at runtime with the default constructor AugmentedImageDatabase and add the individual images or (2) load a saved database that already contains your images with the static method AugmentedImageDatabase.deserialize(). You can still dynamically add more images later with option (2).

In our sample code, we go with (2). The ARCore SDK for Android provides a command line tool called arcoreimg that makes it easy to generate an image database file. You can provide your images by specifying the directory containing your images or by creating an image list file. We recommend the latter as it allows you to specify each image's width which improves tracking performance.

Below shows the image list file that we use for this sample.

car_database.imgdb-imglist.txt

truck.png|truck.png|0.026
blue_car.png|blue_car.png|0.026
red_car.png|red_car.png|0.026

In each line, we specify the file name, path, and physical width of the image (in meters).

Then you can load your database and configure your ARCore session for image tracking by setting the session config to one that is configured with your image database.

 var augmentedImageDatabase: AugmentedImageDatabase? = null
 context!!.assets.open("car_database.imgdb")
 .use { loadedDB -> augmentedImageDatabase = AugmentedImageDatabase.deserialize(session, loadedDB) }
 config.augmentedImageDatabase = augmentedImageDatabase
 ...
 session.configure(config);

Listen for frame updates

In order to add augmented content, we need to listen for frame updates during our ARCore session. We do this by registering a Scene.OnUpdateListener to our SceneView . This callback will be hit once per frame right before our scene is updated.

 arFragment.arSceneView.scene.addOnUpdateListener { updateFrame() }

Now we can check the track state of our reference images and anchor augmented content.

 val frame = arFragment.arSceneView.arFrame
 val updatedAugmentedImages = frame.getUpdatedTrackables(AugmentedImage::class.java)
 for (augmentedImage in updatedAugmentedImages) {
 when (augmentedImage.trackingState) {
 TrackingState.TRACKING -> {
 // Create an anchor for the image that is tracking
 val node = AnchorNode(augmentedImage.createAnchor(augmentedImage.centerPose))
 arFragment.arSceneView.scene.addChild(node)
 addTextLabel(augmentedImage, node)
 }
 }
 }

Add augmented text

Once tracking starts for a given image, we can inflate our text label and add it above our toy car.

 ViewRenderable.builder().setView(this, R.layout.car_label_layout).build()
 .thenAccept { viewRenderable ->
 val noteText = Node()
 noteText.renderable = viewRenderable
 //scale the text to the size we need it.
 val localScale = Vector3()
 localScale.set(0.15f, 0.15f, 0.15f)
 noteText.localScale = localScale
 noteText.setParent(anchorNode)
 val textView = viewRenderable.view.findViewById<TextView>(R.id.car_label)
 //change the type of car based on index in database
 when (augmentedImage.index) {
 0 -> textView.text = "2014\nChevy Silverado"
 1 -> textView.text = "2008\nMitsubishi Lancer"
 2 -> textView.text = "2016\nPorsche Panamera S"
 }
 }

In order to know which text to display we check the database index of our AugmentedImage. This will be consistent with the order that our images were read in from our image list file car_database.imgdb-imglist.txt.

Conclusion

Augmented Images have become more advanced and powerful with the ability to maintain trackable images and their tracked states as they move within the frame. Furthermore, adding the ability to Augment Images in your application has never been easier to do. Google continues to work hard to abstract the complex problems behind AR to enable developers like us to focus on creating experiences specific to our app users. For more information, be sure to check out the sample application.

Authors

Clinton Teegarden is the Mobile & Devices Lead & Engineer at CapTech. He specializes in delivering products for Fortune 500 clients in Mobile, Android Architecture, & IoT.

Dylan Doggett is a software engineer at CapTech based in Phoenix, Arizona. He specializes in Android development and has worked on several banking and eCommerce applications.