Taking ARCore's Sceneform for a Test Drive

Taking Sceneform from ARCore for a Test Drive

At Google IO '18, Google quietly introduced sizeable improvements to the existing ARCore for Android. These improvements are crucial to making AR features attainable for application developers and collaborative for their users. Along with Cloud Anchors, which enables multi-device AR experiences, and Augmented Images, which enables AR apps to interact with 2D spaces, Google introduced Sceneform. With Sceneform, Google has drastically reduced the barrier to entry of creating AR features for your everyday Android developer. As an Android developer, you no longer need to be well versed in OpenGL or Unity to build AR features in your application.

Before Sceneform

When ARCore was introduced, there was a lot of excitement about the opportunities it created for Android Developers. Sadly, it quickly became apparent that to do AR well, a developer would need to become proficient in OpenGL and/or Unity. This meant that AR features for your average Android developer would be pretty difficult, if not impossible all together.

Introducing Sceneform

Sceneform abstracts boilerplate and complex code into a simple and easy to use API. This makes AR development attainable for nearly all Android developers.

Sceneform also comes with a plugin for Android Studio which visually renders 3D objects with the same engine that is used in ARCore itself. With this plugin, developers know exactly how their objects will render in the AR world. Take, for example, the post-it note object below. The window on the right displays how the object will render in the AR world, and the Sceneform binary file on the left allows us to modify attributes of the object using simple key-value pairs.

The Test Drive

To demo the simplicity that Sceneform brings to ARCore, this blog comes with a sample application. Through this sample, we will demo how to set up a simple 3D "Post-It" note, and allow the user to write a note for their augmented world. A note must be placed on an Anchor point that is trackable. As you use the application, ARCore helps you visualize this by displaying dots when newly mapped areas are added to its understanding of the world.

Key Concepts

Though Sceneform abstracts a lot of the complex code into a nice, concise API, it is important that we start with a few basic definitions of some key classes in ARCore and Sceneform.

• Scene - a class used to maintain a scene graph, or a hierarchical organization of content in the currently known world.

• HitResult - derived from testing hits which can be user touches or derived points from the scene to determine if the specific point in space can be tracked.

• Trackable - something that ARCore can track, and which Anchors can be attached to.

• Node - represents a transformation within the scene graph's hierarchy, and can have 3D objects that belong to it. Thus a node, is how that object looks from a given perspective. A node can belong to the scene or to other nodes that belong in the scene's hierarchy.

• Anchor - used to describe a fixed location and orientation in the real world. An Anchor can be used as a point of reference for nodes to maintain a physical location.

• TransformableNode - a node that can be selected, translated, rotated, and scaled using gestures. This can be used to transform how objects appear in a physical space.

• Renderable - a class used to render objects and widgets in a 3D space.

Rubber to The Road

In the below snippet, we are showing the usage of the ARFragment. The use of the ARFragment is optional, but it removes the need to write some really boring code such as permission management and gesture animations. You could skip this step all together and use the ARSceneView directly.

The next bit of highlighted code is pulled directly from the sample application. It is called from a GestureDetector event, which is used to attempt to place a Post-It note where the user taps. This code, is responsible for taking that gesture event and determining if the point selected by the user is a Trackable point that can be used as an Anchor.

/**
 * Method that takes the user's tap event and creates an anchor from it
 * to attach a renderable post it note.
 *
 * @param motionEvent
*/
private void tapAddObject(MotionEvent motionEvent) {
 Frame frame = mFragment.getArSceneView().getArFrame();
 //make sure we are tracking before we process data.
 if (selectedId == -1 || motionEvent == null || frame == null |
 frame.getCamera().getTrackingState() != TrackingState.TRACKING)
 return;

 //check to see if the list of hit results contains a trackable element
 for (HitResult hit : frame.hitTest(motionEvent)) {
 Trackable trackable = hit.getTrackable();
 if ((trackable instanceof Plane &&
 ((Plane)trackable).isPoseInPolygon(hit.getHitPose()))) {
 //set the 3D model to the anchor
 buildRenderable(mFragment, hit.createAnchor());

 //remove selected item after a successful set.
 selectedId = -1;
 postImageView.setBackground(null);
 break;
 }
 }
}
 

The next method is called by the tapAddObject(MotionEvent motionEvent) mentioned above. This set of code is responsible for building a Renderable, in our case a Post-It note, which can then be attached to a node for displaying at the user's selected position. This Renderable is returned via a CompletableFuture, which runs on a background thread.

/**
 * Method to build the renderable post it note.
 *
 * @param fragment
 * @param anchor
*/
private void buildRenderable(ArFragment fragment, Anchor anchor) {
 ModelRenderable.builder()
 .setSource(fragment.getContext(), Uri.parse("post_it.sfb"))
 .build()
 .thenAccept(renderable -> addNodeToScene(fragment, anchor, renderable))
 .exceptionally((throwable -> {
 AlertDialog.Builder builder = new AlertDialog.Builder(this);
 builder.setMessage(throwable.getMessage())
 .setTitle("Codelab error!");
 AlertDialog dialog = builder.create();
 dialog.show();
 return null;

 }));
}

Finally, the addNodeToScene method is called after our Renderable is loaded and ready for display. This method, creates an AnchorNode, which is used as a reference for the physical position of the TransformableNode. A TransformableNode is used to allow the user to make gesture transformations to the node itself.

This method also adds an Android View layout containing a TextView which will act as the note that the user is leaving. This is loaded via a ViewRenderable and displayed to the user in a position relative to the parent node. The parent node is the Post-It note's node. In this case, the code also adjusts the relative location of the text node in relation to it's parent node via the noteText.setLocalPosition(vector) method. This was used to align the text appropriately on the Post-It note.

/**
 * Method to take a renderable and attach it to the anchor point the user selected.
 *
 * @param fragment
 * @param anchor
 * @param renderable
*/
private void addNodeToScene(ArFragment fragment, Anchor anchor, Renderable renderable) {
 AnchorNode anchorNode = new AnchorNode(anchor);
 TransformableNode postitNode = new TransformableNode(fragment.getTransformationSystem());
 postitNode.setRenderable(renderable);
 postitNode.setParent(anchorNode);

 //add text view node
 ViewRenderable.builder().setView(this, R.layout.post_it_text).build()
 .thenAccept(viewRenderable -> {
 Node noteText = new Node();
 noteText.setParent(fragment.getArSceneView().getScene());
 noteText.setParent(postitNode);
 noteText.setRenderable(viewRenderable);

 //set the note position in relation to its parent node. 
 // In this case, along the y axis of the Post-It note
 noteText.setLocalPosition(new Vector3(0.0f, -0.05f, 0f));
 });

 //code for editing text removed for brevity

 fragment.getArSceneView().getScene().addChild(anchorNode);
 postitNode.select();
}

Finished Product

The Fineprint

Just as with most things in life, Sceneform comes with a few things to keep in mind before you dive into building comprehensive AR experiences.

1. Though the abstraction provided by Sceneform has made development straightforward, at times you can get in the weeds with complex spatial concepts such as Quaternions and Collisions that Sceneform simply cannot abstract away. Developers may require a basic undersanding of such concepts before building complex features.
2. Sceneform will likely not suffice when building complex AR features. You may find that you still need to leverage OpenGL concepts to build certain experiences.
3. Sceneform doesn't remove the need for graphics developed by an experienced 3D artist. Some art may be purchased online to support your application needs.
4. Sceneform is still lacking some Kotlin extensions that could really take development to the next level.

The Bottom Line

The introduction of ARCore's Sceneform makes AR for Android more exciting and fun to write. It finally seems that AR for Android is easily attainable for most enterprise applications. With a basic understanding of 3D objects and space, an Android developer can pick up the Android Studio plugin and have an AR app in no time.