Visualizing Surfaces Detected by ARKit

Visualizing ARKit Planes

Apple has worked hard to make building apps with ARKit as easy as possible. If you’re new to ARKit, it's a framework that uses the measurement sensors on the device and the camera to detect details in the physical environment. When it detects something that's new or that changed in the environment, it will notify you. Then you can create visual elements with these physical details. One example of a physical detail that ARKit can detect planes or surfaces, like the floor or a table. To demonstrate this idea, in this article we will be visualizing planes detected by ARKit.

Depending on the UI in your ARKit app, you might not necessarily show the exact position and size of detected planes at all times. However, there are still situations when visualizing ARKit planes is useful. For example, maybe you’re debugging object placement and want to see how objects are interacting with planes or maybe you’re building an outdoor application where it’s difficult to know how much of a large area has been detected. In my experience, it’s helpful to use these visualizations so as to say to your users, “You can interact with this area.” Visualizing planes is also quick to implement, so it’s a good place to start if you’re giving ARKit a spin for the first time! If you’d like to follow along, here’s a link to the code on GitHub.

ARKit

Before we get started, let’s briefly talk about how ARKit works. ARKit uses a combination of image data captured from device camera and motion data captured from device measurement sensors. With this data, ARKit uses machine learning and computer vision algorithms to analyze the environment. This analysis defines feature points which depending on how you configure ARKit at launch will create objects of interest such as detected surfaces or detected 2D/3D images.

Rendering Frameworks

It’s important to note that ARKit does not by itself generate any visual components for the user. When developing with ARKit, we must choose a “Content Technology” in order to display something. Apple gives us these three options:

• SceneKit – A 3D rendering framework
• SpriteKit – A 2D rendering framework
• Metal – A 3D rendering framework for advanced users with extra functionality

Note that these content frameworks can also be used on their own in applications like games or simulations. However, when using them with ARKit, data about the environment is mixed with virtual content created with one of these content technologies.

For this blog, since we will be visualizing 3D planes, we will render them using SceneKit. Here’s how it will work. ARKit will let us know when it thinks it’s detected a plane. Then we will take the position and sizing information about the plane and create a SceneKit object with it.

SceneKit

Let’s quickly look at how SceneKit works at a high level.

• SCNScene has a tree structure of SCNNodes, with a rootNode as the top most parent.
• An SCNNode is by itself just a point in the coordinate system of the SCNScene’s space. It carries the information of its position, orientation, and scale.
• To give an SCNNode a shape, we can add one SCNGeometry per node. For example, we could add an SCNBox, SCNSphere, or, in our case, an SCNPlane.
• An SCNMaterial is how we can add color or texture to our SCNGeometry.

The Project

Now that we know the basics of SceneKit, let’s see how we will be using it with ARKit. In the project we have the following Storyboard file:

You’ll notice that the Storyboard has an ARSCNView that covers the entire screen. ARSCNView is an object that spans across three separate Apple frameworks: ARKit, SceneKit, and UIKit. From Apple’s Documentation on ARSCNView:

• The view automatically renders the live video feed from the device camera as the scene background.
• The world coordinate system of the view’s SceneKit scene directly responds to the AR world coordinate system established by the session configuration.
• The view automatically moves its SceneKit camera to match the real-world movement of the device.

Now let’s take a look at the view lifecycle of the view controller for this Storyboard:

override func viewDidLoad() {
 super.viewDidLoad()

 sceneView.delegate = self
}

override func viewWillAppear(_ animated: Bool) {
 super.viewWillAppear(animated)

 let configuration = ARWorldTrackingConfiguration()
 configuration.planeDetection = [.horizontal]
 sceneView.session.run(configuration)
}

First, we set this view controller as the delegate for our scene view in the viewDidLoad method. We’ll go over this more when we take a look at the delegate methods themselves. Next, we set the configuration for the ARSession. Take a look at this diagram from the Apple docs:

An ARSCNView has an ARSession object which coordinates all the components required to analyze the environment. These components include the device’s motion sensor hardware, data from the camera, as well as algorithms and libraries that compile this data.

An ARSession requires an ARConfiguration in order to get started. ARKit is able to track the environment, track images, track faces, and scan 3D objects. We have to decide what we want ARKit to do in our application and we use an ARConfiguration to decide which functionality to focus on. We will be using an ARWorldTrackingConfiguration with horizontal plane detection for this project since we’re interested in tracking the physical features of the environment and visualizing horizontal planes.

Now we’re ready to implement the ARSCNViewDelegate methods. There are several methods in the protocol for this delegate, but here we only need to implement two of them:

func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {

}

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {

}

This is possibly the most important part of understanding how to create visual elements based off of results from ARKit. Let’s first talk about ARAnchors. From the Apple Developer Documentation an ARAnchor is “a real-world position and orientation that can be used for placing objects in an AR scene.” There are several classes that subclass ARAnchor depending on the type of object detected, for example ARImageAnchor and ARObjectAnchor. Specifically, we will be interested in ARPlaneAnchors since we’re trying to visualize planes.

I’d like to emphasize that there is no visual component of an ARAnchor on its own. ARAnchors won’t show up on the screen for the user by default as they’re detected. This is where SceneKit comes in and this is why these ARSCNViewDelegate methods are so convenient. These methods will get called when ARKit recognizes a new item in your scene, in this case a horizontal plane via an ARPlaneAnchor. Since an ARPlaneAnchor has the position, orientation, and size of the plane, we have everything we need to create a virtual object to mimic its physical properties. We also have a reference to an empty SCNNode that’s created specifically for this anchor.

func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
 // Cast ARAnchor as ARPlaneAnchor
 guard let planeAnchor = anchor as? ARPlaneAnchor else { return }

 // Create SCNGeometry from ARPlaneAnchor details
 let width = CGFloat(planeAnchor.extent.x)
 let height = CGFloat(planeAnchor.extent.z)
 let planeGeometry = SCNPlane(width: width, height: height)

 // Add material to geometry
 let material = SCNMaterial()
 material.diffuse.contents = UIColor.blue.withAlphaComponent(0.8)
 planeGeometry.materials = [material]

 // Create a SCNNode from geometry
 let planeNode = SCNNode(geometry: planeGeometry)
 planeNode.position = SCNVector3(planeAnchor.center.x, planeAnchor.center.y, planeAnchor.center.z)
 planeNode.eulerAngles.x = -.pi / 2

 // Add the newly created plane node as a child of the node created for the ARAnchor
 node.addChildNode(planeNode)
}

First, we cast the added ARAnchor as an ARPlaneAnchor. Then we take the physical size of the plane anchor and use it to create a SCNPlane of the same dimensions. Note that we’re only using the x and z components of the plane anchor’s extent. Since an ARPlaneAnchor is a two-dimensional shape, its y-component will always be zero. Then we add an SCNMaterial to the geometry. Here we’re choosing a semi-transparent, blue color so we can see beneath our virtual surface. Next, we create an SCNNode using the plane geometry we just created and set its position to match the center of the plane anchor. Finally, we add the new plane node as a child node of the anchor’s node.

Next, we need to fix the fact that we’re currently only adding planes when they’re first detected, but we’re not updating the visuals over time to match the updates from ARKit. To fix this, we need to make changes to our next ARSCNDelegate method, renderer(_:didUpdate:for:).

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
 // Cast ARAnchor as ARPlaneAnchor, get the child node of the anchor, and cast that node's geometry as an SCNPlane
 guard
 let planeAnchor = anchor as? ARPlaneAnchor,
 let planeNode = node.childNodes.first,
 let planeGeometry = planeNode.geometry as? SCNPlane
 else { return }

 // Update the dimensions of the plane geometry based on the plane anchor.
 planeGeometry.width = CGFloat(planeAnchor.extent.x)
 planeGeometry.height = CGFloat(planeAnchor.extent.z)

 // Update the position of the plane node based on the plane anchor.
 planeNode.position = SCNVector3(planeAnchor.center.x, planeAnchor.center.y, planeAnchor.center.z)
}

If you understood what was going on in the renderer(_:didAdd:for:) method, then you’ll probably understand what going on in this method as well since it’s mostly the same code. First, we need cast the ARAnchor to an ARPlaneAnchor like before. Then, we pick the first child node of the updated anchor node. This will be the plane node we added as a child node in the renderer(_:didAdd:for:) method. We’ll also cast the geometry of the plane node as an SCNPlane. Finally, we’ll update the plane geometry’s dimensions and the plane node’s position based off of the plane anchor’s updated information.

Let’s run the app and take a look.

Hm, well that doesn’t look right. Why are these SCNPlanes sticking straight up? From the Apple Developer Documentation on SCNPlane:

“A plane defines a flat surface in the x- and y-axis dimensions of its local coordinate space according to its width and height properties. To orient a plane differently, adjust the transform property of the node containing the plane geometry.”

The issue is that each plane is created in the x- and y-axis dimensions, where we want it in the x- and z-axis dimensions. So, all we need to do is rotate the node to fix the issue:

// Create a SCNNode from geometry
 let planeNode = SCNNode(geometry: planeGeometry)
 planeNode.position = SCNVector3(planeAnchor.center.x, planeAnchor.center.y, planeAnchor.center.z)
 planeNode.eulerAngles.x = -.pi / 2 // Add this line to fix the plane orientation issue

Let’s run to double-check if that worked.

Looks great! You might notice that sometimes it appears as if planes are created in separate spots and then merge as they grow. How come we didn’t have to delete the SceneKit nodes for those planes? The ARSCNView is deleting these SCNNodes automatically for us as needed. Since we’re creating these SceneKit plane objects as child nodes of these plane nodes, they’re deleted when the parent node is removed. (This is true as long as you don’t have some other strong reference to either these parent nodes or child nodes.) There’s another ARSCNDelegate method that we won’t be implementing in this article, but feel free to put a print statement in the renderer(_:didRemove:for:) method to see when this event is triggered in the scene.

Another Option

I’d like to show an additional way to visualize planes in this article. If we take a look at the documentation for ARPlaneAnchor, you’ll notice a property that we haven’t used yet.

/**
Geometry of the plane in the anchor's coordinate space.
*/
@available(iOS 11.3, *)
open var geometry: ARPlaneGeometry { get }

We can use this property in conjunction with ARSCNPlaneGeometry to create a more precise visual representation of the progress ARKit is making towards recognizing planes. The code in the renderer(_:didAdd:for:) method is replaced with the following:

func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
 // Cast ARAnchor as ARPlaneAnchor
 guard let planeAnchor = anchor as? ARPlaneAnchor else { return }

 let planeGeometry = ARSCNPlaneGeometry(device: MTLCreateSystemDefaultDevice()!)!
 planeGeometry.update(from: planeAnchor.geometry)

 // Add material to geometry
 let material = SCNMaterial()
 material.diffuse.contents = UIColor.blue.withAlphaComponent(0.8)
 planeGeometry.materials = [material]

 // Create a SCNNode from geometry
 let planeNode = SCNNode(geometry: planeGeometry)

 // Add the newly created plane node as a child of the node created for the ARAnchor
 node.addChildNode(planeNode)
}

Note that we use the plane geometry’s update(from:) method to get the physical dimensions from the ARPlaneAnchor’s geometry property. Here again we are bridging the gap of the physical data that ARKit provides and the visual output for the user with SceneKit. ARSCNPlaneGeometry handles the orientation and position of the plane, so we don’t need to make any changes to the plane node with this visualization option. Now let’s change our renderer(_:didUpdate:for:) method to work with ARSCNPlaneGeometry.

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
 // Cast ARAnchor as ARPlaneAnchor, get the child node of the anchor, and cast that node's geometry as an ARSCNPlaneGeometry
 guard
 let planeAnchor = anchor as? ARPlaneAnchor,
 let planeNode = node.childNodes.first,
 let planeGeometry = planeNode.geometry as? ARSCNPlaneGeometry
 else { return }

 planeGeometry.update(from: planeAnchor.geometry)
}

It looks even simpler! Here’s what we now have when we run the app:

Which Option Is Best?

With this method of plane visualization, we get a more detailed look at the progress of ARKit’s plane detection. So, which plane visualization option is best? Well, that depends on your app. For example, if you’re just visualizing planes for debugging purposes then it’s really just personal preference. However, if you’re visualizing planes in the final production version of your app then it’s more of a question of UX. While we used a transparent color for our plane material, it’s also possible to use images as materials. If that image is rectangular, for example something that looks like grid lines, then maybe it would make more sense to use SCNPlane in that case. I will say though that the additional feedback you get from using ARSCNPlaneGeometry about exactly where the app has detected a surface can be very valuable. This extra information and prompt the user to focus more on areas of a surface that still need to be detected. But ultimately the choice is your’s.

Conclusion

I hope this article has shown how easy Apple has made it to get started working with ARKit. Here’s the link to the code of the final project on GitHub if you missed it before. Good luck!