Augmented Reality Board Game

Updated for 2019 China-US Young Maker Competition.

I love games. Even as an adult, they're a great opportunity to socialize with others and exercise critical thinking, be it with board games, or table top RPGs. After doing a bit of learning about augmented reality with Android, I decided to combine some IoT technology with mobile AR to make a simple board game for educational purposes. I had been sitting on this idea for just over a year, so I'm pretty thrilled to have finally done it :P

This project will go over creating a simple game board, determining where a game piece is, and then using the game piece data to display interesting augmented reality information on an Android phone.

The story behind the board game (because all boardgames need some sort of story :)) is that the players are aliens who have discovered the Voyager space probe, complete with maps to our solar system. As they traverse our solar system, they gather information about the planets and sun, which will be displayed on their hand-held computers.

You can see the the (current) finished version of the project in this video, and this tutorial should have all of the information you need for making your own :)

Creating the Game Board

The first step I took to creating the game board was to draw out a rough path in GIMP using the Paths Tool. Once I had the path drawn, I put in some clip art planets and the sun (rather than constantly saying planets and sun, I'm just going to say planets from this point on :P) from a coloring book I found online and dots designating where the game piece would need to be to show the information for that planet.

After drawing out the game board, I needed to make it. Luckily, there's a free public maker space in the town I live in, so I took the above image there and converted it into a vector file via Adobe Illustrator so that it could be printed onto a 2ft x 1ft piece of plywood.

After a little over an hour I had the completed board.

In order to know where the game piece is, I decided to use NFC stickers on the back of the board. This worked out because the plywood that I used was only 1/8th of an inch (~3mm) thick, so the signal can be read through the board.

Assembling a Game Piece

To create a game piece, I decided to try out the new Particle boards that I've had sitting around since I bought them via their pre-order. I figured if I have multiple pieces, they can communicate together over a mesh network. I ended up using a Particle Xenon for the game piece, and a Particle Argon as the gateway between the game board and Google Cloud.

Before moving on with this section, I had to configure the Particle boards using their app, and create a mesh network from the Argon with the Xenon connected to it.

The one issue I ran into here is that the Particle boards have a spot for supporting an NFC antenna, but the firmware currently doesn't support it.

Luckily, I had a cheap RC522 laying around, and found this Hackster project that used it with a Particle Photon. I followed Ingo's wiring instructions and copied his RFID library, and was quickly up and running reading the ID off of a card.

After wiring up the NFC reader, I also attached a small battery to the back of the breadboard. The piece itself isn't as small as I would like, but for a prototype it isn't bad. Someday I'd actually like to print off a circuit board and force it to be more vertical.

Programming the Game Piece

The code for the game piece is actually really straight-forward. After importing the RFID library linked in that other Hackster post, return to the main file for your Particle application. The top of the file contains the definition for the pins used for a soft SPI connection between the RFID reader and the Particle Xenon, as well as the declaration for the RFID object. You will also want to add a flag to prevent duplicate messages from being sent when a tag isn't present.

#include "RFID.h"
#define SS_PIN      A2
#define RST_PIN     D2
#define MOSI_PIN    D3
#define MISO_PIN    D4
#define SCK_PIN     D5
RFID RC522(SS_PIN, RST_PIN, MOSI_PIN, MISO_PIN, SCK_PIN);
  
bool flag = false;

Next, the setup method initializes the RFID object.

void setup()
{ 
 RC522.init();
}

The main loop will check to see if an NFC tag is present, and then read the address off of that tag. When the address is read, it is sent over the mesh network to the gateway with an event name of nfc-tag, which will handle sending it to our backend service.

void loop()
{
 uint8_t i;
 if (RC522.isCard())
 {
   flag = false;
   String address = "";
   RC522.readCardSerial();
   Serial.println("Card detected:");
   for(i = 0; i <= 4; i++)
   {
     Serial.print(RC522.serNum[i],HEX);
     address += String(RC522.serNum[i],HEX);
     Serial.print(" ");
   }
   Serial.println();
   Serial.println(address);
   Mesh.publish("nfc-tag", address);
 }

While the Serial.print() methods aren't necessary, they were nice for debugging. You can view the Serial messages using the Particle CLI tools

./particle serial monitor --follow

The next step is where things get a little tricky. I ran into a bug in the RFID library where a successful read would be followed by a failed read, and then would repeat in an alternating pattern. Rather than continuously swapping back and forth, I added a second check in the else statement before actually doing anything to clear the piece's state.

else {
     if (!RC522.isCard()) {
         Serial.println("Card NOT detected");
         if( !flag ) {
             flag = true;
             Mesh.publish("nfc-tag", "none");
         }
     }
 }

One thing I ran into here, which I would love to fix, is when using this library to read and write to specific blocks on the tags, it would reset whatever was written on the next device loop. Not really sure why, but that's something to investigate further at a later time. For now, this project will drive everything based on the addresses of the tags that were placed on the board.

Voyager:
88 4 C1 BF F2
Uranus:
88 4 1F BB 28
Neptune:
88 4 3E BC E
Saturn:
88 4 96 BF A5
Jupiter:
88 4 9D BF AE
Mars:
88 4 A3 BF 90
Earth:
88 4 BB BF 88
Venus:
88 4 B5 BF 86
Mercury:
88 4 AF BF 9C
Sun:
88 4 A9 BF 9A

Programming the Gateway

Before programming the gateway, this project needs an integration between Google Cloud and Particle set up. You can find the latest documentation on how to do this on Particle's website. Mine has an event name of gamesetting.

After setting up the integration, the gateway code is very straight-forward. It uses a callback that will forward any data received over the mesh network to Google Cloud. For simplicity, we'll assume there's only one game piece. This can be expanded later by adding the game piece/device ID to the data that's forwarded to the cloud.

void myHandler(const char *event, const char *data)
{
 Particle.publish("gamesetting", data ? data : "none", PRIVATE);
}
  
void setup() {
 Mesh.subscribe("nfc-tag", myHandler);
}
void loop() {
}

Firebase Function and Database

In order to wrap up our backend, we will need to create a new Firebase project and connect it to our Google Cloud project. You can do this by installing Firebase tools, creating a new directory, and then running the firebase init command from your new directory.

You will also need to go into the Firebase Console and enable the real-time database. I created mine in test mode so I could avoid any authentication headaches.

Once this is done, you can open the index.js that was created in your Firebase project folder to add the code that will be triggered when data is sent from a Particle device to your Google Cloud project.

The first step in your Firebase Function code will be to declare your required objects, get a reference to a Firebase database, and initialize the app.

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
var db = admin.database();

Next, because we're using the tag addresses to represent our planets, rather than any other information stored on the tags, we'll need a map to link these addresses to the planet names.

var planetMap = new Map();
planetMap.set("884c1bff2",	'Voyager');
planetMap.set("8841fbb28",	'Uranus');
planetMap.set("8843ebce",	'Neptune');
planetMap.set("88496bfa5",	'Saturn');
planetMap.set("8849dbfae",	'Jupiter');
planetMap.set("884a3bf90",	'Mars');
planetMap.set("884bbbf88",	'Earth');
planetMap.set("884b5bf86",	'Venus');
planetMap.set("884afbf9c",	'Mercury');
planetMap.set("884a9bf9a",	'Sun');
planetMap.set("none", 		'None')

We can then create a new function that will listen to the Google Cloud Pub/Sub that we created earlier when declaring the integration between Google Cloud and Particle, in this case, arboardgame-status. This function will retrieve the NFC tag address, convert it to a String and then call another function to update the Firebase real-time database that will drive our mobile app.

exports.updateGameBoard = functions.pubsub
 .topic('arboardgame-status')
 .onPublish((data, context) => {
 	const tag = Buffer.from(data.data, 'base64').toString();
 	console.log(tag);
   return Promise.all([
   	updateCurrentDataFirebase(tag)
   ]);
 });

The updateCurrentDataFirebase() function will update the tag value for piece #1 (we can update this to support multiple pieces later) with the planet name for the scanned tag.

function updateCurrentDataFirebase(tag) {
 var ref = db.ref(`/piece/1/tag`);
 return ref.set({
   planet: planetMap.get(tag)
 });
}

After running the firebase deploy command from our Firebase Functions project, we can scan a tag and check the Firebase Functions logs to see that it was successful.

And if we open the Firebase Real-Time Database page in the Firebase console, we'll also see the updated value appear as soon as we have scanned a tag.

Android App Setup

You can start your Android app by opening Android Studio and creating a new empty project with a minimum SDK level of 24. Make sure you have Kotlin support enabled, as this project will be Kotlin based instead of Java. This will create your activity_main.xml layout file, as well as your MainActivity.kt file.

After your project has been created, open your project level build.gradle file. Under the dependencies node, include the classpath for the sceneform plugin.

dependencies {
    classpath 'com.android.tools.build:gradle:3.2.1'
    classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
    classpath 'com.google.ar.sceneform:plugin:1.7.0'
}

Next, go into your module level build.gradle file. You'll start by adding compileOptions with support for Java 1.8 to your android node.

compileOptions {
    sourceCompatibility JavaVersion.VERSION_1_8
    targetCompatibility JavaVersion.VERSION_1_8
}

Next, under the dependencies node of this gradle file, include the sceneform core library. We'll also add additional libraries here as they become necessary.

implementation "com.google.ar.sceneform:core:1.7.0"

Below the dependencies node, apply your ar sceneform plugin.

apply plugin: 'com.google.ar.sceneform.plugin'

Finally, you will want to add the following lines for importing your 3D models.

sceneform.asset('sampledata/models/Sol/Sol.gltf',
        'default',
        'sampledata/models/Sol/Sol.sfa',
        'src/main/assets/Sol')
  
sceneform.asset('sampledata/models/Mercury/Mercury.gltf',
        'default',
        'sampledata/models/Mercury/Mercury.sfa',
        'src/main/assets/Mercury')
  
sceneform.asset('sampledata/models/Venus/Venus.gltf',
        'default',
        'sampledata/models/Venus/Venus.sfa',
        'src/main/assets/Venus')
  
sceneform.asset('sampledata/models/Earth/Earth.gltf',
        'default',
        'sampledata/models/Earth/Earth.sfa',
        'src/main/assets/Earth')
  
sceneform.asset('sampledata/models/Luna/Luna.gltf',
        'default',
        'sampledata/models/Luna/Luna.sfa',
        'src/main/assets/Luna')
  
sceneform.asset('sampledata/models/Mars/Mars.gltf',
        'default',
        'sampledata/models/Mars/Mars.sfa',
        'src/main/assets/Mars')
  
sceneform.asset('sampledata/models/Jupiter/Jupiter.gltf',
        'default',
        'sampledata/models/Jupiter/Jupiter.sfa',
        'src/main/assets/Jupiter')
  
sceneform.asset('sampledata/models/Saturn/Saturn.gltf',
        'default',
        'sampledata/models/Saturn/Saturn.sfa',
        'src/main/assets/Saturn')
  
sceneform.asset('sampledata/models/Neptune/Neptune.gltf',
        'default',
        'sampledata/models/Neptune/Neptune.sfa',
        'src/main/assets/Neptune')
  
sceneform.asset('sampledata/models/Uranus/Uranus.gltf',
        'default',
        'sampledata/models/Uranus/Uranus.sfa',
        'src/main/assets/Uranus')

If you try a gradle sync at this point, you'll notice that it fails. To fix this, download the models and assets zip files that I have included with this project. You will want to create a new directory named sampledata under your app directory, and place the models directory you downloaded under there. You will also want to put the assets directory under your src/main directory. At this point your app should build.

Before moving on to the more exciting parts of this app, we'll need to finish setting up by going into the AndroidManifest.xml file. Within the manifest tag, add the uses-permission and uses-feature tags for camera support and Internet usage.

<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.INTERNET" />
  
<uses-feature android:name="android.hardware.camera.ar" android:required="true"/>

Next, add the largeHeap property to the application tag.

<application
    android:allowBackup="true"
    android:icon="@mipmap/ic_launcher"
    android:label="@string/app_name"
    android:roundIcon="@mipmap/ic_launcher_round"
    android:supportsRtl="true"
    android:theme="@style/AppTheme"
    android:largeHeap="true">

Finally, within the application tag, add a meta-data tag to declare that ARCore support is required.

<meta-data android:name="com.google.ar.core" android:value="required" />

Requesting Camera Permissions

Because we will need to use the camera for our AR app, the user must grant permission to use that hardware. At the top of your your MainActivity class, add a value that will be used to keep track of the permission.

private val CAMERA_PERMISSION = 42

Next, in onCreate(), call a new function named requestCameraPermission().

override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    setContentView(R.layout.activity_main)
    requestCameraPermission()
}

This function will use Android's built in permission requesting framework.

private fun requestCameraPermission() {
    ActivityCompat.requestPermissions(
        this,
        arrayOf(Manifest.permission.CAMERA), CAMERA_PERMISSION    )
}

Finally, you will override the onRequestPermissionResult() function to verify that the permission has been granted. If it hasn't, you can display a message to the user and close the app.

override fun onRequestPermissionsResult(
    requestCode: Int,
    permissions: Array<out String>,
    grantResults: IntArray) {
    if( !hasCameraPermission() ) {
        Toast.makeText(this,
            "You must grant camera permissions to use this app.",
            Toast.LENGTH_SHORT).show()
    }
}
  
private fun hasCameraPermission() : Boolean {
    return ContextCompat.checkSelfPermission(
        this,
        Manifest.permission.CAMERA) == PackageManager.PERMISSION_GRANTED}

Reading from the Firebase Database

Before adding the Firebase Database to your app, you will need to register the app in the Firebase console. Go to the console's project overview page and find the Addapp button towards the top of the page.

After clicking on that button, select the Android icon for your platform.

On the next page, fill in your application's package name. The SHA-1 at this point is optional, as we won't be using any of the features that require it for this one-off project.

You will then be presented with the option to download the google-services.json file for your Firebase project. Download it and place it in the app directory of your project.

Returning to your Android project, open the project level build.gradle file and add the google-servicesclasspath declaration under the dependencies node.

classpath 'com.google.gms:google-services:4.0.1'

Next, go into your module level build.gradle file in your Android app. You will add the firebase-database and firebase-core dependencies under the dependencies node.

implementation 'com.google.firebase:firebase-core:16.0.7'
implementation 'com.google.firebase:firebase-database:16.1.0'

as well as apply the google-services plugin next to where you applied the ar sceneform plugin.

apply plugin: 'com.google.gms.google-services'

In the onCreate() function of MainActivity.kt, call a new function named initData(). In this function, you can retrieve the value stored in Firebase and log it. We'll save this value in the next step when we want to display a planet.

private fun initData() {
    val database = FirebaseDatabase.getInstance()
    val myRef = database.getReference("piece/1/tag/planet")
    myRef.addValueEventListener(object : ValueEventListener {
        override fun onDataChange(dataSnapshot: DataSnapshot) {
            val value = dataSnapshot.getValue(String::class.java)
            Log.e("TEST", "Value is: $value")
        }
        override fun onCancelled(error: DatabaseError) {
            Log.w("TEST", "Failed to read value.", error.toException())
        }
    })
}

Displaying a Planet

Now that we're done with the general setup of getting information from our game board via an IoT device, through a cloud backend and into an Android app, we can get to something a even more exciting: using augmented reality on our mobile device.

The first thing you will want to do is open your activity_main.xml file and replace everything inside of it with the following code. This will provide the AR scene view that will be used by your app to display the camera input and your 3D models.

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">
    <com.google.ar.sceneform.ArSceneView
        android:id="@+id/ar_scene_view"
        android:layout_width="fill_parent"
        android:layout_height="fill_parent"
        android:layout_gravity="top"/>
</RelativeLayout>

Next, return to your MainActivity file and add the following code to define the variables and values that will be used. You can find information about these values and what they will be used for in the code snippet's comments.

//Maps what we get from Firebase with an easier to work with value
companion object {
    val SUN = "Sun"
    val MERCURY = "Mercury"
    val VENUS = "Venus"
    val EARTH = "Earth"
    val MARS = "Mars"
    val JUPITER = "Jupiter"
    val SATURN = "Saturn"
    val NEPTUNE = "Neptune"
    val URANUS = "Uranus"
}
  
//Item that we get back from Firebase
private lateinit var celestial: String
  
//Prevents stacking multiple instances of the same object
private var hasPlacedObject = false
  
//Reference to the ARSceneView declared in activity_main.xml
private lateinit var arSceneView: ArSceneView
  
//Used to intercept tap events on the screen
private lateinit var gestureDetector: GestureDetector
  
//Used to validate if the user has the AR Core app installed on their device
private var installRequested: Boolean = false
  
//Used to clear the base when a new value is available
private var objectBase : Node? = null
  
//Renderable objects for our 3D models
private var sunRenderable: ModelRenderable? = null
private var mercuryRenderable: ModelRenderable? = null
private var venusRenderable: ModelRenderable? = null
private var earthRenderable: ModelRenderable? = null
private var marsRenderable: ModelRenderable? = null
private var jupiterRenderable: ModelRenderable? = null
private var saturnRenderable: ModelRenderable? = null
private var neptuneRenderable: ModelRenderable? = null
private var uranusRenderable: ModelRenderable? = null
  
//Flag for when the 3D models have finished loading
private var hasFinishedLoading = false

In onCreate(), we're going to call a series of methods that will make our code a little more manageable. We'll define those methods in a little bit. You'll notice that we also initialize the arSceneView object by pulling it from our layout file.

override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    setContentView(R.layout.activity_main)
  
    arSceneView = findViewById(R.id.ar_scene_view)
  
    initData()
    initRenderables()
    initGestures()
    initSceneView()
    requestCameraPermission()}

The first method, initData(), hasn't changed too much from when we first created it. We will update it to clear out the base AR object, if it's been created, and reset our object placed flag. You'll notice that we're also using a default value if nothing is stored in the Firebase database. This shouldn't happen, but just in case, we'll default to showing Mercury.

private fun initData() {
    val database = FirebaseDatabase.getInstance()
    val myRef = database.getReference("piece/1/tag/planet")
    myRef.addValueEventListener(object : ValueEventListener {
        override fun onDataChange(dataSnapshot: DataSnapshot) {
            celestial = dataSnapshot.getValue(String::class.java) ?: MERCURY
            hasPlacedObject = false
            this@MainActivity.runOnUiThread { objectBase?.setParent(null) }
        }
  
        override fun onCancelled(error: DatabaseError) {
            Log.w("TEST", "Failed to read value.", error.toException())
        }
    })
}

Next, we'll initialize our renderable objects. This will get a little redundant since we're doing the same thing for multiple objects, but you can see here how we're pulling the models from the assets that were loaded into the project earlier and turning them into renderable objects for our app.

private fun initRenderables() {
    val sunStage =
        ModelRenderable.builder().setSource(this,
            Uri.parse("Sol.sfb")).build()
  
    val mercuryStage =
        ModelRenderable.builder().setSource(this,
            Uri.parse("Mercury.sfb")).build()
  
    val venusStage =
        ModelRenderable.builder().setSource(this,
            Uri.parse("Venus.sfb")).build()
  
    val earthStage =
        ModelRenderable.builder().setSource(this,
            Uri.parse("Earth.sfb")).build()
  
    val marsStage =
        ModelRenderable.builder().setSource(this,
            Uri.parse("Mars.sfb")).build()
  
    val jupiterStage =
        ModelRenderable.builder().setSource(this,
            Uri.parse("Jupiter.sfb")).build()
  
    val saturnStage =
        ModelRenderable.builder().setSource(this,
            Uri.parse("Saturn.sfb")).build()
  
    val neptuneStage =
        ModelRenderable.builder().setSource(this,
            Uri.parse("Neptune.sfb")).build()
  
    val uranusStage =
        ModelRenderable.builder().setSource(this,
            Uri.parse("Uranus.sfb")).build()
  
    CompletableFuture.allOf(
        sunStage,
        mercuryStage,
        venusStage,
        earthStage,
        marsStage,
        jupiterStage,
        saturnStage,
        neptuneStage,
        uranusStage).handle { t, u ->
        try {
            sunRenderable = sunStage.get()
            mercuryRenderable = mercuryStage.get()
            venusRenderable = venusStage.get()
            earthRenderable = earthStage.get()
            marsRenderable = marsStage.get()
            jupiterRenderable = jupiterStage.get()
            saturnRenderable = saturnStage.get()
            neptuneRenderable = neptuneStage.get()
            uranusRenderable = uranusStage.get()
            hasFinishedLoading = true
        } 
        catch(e: InterruptedException) {}
        catch(e: ExecutionException) {}
    }
}

After creating your renderables, you will want to set your app up to respond to screen gestures. For now, define initGestures() like so

private fun initGestures() {
    gestureDetector = GestureDetector(this,
        object: GestureDetector.SimpleOnGestureListener() {
            override fun onSingleTapUp(e: MotionEvent?): Boolean {
                onSingleTap(e, celestial)
                return super.onSingleTapUp(e)
            }
  
            override fun onDown(e: MotionEvent?): Boolean {
                return true
            }
        })
  
    arSceneView.scene.setOnTouchListener { hitTestResult, motionEvent ->
        if( !hasPlacedObject ) {
            return@setOnTouchListener gestureDetector.onTouchEvent(motionEvent)
        }
        return@setOnTouchListener false
    }
}

The onSingleTap() method will take the tap gesture, confirm that our renderables are loaded, and then pass the necessary information to the tryPlacingObject() method.

private fun onSingleTap(tap: MotionEvent?, celestialKey: String) {
    if( !hasFinishedLoading ) {
        return
    }
    val frame = arSceneView.arFrame
    if( frame != null ) {
        if( !hasPlacedObject && tryPlacingObject(tap, frame, celestialKey))
            hasPlacedObject = true
    }
}

tryPlacingObject() is where we're able to check for a tracking state and if the tap occurred along a flat plane. If it did, we can create our planet and draw it on the screen using the createCelestial() method and appending it to the anchorNode.

private fun tryPlacingObject(tap: MotionEvent?, frame: Frame, celestialKey: String) : Boolean {
    if( frame.camera.trackingState == TrackingState.TRACKING) {
        for (hit in frame.hitTest(tap)) {
            val trackable = hit.trackable
            if (trackable is Plane 
                && trackable.isPoseInPolygon(hit.hitPose)) {
                val anchor = hit.createAnchor()
                val anchorNode = AnchorNode(anchor)
                anchorNode.setParent(arSceneView.scene)
                objectBase = createCelestial(celestialKey)
                anchorNode.addChild(objectBase)
                return true
            }
        }
    }
    return false
}

The createCelestial() method is where we actually draw the object, size it and place it in relation to our anchor point. We'll revisit this method later as we add some animations to our planets.

private fun createCelestial(celestialKey: String) : Node {
    val base = Node()
    val celestialObject = Node()
    celestialObject.setParent(base)
    celestialObject.localPosition = Vector3(0.0f, 0.5f, 0.0f)
    celestialObject.localScale = Vector3( 0.5f, 0.5f, 0.5f)
  
    if( celestialKey == SUN ) {
        celestialObject.renderable = sunRenderable
    } else if( celestialKey == MERCURY ) {
        celestialObject.renderable = mercuryRenderable
    } else if( celestialKey == VENUS ) {
        celestialObject.renderable = venusRenderable
    } else if( celestialKey == EARTH ) {
        celestialObject.renderable = earthRenderable
    } else if( celestialKey == MARS ) {
        celestialObject.renderable = marsRenderable
    } else if( celestialKey == JUPITER ) {
        celestialObject.renderable = jupiterRenderable
    } else if( celestialKey == SATURN ) {
        celestialObject.renderable = saturnRenderable
    } else if( celestialKey == NEPTUNE ) {
        celestialObject.renderable = neptuneRenderable
    } else if( celestialKey == URANUS ) {
        celestialObject.renderable = uranusRenderable
    }
    return base}

Returning to onCreate(), you'll see that there's one more method that we call: initSceneView(). This will simply add the update listeners required for ARCore in order to process each frame.

private fun initSceneView() {
    arSceneView
        .scene
        .addOnUpdateListener { frameTime ->
            val frame = arSceneView.arFrame
            if( frame == null )
                return@addOnUpdateListener
            if( frame.camera.trackingState != TrackingState.TRACKING)
                return@addOnUpdateListener
        }
}

While the bulk of drawing a planet is done, we still have some Android lifecycle goodness to take care of. We will need to override the onResume() method and initialize our AR session, as well as resume any views that were already created.

override fun onResume() {
    super.onResume()
    if (arSceneView.session == null) {
        try {
            val session = createArSession(installRequested)
            if (session == null) {
                installRequested = hasCameraPermission()
                return
            } else {
                arSceneView.setupSession(session)
            }
        } catch (e: UnavailableException) {
            Toast.makeText(this, 
                "Exception occurred.", 
                Toast.LENGTH_LONG).show()
            finish()
        }
    }
    try {
        arSceneView.resume()
    } catch (ex: CameraNotAvailableException) {
        finish()
        return
    }
}

If the session hasn't already been created, then we will need to do that in our createArSession() method.

private fun createArSession(installRequested: Boolean) : Session? {
    var session: Session? = null
    if( hasCameraPermission()) {
        when( ArCoreApk.getInstance().requestInstall(this, !installRequested)) {
            ArCoreApk.InstallStatus.INSTALL_REQUESTED -> return null
            ArCoreApk.InstallStatus.INSTALLED -> {}
        }
  
        session = Session(this)
        val config = Config(session)
        config.updateMode = Config.UpdateMode.LATEST_CAMERA_IMAGE
        session.configure(config)
    }
  
    return session
}

Finally, when it's time to pause or destroy our app, we will need to make sure that the AR scene is also paused or destroyed.

override fun onPause() {
    super.onPause()
    arSceneView.pause()
}
  
override fun onDestroy() {
    super.onDestroy()
    arSceneView.destroy()
}

At this point you should be able to plug in your devices and use the board game. As you move along the tags under the board, you can display the corresponding planet.

If you move the piece to another part of the board, the planet will automatically clear and allow you to place another.

Rotate the Planet

While you're able to see the planets now, they're just sort of there without doing much. Let's fix that. We'll start by making sure each planet is able to rotate along it's axis. In order to do this, we'll create a new class called RotatingNode. This class will extend the ARCore Node class that we were already using, and have a set of variables that can be set by MainActivity to change the speed, direction and axis of rotation.

class RotatingNode(val clockwise: Boolean = false,
                   val axisTiltDeg : Float = 0.0f,
                   val rotationSpeedMultipler : Float = 1.0f,
                   val degreesPerSecond: Float = 90.0f) : Node() { 

This class will also have an ObjectAnimator declared at the top of the class

private var animator : ObjectAnimator? = null

Next, we'll override the onActivate() method to initialize our ObjectAnimator, set the animation target as the current node, set the duration and then start the animation.

override fun onActivate() {
    if (animator != null) {
        return
    }
  
    animator = createAnimator()
    animator!!.setTarget(this)
    animator!!.setDuration(getAnimationDuration())
    animator!!.start()
}

The createAnimator() method is where the bulk of the work occurs. It will start by creating the base quaternion related to orientation angles.

private fun createAnimator() : ObjectAnimator {
    val orientations = arrayOfNulls<Quaternion>(4)
    val baseOrientation = Quaternion.axisAngle(Vector3(1.0f, 0f, 0.0f), 
        axisTiltDeg)
    for (i in orientations.indices) {
        var angle = (i * 360 / (orientations.size - 1)).toFloat()
        if (clockwise) {
            angle = 360 - angle
        }
  
        val orientation = Quaternion.axisAngle(Vector3(0.0f, 1.0f, 0.0f), 
            angle)
        orientations[i] = Quaternion.multiply(baseOrientation, orientation)    
}

Next this method will initialize the ObjectAnimator, set the orientation values, and then sets various properties to notify the node that it should rotate forever.

val animator = ObjectAnimator()
animator.setObjectValues(*orientations as Array<Any>)
  
animator.propertyName = "localRotation"
animator.setEvaluator(QuaternionEvaluator())
animator.repeatCount = ObjectAnimator.INFINITE
animator.repeatMode = ObjectAnimator.RESTART
animator.interpolator = LinearInterpolator()
animator.setAutoCancel(true)
return animator

The getAnimationDuration() returns a value derived from the degrees the planet should rotate per second, and the speed multiplier. If the app only uses default values, it should take about four seconds to complete one rotation.

private fun getAnimationDuration(): Long {
    return (1000 * 360 / (degreesPerSecond * rotationSpeedMultipler)).toLong()
}

As each frame progresses, the onUpdate() method will be called. This method will determine what fraction of a completed animation cycle has been completed, and then cause the next step to occur. If the speed multiplier is 0, then the animation simply doesn't occur.

override fun onUpdate(frameTime: FrameTime?) {
    super.onUpdate(frameTime)
    if (animator == null) {
        return
    }
    if( rotationSpeedMultipler == 0.0f ) {
        animator!!.pause()
    } else {
        animator!!.resume()
        val animatedFraction = animator!!.getAnimatedFraction()
        animator!!.setDuration(getAnimationDuration())
        animator!!.setCurrentFraction(animatedFraction)
    }
}

When it's time for an animation to stop, the onDeactivate() method will be called and the animation will be canceled and freed.

override fun onDeactivate() {
    if (animator == null) {
        return
    }
  
    animator!!.cancel()
    animator = null
}

Now if we return to MainActivity, we can go into the createCelestial() method and change the declaration of our celestialObject from Node to RotatingNode.

val celestialObject = RotatingNode()

Now you should be able to place a planet and watch it rotate. If you move the game piece to another planet spot on the board, you'll be able to place the new planet and have it rotate at the same rate.

Showing the Planet Name

The next thing we'll want to do is add an additional view to the screen in order to show the planet's name. The first step we'll take is creating a layout file for the TextView containing the planet's name. This will be called celestial_card_view.xml

<?xml version="1.0" encoding="utf-8"?>
<TextView xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/planetInfoCard"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_weight="1"
    android:background="@drawable/rounded_bg"
    android:gravity="center"
    android:orientation="vertical"
    android:padding="6dp"
    android:text=""
    android:textColor="@android:color/white"
    android:textAlignment="center" /> 

rounded_bg.xml is a simple shape drawable with rounded corners and a blue background.

<?xml version="1.0" encoding="utf-8"?>
<shape xmlns:android="http://schemas.android.com/apk/res/android">
    <solid android:color="#E61976d2"/>
    <corners android:radius="5dp"/>
</shape>

Next, create a new class named CelestialBody that extends Node and implements the Node.OnTapListener interface. This is what we will use in MainActivity, and it will contain the RotatingNode that was originally used in MainActivity.

class CelestialBody (
    val context: Context,
    val celestialName: String,
    val planetScale: Float = 1.0f,
    val tilt : Float = 0.0f,
    val clockwise: Boolean = false,
    val rotationSpeedMultipler: Float = 1.0f,
    val rotationPerSecond: Float = 90.0f,
    val renderable: ModelRenderable) : Node(), Node.OnTapListener {

The init block for this class will assign the listener to itself

init {
    setOnTapListener(this)
}

Additionally, you will need to declare three variables at the top of the class

private var infoCard: Node? = null
private val INFO_CARD_Y_POS_COEFFICIENT = 0.55f
private var planetNode : RotatingNode? = null

The planetNode represents the rotating planet model, infoCard represents the planet's name that will float above the planet, and INFO_CARD_Y_POS_COEFFICIENT determines how far above the planet model the name should float.

The onActivate() method is where you will create your name card,

override fun onActivate() {
    if( infoCard == null ) {
        infoCard = Node()
        infoCard!!.setParent(this)
        infoCard!!.isEnabled = false
        infoCard!!.localPosition = Vector3(0.0f, 
           planetScale * INFO_CARD_Y_POS_COEFFICIENT, 
           0.0f)
        ViewRenderable.builder()
            .setView(context, R.layout.celestial_card_view)
            .build()
            .thenAccept({ renderable ->
                infoCard!!.renderable = renderable
                val textView = renderable.view as TextView
                textView.text = celestialName
                infoCard!!.isEnabled = !infoCard!!.isEnabled
            })
    }

as well as your planet model

   if( planetNode == null ) {
        planetNode = RotatingNode(
            clockwise = clockwise,
            axisTiltDeg = tilt,
            rotationSpeedMultipler = rotationSpeedMultipler,
            degreesPerSecond = rotationPerSecond)
        planetNode!!.setParent(this)
        planetNode!!.renderable = renderable
        planetNode!!.localScale = 
           Vector3(planetScale, planetScale, planetScale)
    }
}

Return to the createCelestial() method in MainActivity, and replace the method with the following:

private fun createCelestial(celestialKey: String) : Node {
    val base = Node()
    val renderable = getRenderable(celestialKey)
    if( renderable != null ) {
        val celestialObject =
            CelestialBody(context = this, 
                celestialName = celestial, 
                renderable = renderable)
        celestialObject.setParent(base)
        celestialObject.localPosition = Vector3(0.0f, 0.5f, 0.0f)
        celestialObject.localScale = Vector3(0.5f, 0.5f, 0.5f)
    }
    return base
}

You will also need to create the getRenderable() method to return the proper renderable.

private fun getRenderable(celestialKey: String) : ModelRenderable? {
    return when( celestialKey ) {
        SUN -> sunRenderable
        MERCURY -> mercuryRenderable
        VENUS -> venusRenderable
        EARTH -> earthRenderable
        MARS -> marsRenderable
        JUPITER -> jupiterRenderable
        SATURN -> saturnRenderable
        NEPTUNE -> neptuneRenderable
        URANUS -> uranusRenderable
        else -> null
    }
}

At this point you should be able to run your app and see the planet's name above it. One thing you'll notice is that as you move around the model, the info card stays fixed, meaning it can be difficult to read from some directions.

To fix this, override the onUpdate() method and add the following code to cause the info card to always follow your device's camera.

override fun onUpdate(p0: FrameTime?) {
    super.onUpdate(p0)
    if( scene == null || infoCard == null ) {
        return
    }
    val cameraPosition = scene!!.camera.worldPosition
    val cardPosition = infoCard!!.worldPosition
    val direction = Vector3.subtract(cameraPosition, cardPosition)
    val lookRotation = Quaternion.lookRotation(direction, Vector3.up())
    infoCard!!.worldRotation = lookRotation
}

Showing More Planet Information

The final thing we'll do in this project, for now, is display some information about the planets when the screen is tapped. We can do this by creating BottomSheetDialogFragment that will contain the data for each object in our game. First, create a new layout file named info_layout.xml, which will contain a single TextView for displaying data.

<?xml version="1.0" encoding="utf-8"?>
<TextView xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/infoText"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_weight="1"
    android:padding="6dp"
    android:background="#E61976d2"
    android:textColor="@android:color/white"
    android:textAlignment="center" />

Next, create a new class that extends BottomSheetDialogFragment. For this example, I named it InfoFragment. This class will also contain a variable for the selected planet, a reference to the TextView from our new layout, and a map of information to display to the user.

class InfoFragment : BottomSheetDialogFragment() {
    var planet = ""
    lateinit var textView: TextView
    val info : Map<String, String> = mapOf(
        MainActivity.MERCURY to "Mercury is the closest planet to the Sun and due to its proximity it is not easily seen except during twilight. For every two orbits of the Sun, Mercury completes three rotations about its axis and up until 1965 it was thought that the same side of Mercury constantly faced the Sun. Thirteen times a century Mercury can be observed from the Earth passing across the face of the Sun in an event called a transit, the next will occur on the 9th May 2016.",
  
        MainActivity.VENUS to "Venus is the second planet from the Sun and is the second brightest object in the night sky after the Moon. Named after the Roman goddess of love and beauty, Venus is the second largest terrestrial planet and is sometimes referred to as the Earth’s sister planet due the their similar size and mass. The surface of the planet is obscured by an opaque layer of clouds made up of sulphuric acid.",
  
        MainActivity.EARTH to "Earth is the third planet from the Sun and is the largest of the terrestrial planets. The Earth is the only planet in our solar system not to be named after a Greek or Roman deity. The Earth was formed approximately 4.54 billion years ago and is the only known planet to support life.",
  
        MainActivity.MARS to "Mars is the fourth planet from the Sun and is the second smallest planet in the solar system. Named after the Roman god of war, Mars is also often described as the “Red Planet” due to its reddish appearance. Mars is a terrestrial planet with a thin atmosphere composed primarily of carbon dioxide.",
  
        MainActivity.JUPITER to "The planet Jupiter is the fifth planet out from the Sun, and is two and a half times more massive than all the other planets in the solar system combined. It is made primarily of gases and is therefore known as a “gas giant”.",
  
        MainActivity.SATURN to "Saturn is the sixth planet from the Sun and the most distant that can be seen with the naked eye. Saturn is the second largest planet and is best known for its fabulous ring system that was first observed in 1610 by the astronomer Galileo Galilei. Like Jupiter, Saturn is a gas giant and is composed of similar gasses including hydrogen, helium and methane.",
  
        MainActivity.NEPTUNE to "Neptune is the eighth planet from the Sun making it the most distant in the solar system. This gas giant planet may have formed much closer to the Sun in early solar system history before migrating to its present position.",
  
        MainActivity.URANUS to "Uranus is the seventh planet from the Sun. While being visible to the naked eye, it was not recognised as a planet due to its dimness and slow orbit. Uranus became the first planet discovered with the use of a telescope. Uranus is tipped over on its side with an axial tilt of 98 degrees. It is often described as “rolling around the Sun on its side.”",
  
        MainActivity.SUN to "The Sun (or Sol), is the star at the centre of our solar system and is responsible for the Earth’s climate and weather. The Sun is an almost perfect sphere with a difference of just 10km in diameter between the poles and the equator. The average radius of the Sun is 695,508 km (109.2 x that of the Earth) of which 20–25% is the core.")

The onCreateView() method will inflate our layout, and retrieve the reference to the TextView.

override fun onCreateView(
    inflater: LayoutInflater,
    container: ViewGroup?,
    savedInstanceState: Bundle?): View? {
    val view = inflater.inflate(R.layout.info_layout, container, false)
    textView = view.findViewById(R.id.infoText) as TextView
    return view
}

Finally, onStart() will take the text from the map based on the selected planet and display it in the TextView.

override fun onStart() {
    super.onStart()
    textView.text = info.get(planet) ?: ""
}

Returning to MainActivity.kt, go to the bottom of the file and create a new interface that will be used to relay the click event from our AR scene to the Activity.

interface InfoCallback {
    fun showInfo()
}

You can update the declaration of MainActivity to implement InfoCallback

class MainActivity : AppCompatActivity(), InfoCallback {

create an InfoFragment towards the top of the class

private val infoFragment = InfoFragment()

and then set up and show your Fragment when this callback is triggered.

override fun showInfo() {
    infoFragment.planet = celestial
    infoFragment.show(supportFragmentManager, "infoFragment")
}

In order for your app to trigger this callback, you will also need to pass it into the constructor for the CelestialBody object in your createCelestial() method.

val celestialObject = CelestialBody(context = this, 
        celestialName = celestial, 
        renderable = renderable, 
        infoCallback = this)

You will also need to update your CelestialBody constructor to accept this callback.

class CelestialBody (
    val context: Context,
    val celestialName: String,
    val planetScale: Float = 1.0f,
    val tilt : Float = 0.0f,
    val clockwise: Boolean = false,
    val rotationSpeedMultipler: Float = 1.0f,
    val rotationPerSecond: Float = 90.0f,
    val renderable: ModelRenderable,
    val infoCallback: InfoCallback?) : Node(), Node.OnTapListener {

Finally, in the CelestialBody class, override the onTap() function and have it trigger the callback, if it exists, whenever the Node is tapped.

override fun onTap(p0: HitTestResult?, p1: MotionEvent?) {
    if( infoCallback != null ) {
        infoCallback.showInfo()
    }
}

If you run your app now, you'll notice that information about whatever planet ends up on the screen comes up, however the background is dimmed when the bottom sheet comes up.

To get around this, return to your InfoFragment.kt file and go into the onStart() method. Using the following code you can override the Android dialog window's attributes, allowing you to remove the background dimming.

val attributes = dialog.window.attributes
attributes.dimAmount = 0.0f
attributes.flags = 
    attributes.flags.or(WindowManager.LayoutParams.FLAG_DIM_BEHIND)
dialog.window.attributes = attributes

Conclusion

At this point we have a custom game board, a game piece that knows where it is on the board, a cloud backend supporting our game, and an augmented reality Android app that provides information about various parts of our solar system. While this project has gone pretty far, there's still a lot of things that would be great for improving it. Things I'd like to come back to and finish include:

• Taking advantage of the code in place to change planet axis, rotation speed and scaling
• Reintroduce orbits around the sun when the Voyager tile is selected (the Google sample does this, but only this. I'd want to keep the functionality that was broken out for this project :))
• Along the same lines as orbits, having moons around the planets would be awesome, and a great first step when reworking with orbiting nodes.
• Integrate the Google Assistant to read off information, rather than hard coding it into the information fragment. I did give this a shot during this run, but using gRPC is more of a pain than I want to deal with right now
• Find a way to access the NFC data blocks from my Particle game piece so I can use any programmed tag, rather than checking against the tag address
• Add in support for more than one game piece (I could have removed the Xenon all together and just used the Argon I have, but I wanted to try their mesh network, too)
• Create a smaller game piece
• Make a nicer game board

All in all I'm pretty happy with how this turned out, as I had been sitting on the idea for a bit over a year, but never actually got around to making it. Hopefully it inspires folks to try some of their own awesome AR and connected hardware projects.