Post by Joe Strickland (2017 Cohort)
Back in the summer of 2018 I attended the DEN summer school in Bournemouth. One of the big draws of the summer school for me was the programming in Unity course that was being offered. Having come from a psychology background, I had no programming knowledge but it was becoming clearer and clearer that this was going to be something that held me back during my PhD, especially when it came to prototyping ideas for experiences. The course itself was pretty good, we ran through several different elements of using Unity including the basics of building scenes, game object physics, and exporting our scene onto a smartphone and viewing it with a cardboard header as a VR experience. We also started using Vuforia and making basic AR content. This workshop gave me a good basic understanding of Unity, but more importantly, it showed that what I wanted to learn and eventually make was well within my grasp. This was very important for motivating me to carry on learning how to build Unity experiences, as well as code in general.
Once the summer was over, my supervisors and I sat down and started discussing short-term goals to get me learning everything I’d need to learn in order to build interactive AR experiences myself. The first of these goals was to learn Python and C# in order to understand the logic of coding and be able to write my own Unity scripts to control different elements of the software. My supervisor ran me through all the basics in Unity that I might need for the specific things I was going to make, a welcome refresher after the summer school course, and I was sent off to learn my languages. Personally, I found Python quite easy to learn. The logic of the language made sense to me and the online resource I had been recommended taught it in a very hands on a practical way, with many small assignments to try out new coding knowledge and to keep old knowledge fresh and reinforced in your memory. Also, the course was broken up into bite size chunks and I found doing a lesson a day over the course of a month a very productive way of learning this language.
C# scripting was a little harder for me to grasp. I don’t know whether it was the difference between it and Python throwing me off or knowing that having to learn this was going to be more important for my PhD, but it took a lot more to try and figure out what I was doing with it. Learning this was done through some of the Unity provided tutorials, as well as other user generated tutorials on YouTube. I was also learning how to use Unity to specifically make the first short term goal project I had been assigned; making videos plays in Unity. The Unity video player isn’t completely user friendly and it took a lot of trial and error and searching Unity message boards and community sites to find out how to get it to work in the way I wanted it to. Having got it to work I moved on to controlling it a bit more and building an experience where the audience can press keys to trigger the playing of different video clips. I crafted a game object for each video clip we had and had them generated and destroyed whenever we needed that video playing, depending on the input of the audience. What I ended up with was a functional interactive film about a man trying to find his heart medication, where the audience could decide whether he moved left, right, or had a heart attack at various points in the film. When I showed my supervisors they liked it but found how I had made the film incredibly inefficient, so they tasked me with remaking it so that different videos played on the same game object and not on different ones. This next step proved challenging but eventually I managed to write a functioning Unity script which changed the state of the game object and, once a game object was of a certain state, it would play different videos with different audience responses. It would then change its stage again to allow the experience to progress. This experience pleased my supervisors, but they didn’t like how making decisions at the wrong times messed the game up, so I had to add delays into the script that stopped audiences making decisions at the wrong points in the experience. Fortunately, this wasn’t to difficult to do, although trying to use time as a function while coding with the video player in mind did prove confusing.
I was also asked to build a restaurant scene and fill it with moving virtual characters, but this was very similar to the summer school exercises and the Unity developer tutorials so this didn’t prove too tricky. Characters were downloaded from Adobe Maximo, so came with animation cycles attached and a few YouTube tutorials later I had people looking around and being furious at virtual restaurant tables.
Finally, I was asked to build an AR tester experience. I had to place a virtual character, like those from the restaurant scene, into a real world environment and have them occluded by a real world object, specifically sitting behind and hidden by a real world table. This is something that is surprisingly hard to find official Unity information for. There is lots of help for tracking markers and placing AR content in the real world but not so much for having that content blocked by real objects. I eventually found a YouTube tutorial which addressed a similar problem in a way which allowed me to figure out how to solve my own. They showed how there was a depth occluder material that you could use to create invisible game objects that would block the audience’s vision of the virtual content. Creating a cube the size of a table top and placing it over the lap of my sitting virtual character, then using a placemat as a tracking marker in the real world to position the avatar behind a real table allowed the virtual character to appear as if they were sitting in the real world. The illusion was particularly impressive when the character moved and there arms would disappear and reappear below and above the line of the back of the table. See attached photo for a snapshot of the experience.
If I had any advice to any other researchers looking to get into creating XR experiences, or even just learning to code, it is there’s no time like the present to start learning. There are plenty of great resources online for free that go through everything you’ll need to know step by step, while also allowing you to navigate through lessons to learn the specific things that you need for whatever project you might be working on. Though getting an understanding of the basics is fundamental you can pick or choose what of the more specific stuff to learn to suit your needs fairly easily. Also, just like any skill, you’ll need to keep practising. Find some little challenges to work towards, like I had set out for me. There were a few times I’d not focus on coding for a few weeks and then notice that I had forgotten something I definitely knew before and had to go back over previous lessons or code that I had written to find it. Don’t fall for this like I did, keep it up at a steady pace and you’ll be writing code in no time.