The making of Hulu’s A Curious Mind VR experience with Cream VR’s Andrew MacDonald
Andrew MacDonald is one of the creative directors of Cream VR. Prior to co-founding Cream’s VR department in 2015, Andrew directed music videos and, later, documentary and factual entertainment. Cream VR, based in Toronto, has created more than 20 pieces of original VR content, including 360 3D series and interactive volumetric experiences for agencies, brands, networks and studios.
Since 2015, Andrew has become a bit of an expert in VR production, showing his work at Cannes and speaking at NAB and the Vancouver International Film Festival, among others.
Immersive Shooter had the chance to chat with Andrew about his latest piece, A Curious Mind, and Cream VR’s big plans for its next project.
The Making of A Curious Mind
How did Cream VR working on A Curious Mind come to pass? Was it always intended to be VR?
Our CEO, Dave Brady, always talked about doing something of the caliber of Carl Sagan’s Cosmos in VR. He had some meetings with Hulu, as they were making a VR play as well. They’d gotten some money from Microsoft to create content for their MR headset launch, so they commissioned three pieces from us. One of those pieces was A Curious Mind with Dominic Monaghan as he travels through space, deconstructing scientific and philosophical questions.
What type of camera did you use to shoot it and why?
Everything in A Curious Mind is CGI except for our character. We wanted to shoot our host and then place him into that 360 environment, so we needed to shoot Dom as a plateA part of a composite, usually intended as a background, though here it refers to a duplicate version of the main shot w... More and build the rest as a CGI set.
To capture Dom, we did a stereo 180 shoot in a green screenA subject shot against a background of a particular shade of green so the background can be removed and replaced in post... More environment. We used two Black Magic Micro Studio cameras equiped with Lumex and Izugar lenses. Since it was stereo and we needed the cameras to be as close as a pair of eyes, we couldn’t use larger cameras. We also needed gen-lockable cameras and with the Black Magic sync boxes, we could get scan-line sync (even though we couldn’t get global shutterA camera shutter that opens and closes all-at-once, exposing the entire sensor simultaneously (as opposed to a rolling s... More) for the cleanest picture possible.
Cream VR used two Black Magic Micro Studio cameras equiped with Lumex and Izugar lenses to capture Dom.
What was the biggest challenge you had to overcome in the making of A Curious Mind?
The biggest challenge was the motion trackingA technique for recording the movement of an object within a scene, to hide or paint that object or to affix a composite... More. We had initially looked at using motion control equipment to mechanically track the elements, but it was going to be very expensive. At the same time, we were sort of on the cusp of an alternative technology, so we took a gamble and said we’d solve it afterwards using PFTrack and Canvas 360.
PFTrack analyzes a piece of footage to see how the pixels move and then generates a depth mapDigital data representing the distance of the objects in the scene from the camera. of the scene. We then used those depth maps in combination with Canvas 360, which enables you to work in a skybox environment.
The Cream VR team taped a pattern on the green backdrop to make tracking in post easier.
We hired Thought Cafe to takeIndividual instance of a shot; a take = each time the camera is started and stopped. the lead on the creation and execution of the background 360 motion graphics and after great debate We were convinced we’d be able to solve all our motion trackingA technique for recording the movement of an object within a scene, to hide or paint that object or to affix a composite... More in post. That’s why we put up green tape targets in the background because it gave the cameras more information on all the pixels and how things were moving. Then, we used those software tools to analyze the footage in post and solve the camera motion.
We also used Cinema 4D in there, as well. But, we were having a problem with it because we were shooting Dom in stereo with wide lenses. Because it’s VR, we can’t lose his feet. If we moved the cameras farther away, he’d be really small in VR. In order to get his feet and stay close, we had to tilt the camera, so we ended up with inclination in our shots. When we tried to solve that, Cinema 4D solved it but Dom was then leaning backwards. We ended up reaching out to Maxon, and they were able to give us an unreleased patch for Cinema 4D to get him standing vertical.
In the behind-the-scenes video, your team mentions Mettle by name. Can you talk a bit more generally about your post production workflow on the piece and why/what features of Mettle were so instrumental?
Being able to work with Mettle in Adobe After Effects gave us the ability to do all the elements we needed in Cinema 4D and similar software and do the final assemblyAn extremely rough edit of shots that places them in the order they appear in the script. in After Effects with Mettle. RotoscopingDigitally modifying video by tracing objects within the image one frame at a time. and keying was integral to get Dom [Dominic Monaghan] into the scene. We used mostly AE functions, but Mettle let us work in a Skybox environment.
Now that A Curious Mind is done, what will you be working on next?
Behind the scenes on Cream VR’s volumetric game project, Be Curious
We’re actually working on an extension of A Curious Mind, also with Dom, called Be Curious. It’s sort of Cosmos meets the Holodeck(from Star Trek) A chamber where a person is immersed in a holographic simulation of a physical environment indistinguis... More. We’re trying to put all those grand concepts of science into a volumetric space you can interact with, in a piece that’s both a learning experience but also entertainment.
Cream has always done that kind of work, and so have I for that matter, working with the Discovery and History channels and National Geographic. We want to bring that skillset into the VR space.
It’s a mix between factual entertainment and a documentary–a new product in between a game and a documentary. To gamify it, we’re making it so the viewer makes certain discoveries based on science. For example, you and Dom have to measure the speed of light using the same tools–mirrors and a fan–that Foucault used in 1847.
How are you capturing/building that?
For Be Curious, Cream VR has already created a high quality, low polygon 3D avatar of Dom
We’ve already created a high quality, low polygon 3D avatar of Dom. Our creative director, Tristan Cezair, had an idea to use photogrammetryThe use of photography to survey and map a location from a variety of angles. and then use a point cloud to make the model and end up with a still volumetric model of a face. Then, we rigged it with bones and shot a motion capture performance that would ‘drive’ the performance of the volumetric face and thanks to the innovations in machine learning facial motion trackingA technique for recording the movement of an object within a scene, to hide or paint that object or to affix a composite... More we now had clean video (no tracking dots) that we could project as a texture onto the animated 3d face.. To do that, we teamed up with Sheridan College in Toronto to use their motion capture studio. We built some rigs out of Black Magic cameras to shoot the performance.
For A Curious Mind, we used platforms we already knew how to use, like After Effects. But, that’s the old-school film way, sticking plates together. When you do it that way, you’re locked in. For Be Curious, we’re building the whole set in a game engine so the user is free to roam wherever they want and interact.
When do you think that project will come to fruition?
The setupA camera position for a given scene. You might shoot more than one shot from a single set-up (wide shot and close-up). Cream VR used to capture the video underlay and photogrammetryThe use of photography to survey and map a location from a variety of angles. overlay they use to animate Dom in Be Curious
We aren’t sure yet. We received some funding from the Canadian Media Fund to build a teaser that we’re using to propose building this volumetric game that we’re putting together over the next two months. Then, we’ll be looking for another $1.1 or $1.2 million to build the full game.
Canada has actually invested tens of millions on VR over the last two or three years, but like everyone the path to ROI are still being developed. We know our area of expertise , we have a celebrity attached, and having studied the market since its inception we are homing in on a business model with a good ROI We hope it’ll be entertaining enough to do well on SteamVR and educational enough to be used as a part of schools’ curriculum. Since our avatars are well optimized and low poly.
The thing is make a good piece of content and put on as many platforms as possible. We’ll have the full experience on PC VR, an optimized experience for standalone VR, and then a fun mobile phone AR game.
About Andrew MacDonald
From left: Andrew MacDonald, Dominic Monaghan and Tristan Cezair
How did you first get into VR?
As a freelancer, I shot for Cream several times. During a slow period in the summer of 2015, I pitched them an idea for a show. He said he wasn’t interested in that, but he asked me if I’d heard of VR. I told him I’d heard of it and I understood the process. He said they wanted to get into it and asked if I could produce a VR short.
I pitched Dave the idea for Agnus Die and gave him a bid to build a camera and shoot Agnus Die in one month. I found a great motion graphics guy Tristan Cezair who was already working in the VR space, and together we teamed up to build our own GoPro rig and shot Agnes Die. In that film, we made a VR tribute to the shower stabbing scene from Alfred Hitchcock’s Psycho, which was a very complex scene to shoot in that time period. To recreate it in VR The main problem was we couldn’t get close to anything because of the overlap issues of the cameras, so we modified the GoPros with Entaniya lenses so we could put the rig in the shower and come out with a complete stitch.
Screengrab from Agnes Die
That piece got us a bit of notoriety. Lewis Smithingham from 30 Ninjas found it and told me how it had further confirmed Doug Liman’s belief that narrative VR could work.
“When we started working on Invisible, there were very few examples of narrative (read: fictional) 360. The few that did exist were pretty much all shot POV or in long takes. We knew we wanted to do something different. We wanted to use editing. We then saw Agnes Die. We were so stoked. The final cut in particular is incredible. There is a cut that requires the viewer to look down in order for it to land perfectly. And everyone always follows it. It’s the perfect example of 360 video and cinematic editing. It’s a brilliant piece and remains one of my favorite pieces of 360 video to this day.”–Lewis Smithingham
Then, we used that camera to shoot behind-the-scenes pieces for shows Cream was working on, specifically for the Smithsonian and Discovery. Discovery saw our behind-the-scenes VR pieces on their big Civil War series and asked us to do a bespoke VR piece of ancillary content.
So, a one-month project turned into two months, which turned into three months, which turned into three years. It’s nice that we can be small and nimble and that we can operate within an existing company that already has revenue and clients.
How many people make up the VR department today?
Screengrab from A Curious Mind
5. When we first started, I I quickly realized how reliant VR is on post. That’s when I found Tristan, who was an early adopter in VR, knew the post production side really well, and had a background in CGI. Together we had all the skill sets needed. All the pieces we’ve done–even Agnes Die–include some form of graphics and effects. A Curious Mind took that as far as it could possibly go!
So, you started with a GoPro rig. From that point forward, what has your ‘camera evolution’ been?
When we first started looking for options, we decided not to go with Ozo or other expensive cameras because we thought the camera market would be changing so fast that by the time we’d buy an Ozo it would already be outdated. It’s a brilliant camera, but it’s the kind of thing you rent rather than buy. A few people bought them and have been burned by that decision.
That’s why we started with a GoPro rig that we then modified with Entaniya lenses. Then, at NAB 2016, I met Ken Ishimoto from Entaniya and asked it they could make a larger format of their lens. They later came out with the micro 4/3s version of their lens, and we got numbers 6, 7, 9 and 22 off the production line. And that’s what we’re using currently for 2D 360. To me, as a cameraman, I know the cameras will always change but glass is at its peak technology. This way, we can takeIndividual instance of a shot; a take = each time the camera is started and stopped. those lenses off last year’s camera and put them on this year’s camera.
Download Cream VR’s app to view their immersive work.
If we ever shoot in 3D, we go and rent an Ozo or Kandao Obsidian or Insta360. We aren’t buying any 360 cameras right now because the technology is changing so rapidly.
The latest thing is cameras that can shoot six degrees of freedomThe ability to move left-right, up-down, and forward-backward (in addition to being able to rotate around the x, y, and ... More. To be considered for a film festival these days, it seems like that’s the gimmick you have to follow. For example, by the time we finished A Curious Mind, it seemed film festivals were only interested in 6 DoF content.
That’s very frustrating and limiting for filmmakers. In three years, we went from 2D 360 to full-blown 6 DoF. There just hasn’t been enough time for people hone the craft to make that “hit” content for each new medium. There’s so much we have yet to discover.
Speaking of discovering what works and what doesn’t, do you have any lessons for new immersive shooters?
One piece of advice is to focus on the 180. When we’re shooting 360, I envision that the audience is in a fixed chair. Even though you have 360 degrees to play with, you don’t want to throw things everywhere. By focusing on the 180, it’s easier to assume where the viewer is looking. That doesn’t mean that you can’t use the full reverse, but you have to use it cleverly and intentionally.
What do you see as the biggest hurdle for VR taking off?
User base. We thought that with mobile VR, everyone would have a VR viewer in their pocket. That was a neat idea, but it’s over now. Cardboard was a novelty, but where VR becomes a useful product and a rich experience is in a proper headset. For VR to takeIndividual instance of a shot; a take = each time the camera is started and stopped. off, we need a user base of more than 1 percent. But, I think headset manufacturers are heading in the right direction Sony’s PSVR is an affordable entrypoint and easy platform to use and with the upcoming standalone headsets like Santa Cruz and the Vive Focus, will bring in a whole new group of people.
A Curious Mind is available on Hulu’s VR app.
