Wait, there’s a talking goldfish!? These are the killer VFX scenes from s2 of ‘The Umbrella Academy’

Source: Befores and Afters

THE MAKING OF THE OPENING BATTLE ONER, BABY POGO, A FISH-HEADED ANDROID, GHOST BEN AND THE FINAL CONFRONTATION.

SPOILER WARNING: There’s a talking goldfish on the head of an android, a baby chimpanzee, an end-of-the-world battle and a whole lot of other visual effects in season 2 of Netflix’s The Umbrella Academy, created by showrunner Steve Blackman.

Key points:
– The bluescreen backlot for the opening scene
– Making AJ Carmichael’s fish bowl head (and fish)
– How Weta Digital crafted a Young Pogo
– Mocap ghosts!
– Why unexpected snow meant a lot of roto for the final battle

That work was overseen by visual effects supervisor Everett Burrell, who returned from season 1 to co-ordinate a massive VFX effort from several effects vendors for this new season, which largely takes place in 1960s Dallas, Texas.

After being blown away by the huge range of VFX work in these newest 10 episodes, I decided to ask Burrell about just five specific scenes and characters and how they were accomplished. Here’s how the VFX supe broke them down.

1. A ‘bluescreen playground’

In season 2, the Umbrella Academy siblings end up in different years during 1960s Dallas. When Five (Aidan Gallagher) also arrives in the past, he witnesses his family fighting Soviet soldiers on the streets, right before a nuclear blast hits. The battle sequence was a key early set-piece for the show, and, as Burrell told befores & afters, was designed as a oner by episode 1 director Sylvain White to showcase the different sibling powers.

The action takes place on a Dallas street, which was actually a location in Hamilton, Ontario (the street sans-battle impact served for many scenes during this season). However, since so much destruction takes place in the battle, the ‘oner’ was actually filmed on a bluescreen backlot set.

“We started by Lidar’ing our master location street in Hamilton,” outlines Burrell. “We brought that into ‘CG computer land’ and then built a virtual version of that set. Sylvain could, on an iPad, walk through that location to plot the shot out. Then we put a bunch of pieces in; a tank, Vanya and other things around and let Sylvain walk around with a virtual camera and get a sense of everything. Spin VFX took these virtual assets and used them to build a previs.”

From that previs, it was quickly determined that the Hamilton street could not be used for shooting the battle. “It was too dangerous,” says Burrell. “The tank was too heavy. It would probably fall into the sewer! We couldn’t destroy the storefronts. We couldn’t control the street for four or five days or two weeks solid to build and prep. So, instead, we built a 360 degree bluescreen on our backlot in Toronto. We called it the blue screen playground.”

“We represented the street size and where the storefronts would be. The only thing real on that street was a couple walls and then the ground and the cars and the actors and the extras. Beyond that was all CG based off the Lidar scan of the real street.“

Filming all the elements for the sequence proved to be a major logistical challenge. Notes Burrell: “We also had to shoot all of Aidan’s parts first, because he’s a minor and can only work so many hours. Then we had to bring in Cameron Britton as Hazel on a separate day to shoot him separately because he was working on another project.”

About eight different sections would ultimately be stitched together by Spin VFX. The studio’s principal effects shots also included Vanya stopping a tank shell, Klaus raising the dead, ‘Ben Ghost’ exhibiting tentacles, Alison ‘rumoring’ three Russian soldiers until they explode, and several other sibling effects. Finally, there was the atomic cloud, which forms into an umbrella shape, from the nuclear blast.

“The sequence was one of the hardest, most complicated, things I’ve ever done,” reflects Burrell.

2. A talking fish, in a bowl, on top of a body

Heading the Commission — the agency that keeps tabs on ‘time’ — is AJ Carmichael, a goldfish inside a robotic body. Realization of the character would combine an on-set actor performance with a CG fish bowl and fish realized by Weta Digital.

Based on initial designs and R&D work, Burrell had Frank Ippolito at Thingergy Inc. build the neck and the bottom of the bowl, which was worn by the on-set actor. The actor also sported a gray hood with tracking markers on it while performing in the scenes.

“We then gave that to Weta Digital and they replaced this with CG glass and the CG fish,” explains Burrell. “They would produce the dynamics, the water, the bubbles, including when AJ smokes that cigarette — when he intakes the smoke into the bubble in the water and it becomes smoke rings to get his nicotine fix.”

“Weta Digital did a phenomenal job making it happen,” adds Burrell. “The experiments and the R&D they did was just amazing. The bubbles in the water and the splashing of the bowl and all the reflections; I think it’s a phenomenal piece of work.”

For a scene in which Five smashes AJ’s head with a cricket bat, actor Aidan Gallagher first rehearsed the scene with the actor performing AJ in the plate. “Then,” says Burrell, “I pulled the AJ actor out and let Aidan really swing through it as if he was smashing the glass and going through the water and the glass. Once we got Aidan’s take done, we pulled the AJ actor back in and then he would act to the performance that he saw on the monitor. Then Weta Digital simulated that smash.”

3. Return of Pogo

The first season of The Umbrella Academy leveraged Weta Digital’s extensive experience with creating CG primates to help deliver Pogo, aided with an on-set actor and voice performer. This season introduces young Pogo, also a Weta Digital creation.

To film scenes featuring young Pogo, Burrell knew that having an on-set performer there to interact with other actors and serve as a guide for Weta Digital’s team would be crucial. But finding the right performer became a tricky task.

“There was one actress who had worked on the Planet of the Apes films, she was a stunt performer, but she was booked on the new Ghostbusters movie,” outlines Burrell. “It had to be someone who was four feet tall and be able to perform and act and emote the right way. Young Pogo didn’t have any lines but we still needed the stand-in to perform for the other actors.”

“In the end,” continues Burrell, “it was a combination of two different players we had on set. One was more of a stunt person. One was more of a performer, like a gymnast. Between those two actresses we were able to get that reference performance. We’d shoot the actor playing Pogo in a gray tracking marker suit, and then we’d pull them out and do clean plates and then our reference passes. Also, Ken Hall came back and played Pogo again in episode 10, because there was a flashback to old Pogo.”

Young Pogo is seen in a trip to space, almost perishing from a fiery re-entry. Somehow he is brought back to life by Reginald Hargreeves (Colm Feore). This moment includes an extraordinary close-up on the CG Pogo as the chimp’s wounds are healed. “What’s interesting about Pogo being healed,” advises Burrell, “is that Pogo’s eyes change from normal chimp brown eyes to the same Pogo eyes from the first season. Here, he’s evolving into more of a human so we wanted his eyes to be similar to season 1.”

4. Mocap’ing with ghost Ben

‘Ghost’ Ben (Justin H. Min), who has been tailing Klaus (Robert Sheehan) for several years, is at one point allowed ‘inside’ Klaus to converse with a woman he has fallen for. However, Klaus later attempts to extract Ben, leading to shots of Ben’s ghost form emerging partially out of Klaus’ body. Making those shots possible involved motion capture, meticulous match-moving, 3D digi-double animation and other effects from Spin VFX.

“We had Robert Sheehan as Klaus hooked up to a real-time motion capture suit that was able to capture his movement under his wardrobe,” describes Burrell. “We had witness cameras as a backup, but it was a neat new tool to be able to use.”

“Then we had a really detailed photoreal CG version of Ben, which Spin had built for the tentacle shots in the opening battle, that was match-moved and tracked. We also did a pass with Justin doing weird distortions and weird motions as great reference. We had that to lay on top of Robert in the Avid when we did our temp pass. Ultimately we tracked the CG Ben in based on Robert’s performance.”

Burrell notes that Sheehan was highly successful in miming how to pull Ben out of his body and have it be pushed back in. “We had shown Robert all the R&D and all the tests that we had done. That was one of the things we had in the CG test where we’d pull the Ben ghost by the hair and try to get him out of Klaus, and Robert really picked up on that. It’s his performance that really sells it, but I think it’s also a great looking digi-double, and it’s so tightly match-moved because of that suit that Robert was wearing.”

5. The final confrontation, snow, and a lot of roto

The siblings end up facing off against a horde of Commission agents on a farm, a location where Vanya (Ellen Page) has gone to save a boy, Harlan, who has inherited her destructive powers.

The Commission agents ‘materialize’ at the location and make a push forward with weapons, only to be stopped in their tracks by Vanya. Dozens of extras were filmed at the shooting location, where, recalls Burrell, a ‘huge problem’ emerged.

“We shot that scene mid-November out on the farm. Half the scene was shot out on normal grass out in the field and then, it snowed. And it never stopped snowing. So, for half the scene, we had to roto half the actors out and put in grass. And then the other half of the scene we had to do the opposite, we had to put in snow for the reverse shots.”

To accommodate the fallen snow, production came up with the idea that Harlan’s powers were so uncontrolled that he had caused his own weather system which subsequently creates an isolated snowstorm. “But that was written in after the fact, because that was never the intent!” attests Burrell.

The number of Commission agents was enhanced thanks to crowd simulations from Spin VFX. Burrell shares that, “I said to Steve Blackman, ‘Well, how many agents show up? 50?’ He goes, ‘No, no, no.’ I go, ‘A hundred?’ He says, ‘No, no, no. Like, 4,000.’ When we put in a thousand into the scene to see what it looked like, it looked like only a hundred. So then it became like 8,000 agents back there…”

“We scanned all the extras who played the agents,” says Burrell. “Spin then really rose to the occasion with the crowd sims. Plus, all the guns and briefcases had to be animated. And then we had to do all that roto of the live-action actors running to make them run on grass instead of snow. The befores and afters of that are quite fantastic!”

In the end, several VFX studios would contribute to season 2’s extensive effects, including Weta Digital, Spin VFX, Folks, Soho VFX, Pixomondo, Deluxe VFX, Digital Film Tree, BOT VFX, Studio 8, Exceptional Minds and MARZ.