The Umbrella Academy: Weta Digital Discusses Pogo and Carmichael VFX

Source: comicbook

Founded by Peter Jackson, Richard Taylor, and Jamie Selkirk in the early 1990s, Weta Digital has taken the past 30 years to turn into a global leader in visual effects.

The company has won a handful of Academy Awards and has worked on most of the biggest productions Hollywood's ever seen, with the likes of Avengers: Endgame, Godzilla vs. Kong, James Cameron's Avatar franchise, and virtually every movie in the DC Extended Universe — and that's just beginning to scratch the surface.

Most recently, upwards of 200 of the company's employees worked on the second season of Netflix's The Umbrella Academy, on shots involving the show's two most labor-intensive VFX characters — Pogo and AJ Carmichael, the talking goldfish. Leading the way for Weta on The Umbrella Academy were visual effects supervisor Chris White, art director Gino Acevedo, and animation supervisor Aidan Martin.

Earlier this month, we hopped on a Zoom call with the three to chat all things Umbrella Academy, from transforming Pogo into Baby Pogo to the cigarette-smoking Carmichael. Keep scrolling to read through our chat. 

S1 vs. S2
ComicBook.com: I'm not sure if you guys saw the little show that came out last year called The Mandalorian, but you have a thing like Baby Yoda. Then Umbrella Academy comes out and you say, "Gee dang it. Look at that, it's Baby Pogo!"

You already had Pogo built for Season 1, so on the surface, how different was it going back to Season 2? Were you able to reuse the assets and works? Or is it a completely different thing and you had to build Baby Pogo from the ground up?

Chris White: He was pretty much built from the ground up because he was so much younger; just kind of like a toddler or slightly older. That was part of the thing that Gino and I worked together on the most in the beginning because Gino's art director at Weta on Umbrella Academy. We received some maquettes from the client of the designs for Baby Pogo, but Gino had to do some paint overs because part of the things that we wanted to make sure we did was use those for inspiration but also make sure he's anatomically correct.

Also, part of the work that Gino did was incorporate some of the nuances for older Pogo into Baby Pogo. Even some of the pigment that you see in older Pogo, some of those same spots, we carried those textures back to younger Pogo. Just little shapes. Gino can talk more about it. Even though he's quite different looking that you can see the older Pogo in him when he comes up.

Gino Acevedo: Yeah. It started with an original sketch that a good friend of ours did — the original sketch for old Pogo and for young Pogo, and that's where we got the actual maquettes from. But, as Chris said, then we had to go back and look at it and try to incorporate some of the Pogo back into young Pogo so that he's definitely related, obviously.

It was a lot of fun just going back and really studying how the pigmentation of the skin actually works. Even going back and looking at human stuff. How we get age spots and things, and how they might start out as a very simple blemish but as you grow older it ends up growing into more of an age spot. That's the way the pigmentation and the chimpanzee skin works as well.

CW: Yeah. There's one other thing too. You mentioned about the cuteness factor because Baby Yoda was out at that time. We had a requirement that he's got to be pretty cute. One thing that Gino had suggested is that baby chimps do this little pucker with their lip. Really little, small... I can't describe it better than a little pucker. That was something that wasn't in his original design that we had suggested adding into him. It's a subtle thing, but it just added to the cuteness factor of it.

Trips to the Zoo
How many hours of reference video do you think you watched to get Baby Pogo just right?

CW: Oh, jeez. We're always watching reference.

Aidan Martin: Well, that's an interesting question because we've been looking at apes and baby apes for a long time. So we didn't have to go back far in the reference archives to find a lot of cute reference. We had a lot of similar baby chimps on the last Planet of the Apes, so we already had a really big stockpile of baby chimp reference at that point.

For Baby Pogo, in particular, because we were generally matching what the on-set performer did or the principal photography shot, we weren't going outside the bounds of... Baby Pogo had position A and then by position B, at this time... We weren't going outside the bounds of what was already set out in the footage.

We're really lucky here to have our lead animator Craig Young, who's been doing a lot of the motion capture for us, and he has been doing chimps since the day he was born. He is actually a monkey. So we're really fortunate to have Craig who we can just take down to the mocap stage and just be like, "All right, man. We got to get from here to there as chimpy as possible," and it's just as easy as that to get Craig to move around. "We need it a bit faster. Can we add a little step or a skip there?" or something like that.

In terms of references concerned, we use a lot more reference for the facial this time. Because we weren't going off Adam Godley's facial performance, we weren't actually matching the voice track this time, we used a lot more chimp facial reference for baby Pogo's face. I guess, we're not scouring for hours looking for new reference. We have our go-tos at this point, especially for someone like Pogo, or there's the cute-chimp-ref file and we know where to go at this point.

Look and Feel
You did bring up the Planet of the Apes movies. I think you guys did work on Jungle Book and Space Force. What's really interesting to me is there are all these different franchises, and chimps and primates are real things, but yet you three managed to make sure Pogo has that Umbrella Academy look and feel.

AM: Especially, with someone like Pogo, he's quite stylized, he's quite characterized, and that's even from the original contact drawings. Well, even from the comics, he was quite stylized. Then to the original artwork for the series, he's very stylized. Then we get the maquettes, you can see he's not your average-looking chimp. He still has his own unique features and that stuff just builds up.

In fact, Gino can attest to this, we normally start with something that seems quite character-driven and then we generally spend a lot of time trying to make it more realistic by getting an actual chimp skull scan, putting it into the digital maquette, and then adjusting the maquette so it actually fits into the real world of what an actual chimp does.

GA: Even some of the other things that we have, model-wise and for references... I've got a huge collection of ape stuff. This is a chimp hand that we use for a lot of reference and here we got a chimp foot that we use. Anatomically and texture-wise we can't get any better than using the real thing. We've got a fantastic relation with our Wellington Zoo that allows us to get in there... not to get in with the chimps, but to get up close and personal with the chimps and to be able to get some great photographs and great video footage and things like that.

Since Pogo was a character that crossed over from Season 1, at what point did the Weta team come in? They order Season 2 and they form the writer's room and start working the script. At what point are you guys notified? Did you start working on the assets prior to them actually filming since you kind of knew what you were getting?

CW: Yeah. This season was actually kind of interesting because it started pretty quickly that we actually started working on Baby Pogo... it had quite advanced before we received footage of the actual shots that he was in. It was so much of the case that we used some of the shots from Season 1... There's a shot of Vanya eating porridge at the table while Hargreeves and older Pogo are standing behind him.

We took that plate, painted out Vanya, and put in Baby Pogo eating the porridge at the test to send through to Steve Blackman to see how he was already looking before they had shot the footage. There was another shot we did as well that we add him too. But that was one of the cases where we wanted to get him in context for them to see even before they had shot some of the scenes. It was slightly different from this one. We were moving pretty fast.

GA: That was so cool to see. The first time that I saw it all come together, it was just really exciting to see the whole thing from concept to the model, to texture, and to finally see it in the shot with Vanya and stuff like that. I thought, "Oh, this is going to be really cool."

Own Worst Critic
I want to touch on that because when I'm writing or I'm typing something, I think I'm my own biggest critic. I go back and change a word here and a word there. As a visual effects artist, how do you determine when something's done? The studio tells you, "Done," and you're like, "Fine. I'll put it up?" When do you, as the artist, decide, "All right, I think I'm done with this?"

GA: You see it on Netflix.

AM: The cynical answer is time and budget. I'm sure there's a better answer than that.

CW: Yeah. It's always a tough one because some shots you just know. You're like, "Okay, that's pretty good. That's working." Then there are other ones where there's always something bugging you. That's the hard part, is figuring out what's that last little bit that's just not right. When you're making this stuff there's always some things that come up that you just don't even expect.

There are ways in lighting Pogo where it's just the way his hair parts at the top can actually give us a little bit of grief the way light hits the top of his hair. Little things like that that people wouldn't know when you're working on these type of projects. But there are always little things that you have to work through and figure out.

AM: Yeah. Artistically, you can really... When you're done with a shot, you can go, "Yeah, I nailed that. I feel really good about it." Then somebody else will just have a bit of... something will bug them about it and you'll just have to keep going back again and adjusting and adjusting. It's always the shots where there's the interaction between digital creatures and real persons. Those shots... those are always really tricky to make believable and make you feel like it's actually happening. Those ones are always ones that take the most time.

CW: Yeah. There's one shot where Vanya hugs Pogo... I think it's the first episode of Season 1.

AM: Season 1. Yeah. It's like the third shot or something.

CW: It's the third shot. It took one of the longest times of her just interacting and touching his back. It took the longest time just for that particular part. Even though it was a bit challenging with baby Pogo hugging Grace... Even after we finished lighting it, I think in the composite, we were working for another month or so just getting those two to look like they're interacting together.

AM: Or even Grace's hand and Pogo's hand coming into her hand, we kind of just kept going around in circles for the motion on that one before we even got to lighting. That was a pretty tricky one.

GA: I think it's funny because with the performance capture people think, "They just put the digital model on top of that and it's done." But, as it's just been explained, there's so much work and so many tweaks, and going back in there and just massaging it just to make it look perfect.

AM: Yeah. It's that last 10% always take a long time.

Absolutely. Gino, you bring up a good point. People flip it on Netflix and they'll say, "Oh, that chimpanzee is certainly a visual effect." I know I'm talking to you three on Zoom right now, but Weta Digital's much more than three people. Do you have numbers of how many people actually worked on this project?

CW: I think the core crew was around 50, but I think it was over 200 that touched it by the time we were all done.

Introducing Carmichael
As if Pogo wasn't comic-booky enough, Season 2 also introduced AJ Carmichael. So, let's talk about that. The first question I want to ask is how you manage to bring someone like that to life? As I understand, there was someone who served motion capture on the set, but I also think you guys had office pets if I understand that correctly.

GA: AJ and Carmichael. Because we're always trying to match real-life reference, in this case, I was sent out to the pet shop to go do some hunting around for some fish. First of all, I had to do some research to find out exactly what kind of fish that Miles, who did the concept design... what kind of fish that he used.

It turns out it was a special kind of a goldfish called a Shubunkin goldfish, which is related to the coy. Luckily, our local pet store had some. So I went in there and they 80 goldfish swimming around in here and I had to pick out a couple. At that time, we weren't quite sure about the color patterns, were we Chris? Because they wanted either just a goldfish or they wanted to have calico kind of colors, but they didn't want it to look too diseased or anything, wasn't it? Something like that.

CW: Yeah.

AM: Or wanting it to look too Finding Nemo either.

GA: Yeah. So I bought one of each and named them AJ and Carmichael, and they're still here. In fact, after we're done here, I've got to go feed them.

CW: I guess, the point also with the fish is, and Aidan can talk a little bit about how they were referenced for animation... but they were referenced for lighting and textures. If you have viewers that are interested in visual effects or comics, one of the things that I would point out is that we spent a lot of time doing research.

As we're doing our characters, we do a lot of research into how they look and history of them, what these type of fish are, how their bone structures move, how to make them look most realistic. But then there's also the creative part. If you look at AJ, the spots that he has on them are not just random spots. Gino painted them so they look a little bit like eyebrows, kind of a man look. He's got a little patch on his head and I think even, Gino, he's got a little mustache.

GA: He's got a little mustache, yeah.

CW: Yeah. So those kinds of qualities are more characteristic and then also it reinforces the animation that Aidan was doing that he kind of had this... I guess you could describe it... I don't know... this hard look and that helped reinforce that. Aidan, you could talk a little bit about his... You used a reference for animation.

AM: Yeah, so when we were actually designing the motion for AJ... and AJ's not just a fish. He's a tank on top of a cyborg body. On-set we had Courtney Stevens playing the body and giving us a voice performance. He was wearing the collar piece. So generally what we would do is we would track his body in the 3D space and then we were replacing, from the collar up, with CG. There's a couple of shots where we completely replaced the whole body because it just made a bit more sense for us to do that at the time, but it's at minimum, a couple of shots.

So we'd have to start with the match move of his tank moving around. We weren't too sure initially whether the tank would move. Everett and Steve Blackman, who are the creative side on the show side, didn't have an idea either so we all just did a bit of exploration on how a tank would actually move. Whether it was stiff, so when the body moved the tank would just move like this and the water would just slosh around. Or if he was like a chicken and the tank was always trying to stay upright and counter itself.

We ended up just having a little bit of the head just do a little... when the body moved the head would just slightly over articulate. So it had a little bit of its own life and it seemed reasonably natural. Then inside that, obviously, we have our goldfish, which we tried to keep as naturalistic as possible, especially when you have the goldfish actually talking, delivering dialogue, which, for us, was unusual. I don't think we'd ever done a hero fish like that before. With AJ and Carmichael who are our real fish, we went to Gino's workshop and we set up a three-camera reference shoot with a green screen and we just had the camera's on them just for about 20 minutes getting them from all three angles. So we had a front, side and top-down camera.

Then we took all that footage, synced it all up in editorial with a timecode on it so we knew that all the frames all matched up. Then Craig and I would sit down, look at all the shots and we'd just pick the sections of the fish motion that seemed that it was delivering the actual lines or just seemed to match the sort of attitude. On top of that, we didn't just have animators just copying exactly what we saw in the reference, but we built what we call here a layer stack, which is a big stack of separate motions. We use it for a lot of flying things and swimming things, it's very handy for. If you're familiar with how game animations are made, it's essentially a big stack of pre-canned animation that you can blend between to get to other states of motion.

So we would use our layer stack and blend out pre-canned motions that we built to match the reference as closely as possible so that AJ, in each shot, is always on model, if that makes sense? So you're not having different animators adding too much swish or being a bit stiff. He's always using the same pre-canned motions to match. They're always the same motion from the exact same fish each time. It was also important that you're referencing the same fish all the time, because if you had two different fish or two different breed of fish they might move slightly differently. So we're always referencing the exact same fish using the same pre-canned, layer stack motion so that we could keep everything as tight and as on the model as possible.

GA: One of the things that was really cool too that we got to do was, again, try to find some real-world reference. Chris and I got together and we decided we're going to try to recreate the dome itself. So I went out and found these pretty cool domes that weren't the actual size of what was in the film, but a bit smaller. I made this little container and one of the cool effects was that we wanted to see how the bubbles traveled on the inside of the dome.

One of the cool things was to see what happens when smoke goes inside of a bubble. We had one of the guys come over who has one of those vapor things and we had these tubes in there. He's take a big hit, blow into the tube and these bubbles would come up and it was so cool, because the bubbles would come up filled with smoke and as soon as they broke the water line there would like this little poof and the vapor would just travel up. It was great reference and it was useful stuff obviously for the effects guys to use. Once again, love making these practical things to use as reference for the digital asset.

Water Woes
You bring up the smoke in the water. I've come to learn visual effects artists always point out how difficult water is to work with. Is that the same in your experience? I know just recently I talked to someone and they absolutely hated working with water.

CW: Yeah. I can speak from experience, water is always a bit difficult. The one thing that was nice about this project is that we've been spending the last two and half years rebuilding our whole water pipeline from the ground up. We have a ton of developers, scientists. We're just rewriting the whole thing. Part of what you're seeing with AJ is some of the first uses of our new water software. The sloshing around, the little bubbles coming up, all of that was driven from that new software. It worked pretty well.

We just had to tune the water differently for each shot. Sometimes we'd make it more viscus if he was sloshing around a bit more. Or sometimes we'd actually have to change the quality of the water, because AJ being in the water would get magnified by the glass and by the water. So sometimes we'd have to cheat that quality so you got a good view on AJ. It's a bit of a challenge when you have a tiny, little fish speaking, inside of this big body, inside of this tank. So each shot had to be adjusted accordingly.