Q&A: How Digital Effects Gave ‘The Muppets’ New Freedom

Rest assured, Muppet purists: You won’t see any computer-generated Kermits, Gonzos, or Fozzies in The Muppets, which opened in theaters last week. Every time the Muppets appear on screen in the movie, they’re the real deal: fuzz, felt, and fur creatures given their voices, movements, and expressions by human puppeteers.

But you also won’t see any visual evidence of those puppeteers in the film, despite the fact that some scenes required several puppeteers to be in full view of the camera. This is where the film’s extensive visual effects make their mark.

For the most part, the Muppets appear as autonomous, stand-alone beings in the film, free of their strings, rods, wires, and puppeteer operators. Unlike most effects-heavy movies, The Muppets uses most of its digital trickery to conceal what’s actually there instead of adding things to the scene.

To get a behind-the-scenes look at what was involved in giving the Muppets their on-screen autonomy, I spoke with Max Ivins, visual effects supervisor for Look Effects, whose team worked on hundreds of scenes in the film.

PCWorld: When working on the digital effects for The Muppets, did you feel a lot of pressure to make the effects conform to viewers’ expectations and to the legacy of the show and the previous films?

Max Ivins: I didn’t feel a lot of added pressure. At the beginning of the project, when we were talking about doing the effects for the movie, I was like, “Really? What are we going to do? Put legs on them? Are we doing CG Muppets? I don’t get it.” But that’s not what they wanted to do.

The biggest factor, in terms of what we were used for, was to give the puppeteers more freedom to do the puppeteering. There were a lot of characters shot on a blue stage with blue props, and puppeteers dressed head-to-toe in blue standing behind the puppets. We removed the puppeteers later, so it gave the puppeteers a lot more freedom in that they didn’t have to hide from the camera to do everything.

And the best thing about the movie is that it’s about the Muppets–it’s not about these spectacular effects. In a way, our job was to make it seem as if we were never present. There was sort of a conscious effort to remove the digital look from things. All of it needed to feel tangible, even if it was obviously not “real.” It’s a puppet. But they didn’t want it to seem as if there was an extraordinary leap in technology, even though we definitely used that.

PCW: Did the digital-effects team have to follow any sort of rules as to what you could and could not do to enhance some of the things the Muppets were doing? For example, adding digital effects to their eye movements or expressions or anything like that?

Ivins: We didn’t do any expressions or change any of the puppeteering. There was one shot involving a reflection in a mirror, and they shot the action from a camera that was just off to the side of the mirror. In that scene, the mirror wasn’t as distorted as they wanted–it was like a fun-house mirror. So we took the reverse angle and put [the character] into that and ended up changing the eye line to make it work as a reflection. That was the sole thing we ever did to any Muppet’s face.

Otherwise, it was pretty much just rod removal–the only times we ever retouched any of the Muppets was to take out a rod that was in front of them. We didn’t add any limbs or arms. If we did use something to repair where a rod had been or anything, we just took the actual photography of that limb or whatever, and cloned it over into the right place.

We didn’t really go back and reference any of the movies–we referenced more the television shows, following what the director gave us as a reference. That was mostly about reproducing the “arches” shot, which was the opening for the TV show.

There’s a scene with a wall about five arches tall, with a Muppet in every arch–about 46 different Muppets in it, all on blue screen. It was one of those shots where it’s like, “Oh my gosh, it’s endless.”

PCW: So you had to digitally remove 40-something puppeteers from that shot for the movie?

Ivins: Oh, no–that was shot all on blue screen and composited into the final shot. For that scene, the puppeteers did it traditionally. Most of the Muppets are waist-down in the original, so it was more of a “traditional puppeteering” thing where the puppeteers are all below the frame we were going to show. So the only thing we had to do for that was rod removal on almost every single one of them [laughs].

That’s one thing we did for the whole movie: Remove all the rods for the hands. We didn’t leave any rods in on purpose–I’m sure there are rods that are visible, but you probably can’t tell if they’re rods or not.

I think two weeks before we were done, we were still finding puppeteers’ heads in shots. “Wait a minute, what’s that thing over here? That’s somebody’s head!” [laughs] “Oh no, not another one.” Some of the shots, you’ve got like 15, 20 puppeteers crouching down, kneeling, lying on their backs, operating a big crowd of puppets … this giant crowd under the crowd. A big crowd of Muppets, and underneath that, a big crowd of puppeteers, all looking at a monitor to see how their Muppets are positioned. So they might set up and think, “That’s good, no one sees my head,” but then another puppet moves during the shot, and boom, there’s their head. It was amazing how long into the process we got before we found the last puppeteer head in the shot.

PCW: You mentioned using blue screens. Did you need to use blue screens because Kermit the Frog is about the same color as a green screen?

Ivins: Basically. Characters with blue on them were less problematic to pull keys for, and there aren’t that many principal characters that had any blue on them at all. Gonzo has some blue feathers on his head, which was a nightmare, but other than that, blue isn’t in the Muppet palette very much. There are a couple of Muppets that appear in the arches shots that are blue…

PCW: Sam the Eagle!

Ivins: Right–he was shot on blue screen, believe it or not. It really wasn’t a big problem. But Kermit on green screen is a disaster!

They shot tests on blue and green, and sent them to us, and we pulled keys on them. They didn’t want to carry around a green screen and a blue screen, or make a green stage and a blue stage, so in the end we just decided on blue as the less problematic color.

I didn’t realize before the project how many Muppets are hairy and furry. They are furry! And that is not friendly to blue screens. That ended up being the most challenging thing about it–the intricacy of getting fine detail in hair, you have to pull a key that’s very specific to it. If the character is lit brighter on one side than on the other side, or if there’s shadow down by the legs or whatever, the key has to be different there. So a lot of times, there are multiple keys on each character, basically.

And the puppeteers in blue suits, they’re not lit very well if they’re behind a character, so you have to do a lot of rotoscoping and luminance keying to fix those things.

PCW: Were the puppeteers always controlling the Muppets from underneath, or did it depend on the specific scene or Muppet?

Ivins: For most of the project they were [underneath], but there were quite a few head-to-toe shots of the Muppets. For those scenes, puppeteers were in the shots, because it sometimes took four puppeteers to make them walk and move their arms and their head. It was quite an undertaking for the puppeteers to stay behind the puppet and get the right form of motion and everything. For those head-to-toe scenes, the [characters] were shot on a blue-screen stage and composited into the shots later.

They ended up building several pretty involved rigs–in one scene in the movie, Beaker gets shrunk down to about 5 inches tall, and we see him from above, running in a circle. To puppeteer that, they basically built a merry-go-round with a platform about 4 or 5 feet off the ground so that they could have puppeteers control him to run around in a circle while somebody spun the whole thing around. They went to great lengths to compose some of these shots.

PCW: Were all the Muppet scenes filmed separately on a blue-screen stage? Or did the project also include scenes in which the Muppets and real-life actors were interacting in front of the camera?

Ivins: Most of the shots done on the blue-screen stage were solo shots, kind of hero shots, and the puppeteers would react to the plates of the previous footage. There’s a scene where the new Muppet (“Walter”) was shot in this sequence on a blue-screen stage: He’s climbing up on a cabinet, jumping onto a doorknob, swinging into the kitchen, and flying across the room. Jason Segel is in the scene, watching him fly by and talking to him, but Segel’s part was all shot beforehand. Then they just puppeteer to [that action] while looking at it. That was sort of the formula: Give the puppeteers what they’re going to have to “fit to” and “match to.”

But a lot of times it was shot traditionally. The puppeteers are masters at getting really low and using one of those dollies that mechanics use to roll under cars; they would use those and push themselves really low to the ground for the shots where the Muppets would be walking around. A guy would be pushing himself along with his feet on one of these things underneath. They’re really good at that [laughs]. I was really impressed. They’re great at getting into these odd angles. They had a lot of driving scenes where they rigged a car so that the puppeteers could get inside of it and do the puppeteering–they took out the bottom of the car, almost.

In some scenes the actors and the puppets were together on a blue screen. In one shot a Rolls Royce comes driving out of the ocean onto Cannes beach in France, which they shot at Lake Castaic near us [in Los Angeles]. They put a Rolls Royce in the water and pulled it out with a cable, onto a beach that they hauled sand in to build. They had a bunch of people in thong bikinis and Speedos. It was about 7 o’clock in the morning, so it was about 56 degrees or something. We put a big matte painting of the Cannes beach hotel behind them, and then they shot the actors and the Muppets on a blue-screen stage with a rig that looked like a Fred Flinstone vehicle–just boxes for seats, and basically a dolly to mimic the movement. So we tracked that all inside the car so it looked as if they were sitting in it coming out of the water.

In a lot of ways, that was a traditional effects job. It just so happened that there were puppets in it.

PCW: How big was the team that worked on the digital effects?

Ivins: Our company had about three 3D artists and about a dozen compositors. Disney also sent a lot of work to a facility they own for a lot of the simpler rod removal. We did about 350 shots in the movie–we were considered the lead effects facility. The effects supervisor just felt more comfortable being able to sit down with us and go over some of the more creative effects that had to be developed.

PCW: Other than removing the visual evidence of puppeteers, what were some of the more memorable sequences that your team worked on?

Ivins: I recall a sequence where Kermit walks through his mansion, and he has pictures of the band, Fozzie, Gonzo, and Chef on the wall. He starts singing a song, and as he’s going through it, the various pictures come to life. We made a “picture” version of them that they blew up and printed large for the set, and then we took that and transitioned into the puppet responding to Kermit.

That was just a real finesse job. It turns from the painting into something a little more realistic, going from the flat surface to the three-dimensional without using a straight dissolve or anything that would take you out of the movie. It was just a lot of subtle manipulation through lots of versions of it. For those things, it was much easier if we were able to come and see the director, and even see his hand gestures. Those kinds of things were the most challenging in the movie–not really a technical challenge, just the finessing, the keying, and the right balance of what the interior and exterior levels look like, and all that stuff.

There’s also a dream sequence in what the old television style looked like. They have scan lines on the puppets when they come flying out of the TV. All the little nuances don’t become apparent until you start applying all the different looks to it, and you figure out where you’re going to end up creatively.

In one helicopter shot we built a crowd up and down Hollywood Boulevard–that was one of the more fun shots we did. We were on our own particle system to drive about 20 different people we built, to make the crowd energetic and happy. At first, [the result] was like, “They don’t really move around–they look almost like a still with a bit of noise on it.” So we took a percentage of them and made them walk through the crowd and avoid each other and things. And then we added a little more motion, and then a little more motion … and then whoa! It looked like fights were breaking out everywhere. We found out that the difference between a happy crowd and a riot was about 10 percent of motion.

PCW: What kind of hardware and software setups did your team use for the effects? It’s probably not the kind of thing you can pull off with an app on a smartphone.

Ivins: That’s coming around the corner–I’m surprised that now you can do almost all of it on a laptop. Just plug in the mouse and go to town.

Software-wise, Nuke is our workhorse compositor, but we also used AfterEffects. We use Maya for our 3D work. The more computing power, the better, especially for things such as the crowd scenes. We used Nvidia Quadro graphics cards. Watching these computationally intensive simulations, you need to be able to see about a million polygons at the same time. With some of the 3D stuff, you get render times that are way up there. When you see the whole crowd, I think we got up to an hour per frame.

We have several racks of render machines, as any CG company does now–machines with 24 threads that we split into about 8 threads apiece, because it turns out to be more efficient overall. We could render [the scene] in about half the time if we used all 24 threads, but you lose efficiency as far as how much rendering you get done with the amount of power you have available.

PCW: After all was said and done, how did the puppeteers react to the CG work?

Ivins: I think the puppeteers liked the way we used technology. It wasn’t like, “Hey, we’re going to take the puppets and make CG versions, and you guys are out of a job.” It let the puppeteers cover a bigger repertoire of things that they could do with [the Muppets]. I think James Bobin, the director, wasn’t thinking, “I have puppets, so I have to shoot a certain way.” Instead, he could think, “I’m shooting a movie here, and my actors happen to be a frog, a pig, and whatever that thing is.” I think that freedom is what modern visual effects added to the puppeteering and the movie.

They seemed pretty ecstatic to be able to do it. When they put on the blue suit, they thought they were invisible [laughs]. “No, no, no, hold on. We can’t see through you.”

They were like, “The handcuffs are off, man!” It was pretty obvious that they were excited about being able to be in the shot, and we’d do things like quickly throw together what a shot would look like after it was composed. Sometimes, it’s hard to visualize exactly what [the scene is] going to end up looking like. So a couple of times, we just threw together something on a laptop on the set, so they would get an idea of what we were doing with it. Especially early on–I think the first day I was on the set, we were doing a big crowd scene where Kermit talks to everybody in the foyer of a theater. We were positioning them in groups, the way we do a traditional crowd-duplication shot: Shoot a group here, have them change their costume a bit, and then put them in the next chunk. We were doing that with the Muppets, and [the puppeteers] were fascinated by what we were going to do with their performance, and it was really great to see that.

After living with it for almost a year and being on the set, you sort of realize that puppeteering is, in a big way, the original cinema. I think shadow puppets on the cave wall were the first form of projection [laughs]. Shadow puppets in China go back thousands of years. This is the modern form of it, but it’s so close to that original art form that’s been around for thousands of years. And these are almost like the original visual-effects guys. So there was something really cool about working on [the movie] and being able to do it the way we did it. I’m proud of the way they handled it on the production side, and I feel lucky to have been a major component of the whole thing. This is one of those things I’m going to tell my kids about in five years, and I’ll be wishing that I had taken that photograph with Kermit.