Pirates of the Caribbean: On Stranger Tides
A Splash of Mermaids for Fourth Pirates
by Bill Desowitz
VFXWorld | Animation World Network
May 24, 2011
ILM offers new techniques for mermaids and water in Pirates of the Caribbean: On Stranger Tides.
For
Pirates
of the Caribbean: On Stranger Tides,
Industrial
Light & Magic got
to play in the water with mermaids, the Fountain of Youth and shrunken
ships in
bottles for 300 vfx shots. 
It was certainly a departure from the
crustacean-like creatures and apocalyptic mayhem of the
previous Pirates trilogy.
And for Ben Snow, ILM’s visual effects supervisor, it was a nice change of pace from the hard surface challenges of the Iron Man franchise. The mermaids were especially different, appearing beautiful and human outside the water to entice and entrap the pirate victims and then menacing underwater with deadly fangs. But rather than going completely CG, they decided to apply a hybrid approach, in keeping with director Rob Marshall’s glam aesthetic and desire to retain as much of the live-action performance as possible, particularly when it came to the hero mermaid, Syrena, played by Astrid Berges-Frisbey.
“The look of the mermaids was important,” Snow says. “We conceived them as having an inner body that had all the scale texture on them and then an outer membrane that made them look human when they got out of the water. They evolved from being a little more human to a little more creature-like with vestigial gills, but we pulled back on that.
“It’s a different performance capture challenge,” he adds. “In this case, we had to take the scales and match them to the actual bodies and so it was a much more sheer transformation. We came up with new techniques and new tracking costume designs to help automate that: a nice smooth blend of an abdomen to a tail. Or put scales up on arms or faces. Briefly, you see transformations, so we had to match facial performances as well. So we came up with some facial tools.”
According to Tim Harrington, ILM’s animation supervisor, the facial capture was driven by two Mova Contour sessions: one to create the facial animation rig and another performance session of the actress watching previs or the shot (ADR style) on a monitor.
“Astrid did about 80 different facial expressions and we
have a new proprietary system at ILM where we can take a group of
expressions
and decompose them into all of the individual shapes that create our
facial
rig,” Harrington explains. 
“We started by creating a 1:1 match of
Astrid and it
was one of the most accurate digital doubles ever done at ILM. On top
of that,
we wanted to either be able to do facial MoCap or to animate by hand
the way we
did Davy. There were some transformation shots where we were
just going to
copy the performance from the plate, so we needed to have both
approaches and a
system that could handle both [as a hybrid]. We had a new Imocap set up
for her
because she was going to be in water and basically nude. We applied
markers
using a tattoo stencil on her arm instead of the traditional bands of
Velcro to
capture her upper body. It would then go to animation and we would
attach our
mermaid to that and animate the tail [designed by Aaron McBride] using
her legs
as a basis. We came up with a big fin that was long and elegant that
could be
simulated.” New
tools for water and interaction with the mermaids were also created.
“The idea
of taking 30 creatures with all this streaming stuff (flowing hair and
tendrils), and then running them through a full-on cloth sim was never
going to
happen,” Snow suggests. “We worked out a way to bake all of these
simulations
into the cycle, and even some simplified simulation tools that the
technical
directors could run just so we didn’t have to use our creature team to
do first
passes on all these things. And, likewise, we developed a bunch of
little
library splashes and then a means of plugging those in so that as the
mermaids
are attacking, they could automatically generate different types of
splashes.
It was really layers of tools on top of the
simulation engines to make
them run
faster and easier to do.
ILM not only used its PhysBAM fluid sim engine, but it also applied the Plume GPU-accelerated technique developed with NVIDIA for smoke and fire on The Last Airbender. This was used for mist and for overall fast turnaround. ILM also used Houdini and Maya for additional sim help. “It was handy to have automated tools to help the artists and Rob Marshall to read the animation,” Snow adds. “If the mermaid was moving in space, looking almost flat shaded, you could more easily buy the hardware render. But if we didn’t put the splashes in quickly, it didn’t look fast or powerful enough. The water interaction was key to selling the performance, and we learned a lot about getting characters to move through water convincingly. You had to have accurate-looking simulations right out of the box to show that the animation was performing well.”
ILM collaborated with Scanline VFX for the triggering of the Fountain of Youth, beginning with a droplet of water running along a leaf, and then when water flows along the walls of the cave to make a portal. However, for a spectacular liquefying death resulting in a nasty skeleton, ILM scrambled to get some extra water sim by adding a whirling effect. There were some rendered surfaces, meta-surfaces for more of a glossy look, different layers of particle sim created with PhysBAM and the simulation tools and some practical water.
Finally,
for the
effect of shrinking ships in bottles (including the Black
Pearl),
this involved setting up simulations in San Francisco by Chris Foreman
and then working with Mohen Leo and the Singapore team. 
They executed
most of
the shots.
“Initially, we just thought that the ships would be affected by the light in the room that they’re in,” Snow explains. “But then this idea came to have the ships look like they’re frozen in time at the moment of capture: in battle or in a snowstorm in the Arctic or in a storm at sea. The art department made several bottles with real model ships inside. For the wider shots, we replaced our key ones of those and essentially added movement to some of the others. And then as we got closer, they became fully CG. We even played with macro photography on some shots.
“There were a couple of happy accidents where the compositors went over the top with a big explosion. I initially had them hold it back. But I showed Rob Marshall the early take and he requested that we go back to the bigger explosion.”
Bill Desowitz is senior editor of AWN & VFXWorld.