The techniques used in the Blade Runner 2049 Hologram Sex scene.
By Mike Seymour
 |
| Joi played by Ana de Armas |
While there are many thematic and ethical conversations to be sparked by the relationship that a synthetic 'skin job' can have with a digital presence, it posed a complex problem for the visual effects team to solve. The audience need to understand that K was with Mariette, but seeing Joi, and the film-makers had already decided against making Joi appear as a classically depicted blue glowing hologram with interference lines.
The film's production visual effects supervisor was John Nelson and this sequence was handled by Double Negative, with Paul Lambert as DNeg's Supervisor. DNeg did over 350 shots in total, in addition to the holograms, DNeg did environment work building out the 2049 LA.
The principle photography was shot by DOP Roger Deakins without the use of motion control on set with spherical Zeiss Master Prime lenses on the Alexa XT Studio in ARRIRAW at 3.4K resolution. While it would be shown on a very wide aspect ratio (including a 2.39 : 1 version) the film was not shot anamorphically.
Having Joi
Actress Ana De Armas describes Joi as "K's lover and only friend". The character Joi is supposedly named after the urban slang porn acronym for J.O.I : Jerk Off Instruction, which is a tease game between a porn actress and the camera/viewer. Joi the character in the film is an A.I character created for a similar sexual purposes. This is reinforced clearly at the end of the film, when K sadly see's a version of his beloved Joi as a giant sex hologram (see below).
In her first scene, Joi shifts roles (and outfits) rapidly from attentive and warm confidant, to coy seductress. It is only later in that scene that it is made clear to the audience that Joi is a cognitive virtual human hologram, and the only meaningful relationship in K's harsh life. Paul Lambert saw Joi as either being re-programmed or grown beyond her programming to be so much more than just a sex companion. It was therefore very important for the audience to connect with her and care about her.
The film establishes that, while Joi appears solid, she is actually some type of light projection with no solid matter and people can pass through her without resistance. She demonstrates this being on either side of K, passing through him at the data repository while he looks at the archives. (This particular scene at the archives was done by Framestore, based on the same approach that DNeg used).
Joi who seems to 'genuinely' care for K, hires a replicant sex worker Mariette. During their encounter Joi's image is overlaid on Mariette. The scene is reminiscent of a similar scene in Spike Jonze’s Her, when the faceless AI chat bot Samantha (voiced by Scarlett Johansson) similarly hires a human sex worker to be with Theodore (played by Joaquin Phoenix). But as Samantha is never visualised as a face or presence, and remains just a voice, no visual effects were required for that sequence in Her.
To understand how the sex scene was achieved we need to look at how Joi was introduced into the film.
Making Joi
When the production was shooting the plate photography the team did not know how Joi would look. It was actually just one week before finishing principle photography that her look was agreed upon. This meant that the plates were filmed with some scope for DNeg to go in a number of different directions. Well before filming had started, DNeg did a number of tests and discussed at length with the director and VFX supervisor John Nelson possible approaches. Of all these early tests, which ranged from wireframes to particle systems, the only one that seemed to have promise for the story they were telling, was a simple 2D comp with a partial Screen transfer function. This semi transparent, subtle look, while not enough, was preferred to more elaborate and intrusive effects.
Deakins shot the apartment set with no green screen, even the window looked out on a 1/3rd scale miniature city set so as to get as much in camera as possible and avoid green or blue screen. The apartment was very grey in palette without the film's strong colour accents. The apartment set was built full of light strips, and Deakins either used these practical lights to light the actors, or used the practical lights to motivate the master lighting. The set itself was tight and did not allow a lot of room to work. DNeg set up GoPro cameras and one C-300 hidden in the room to provide animation reference if needed and they also shot extensive HDRs, but at this stage they were unsure how or what they would need to do.
 |
| "We knew we would need to do a very precise track of Joi, but we didnt at this point know what we were going to do"- DNeg's Paul Lambert |
The solution was ostensively three layers:
First, Ana de Armas was removed from the shot and a clean plate reconstructed in Nuke.
Second, a 3D version of Ana de Armas as Joi was made from extensive scanning of the actress and a face scan from USC ICT. This 3D model was then importantly cut vertically in half. This produced the back half of Joi but seen from the front. If she had text written across her back at this point one would see that text from the back in reverse. Much as one would see the back of a wine bottle's label if you sliced the bottle directly in half. And like a wine bottle this is only visible as the bottle is made of transparent glass. This back section of 3D Joi was called the Backshell volume. The CG Joi was animated by hand to match the isolated footage of Ana de Armas. This Backshell was placed over the clean background using a transfer function that is similar to a Photoshop Screen function.
Third and finally, the live action isolated Joi was partially composited over her own half shell. Of course, the front image of Joi already had volume, movement and dimension from the way Ana de Armas was filmed so that footage did not need to be 3D or projected.
If the live action isolated Joi was composited on normally, the shot would look exactly like it was originally shot. But when the front Joi is partially transparent the audience sees through to the CG Backshell. And as that the backshell is also partially transparent, one sees through to the reconstructed apartment set.
The net effect is the image has all the realism of live action footage but the volume of a transparent 3D model, and none of the artifacts or performance issues of filming the scene without actress Ana de Armas being on set in the middle of the scene.
When Director Denis Villeneuve first saw the first complete Backshell cg volume test in the last week of filming, "he looked at it, took a breath and said that's it" remembers Lambert.
This new style of Hologram, while normally very subtle, works extremely effectly. The effect can be seen most clearly in a later scene, outside the apartment where Joi is with K in the rain and a light moves behind her, making her very transparent.
 |
| Joi in the rain: The plate photography (Stage 1) |
 |
| Stage 2: CG Joi's Backshell composited over a cleaned up background with only K still in shot ( no live action Ana de Armas). Note the CG Joi is transparent - we are seeing her inside back face. |
 |
| Stage 3: Live action Joi comped over CG Joi and then over the clean background, making her seem both transparent but also a volume. |
As a side note, DNeg added something extra to this rain shot with Joi. There is a special interference pattern that is meant to represent her A.I. engine, not quite being sure what to do about rain. And then over the next few shots it learns and then starts to make the Hologram Joi look wet. Of course, without some A.I. deep learning there is no reason the Hologram of Joi should look wet in the rain, but the team thought it would be nice if the program was unsure at first and once it learnt, her cloths and hair became wet.
Importantly, by using this approach of shooting Joi by having actress Ana de Armas on set with Ryan Gosling, the photography captured her shadows cast in the scene. This set the tone visually for the sex scene that would follow. It meant that the Holograms cast shadows on objects so Joi would need to cast shadows on her sexual surrogate Mariette. While Joi casting shadows on furniture helps sell the realism of Joi to K, (and to us the audience), it makes the vfx much more complicated when she is co-inhabiting the same space as Mariette, while embracing K.
Sex scene
When DNeg had the solution to Joi's appearance, they needed to solve how to have her over-layed onto Mariette during their encounter with K. Some have called this scene a 'virtual ménage à trois' but that mischaracterizes the encounter. K is not making love to two women. He is physically with Mariette while mentally with Joi. Rather than underlining that there were two women, DNeg had to make the subtext point that K was connecting with Joi.
 |
| Mariette |
Additionally, press about the film has reported that Mariette's actress Mackenzie Davis, appears in real life to naturally resemble the on screen Joi, so when Mariette later blends with Joi in the apartment, she would matche more naturally, which helps sell the melding illusion.
Nothing could have been further from the truth for Paul Lambert.
"I was at the ICT Light Stage with Paul Debevec in LA. And I saw both of them being scanned in the Lightstage and I remember thinking, 'these two actresses look completely different', - Mariette has very big eyes, a long neck and she is taller than Joi... they are actually different - very different" he explains.
To do the sex scene with two actresses the approach was similar to Joi. The team shot both actresses doing the same moves on set one after the other, (Mariette leading by going first). This time both actresses were 3D tracked very carefully in post. As the shots were not filmed with motion control, there was a need to carefully realign Ana's Joi to sit on top of Mackenzie Davis's Mariette. This was done by first doing a neck pin, aligning and tracking their necks together. "If Ana was too far out we would reproject Ana's performance over the front geometry of Joi - to subtly shift the position of her body, with the right perspective" explained Lambert. A simple 2D move could never align the body rotation, but the live action projected onto 3D did allow small amounts of perspective changes without noticeable artifacts. "What we found out early on our testing was that we were going to be able to get away with this, without having to do a full CG version of her as they were just very subtle changes,... and you could never read it, plus she was going to be transparent", he added.
From the first pass with the neck pin they would render out the whole sequence and decide where it was naturally working quite well. From this they would pick which imperfections they would keep and which they would digitally remove. The aim was not to perfectly replace Mariette but blend between them. Some 'errors' of misalignment were needed for the audience to see what was happening, while others would need to be removed (or hidden) as they got in the way of the actresses performances.
The team then did the same CG Backshell but worked hard on where the transparency would play. In so doing they at times almost blended Joi and Mariette into a third person or presence in the scene.
 |
| Original plate |
 |
| Merge |
 |
| Mariette |
Handy
The hands allowed for the greatest amount of imperfection and misalignment. But as with the apartment set, Joi needed to cast shadows on not only K but also Mariette. Thus DNeg had to produce a full CG version of Joi to cast shadows onto CG Mariette and similarly even Mariette's hands had to cast shadows on Joi's hands.
 |
| Plate |
 |
| Merge |
 |
| Final |
Reprise
Ana was shot on greenscreen for the final reprise of a version of Joi, as a giant hologram. This was one of the only actual greenscreen shoots, but it was also known that only Joi would appear floating as a hologram she would need to be isolated. This element was also shot to be then used as a practical tool on the main unit set. As mentioned above, when this element shoot happened none of the filmmakers knew how Joi would appear so a green screen seemed like a sensible approach.
On set for the bridge scene Roger Deakins used almost exclusively the light from one three story high LED screen which would temporarily display the plate photography of the naked actress Ana de Armas. "What Roger did on set was truly phenomenal" comments Lambert. With the exception of a couple of small distant spot lights, to hint at some distant flying spinners, the giant smoke filled stage set was only lit by this huge LED screen. Much to Lambert's surprise when he turned up on set, Deakins had had the greenscreen regraded to blue and the whole set was bathed in blue spill. "I thought oh my god - what's happened?... it was not a traditional VFX kind of setup, that was just Roger's palette, he just decided to use this very chrome blue colour,.. and there wasn't any other lights", explains Lambert.
Always the brilliant DOP, when Deakins wanted K to be theoretically closer to the giant JOI hologram (which would be added in post), Deakins had the source video scaled to the LED panels - thereby changing the content to be primarily her skin (blownup) and thus K's fill light moves from blue to pink tones on the plate photography.
"We never regraded the colours of Roger's work, ...we tried to be as faithful as we could to the photography...I really pushed to keep the brilliance of his original photography...there isn't a lot of grading from a vfx point of view, we did make K a bit brighter in some shots, and we added a bit of extra fog, - but colour wise, we didn't touch it - I wouldn't dare to try and touch Rogers lighting, I couldn't improve it!", exclaims Lambert.
Roger Deakins came back to the production for the final 6 weeks of visual effects "and I had a good amount of time with him and John Nelson (at Double Negative), to make sure some of his ideas were fully supported", for example Lambert had added a distance LA city in this sequence, but Deakins pointed out that in LA in 2049 one is not meant to see very far into the distance "it is foggy - it's raining, it's snowing - and it could be all of those things at the same time - you are not meant to see into the distance, so he pulled me back on how far you could see".
For this version of Joi, the team did project the second unit footage onto a 3D version of the actress. Unlike the apartment set, there was no background, or major secondary lights, so they added a subtle back shell, which is only visible in some moments. Also thanks to the roto-matched 3D, the team also added an overall texture to her front shell which is not unlike some form of scan lines. Clearly mirroring the way that any image when projected vastly larger reveals some of its projection structure. "She was still Joi but we moved her more into an advertising space" comments Lambert.
 |
| Hologram lines are visible only in this giant version of Joi. |
For the rest of the final week of principal photography, after Paul Lambert had shown the director the final worked out solution for Joi's hologram, "Denis (Villeneuve) would come up to myself and John say 'Thank you Paul, thank you John' - he was over the moon about it... we had finally found something that was super subtle but quite sophisticated and also hard to pull off, - but it still retained what he wanted to achieve with the visual effects and the Hologram".
No comments