The Lord of the Rings - Film and Video
by Scott Lehane
"In a hole in the ground there lived a Hobbit." These words, first doodled by Oxford professor J.R.R. Tolkien on a student's exam paper in the 1930s would lead to one of the greatest epics of 20th century literature. Tolkien devoted much of his life to creating a rich history for Middle-earth -- recounted in The Hobbit, The Lord of the Rings, The Silmarillion and The Unfinished Tales. His attention to detail would go so far as to create his own languages and then actually write in those languages. Indeed, it is Tolkien's attention to detail that makes The Lord of the Rings one of the most challenging projects for any filmmaker.
"When Peter Jackson first said to me, `I'm thinking about making Lord of the Rings,' one of the things I remember thinking is, Lord of the Rings -- how could you even consider it? I mean it's just extraordinary to even attempt it," said Jon Labrie, chief technology officer at Wellington, New Zealand-based Weta Ltd., which has been working on the trilogy for over four years, first backed by Miramax, and then New Line Cinema. The budget is rumored to be close to $200 million.
Indeed, it takes some audacity for any director to tackle a live-action rendition of a 1,200-page epic trilogy set in a vivid and fantastic world populated by elves, dwarfs, hobbits, trolls, orcs, wizards, ents, wraiths, balrogs, and above all, Gollum. To do it justice would require dozens of main characters, (90 speaking parts in the first film alone), and lead characters that are entirely CG creations. In addition, the story calls for a vast variety of landscapes, and battle scenes with hundreds of thousands of creatures on the field.
To make it all happen, Weta, a somewhat obscure, production, postproduction and visual effects house, which has previously created such films as Heavenly Creatures and The Frighteners, would have to build a state-of-the-art facility from the ground up to rival anything in Hollywood and import hundreds of digital artists from all over the world. And as if the project wasn't difficult enough, the company decided to do all three films in the trilogy -- The Fellowship of the Ring, The Two Towers and The Return of the King -- simultaneously. Altogether, the three films called for over 1,000 effects shots to be produced concurrently.
The first film, The Fellowship of the Ring, was scheduled to be delivered to New Line Cinema, Oct. 1 for a Dec. 19 release date. The Two Towers is scheduled for a December 2002 release, and the finale, The Return of the King will come out in December 2003.
By the end of the third film, a renderfarm of 192 SGI 1200 and 1100 series dual processor Linux servers will have chugged away 24 hours per day, seven days a week processing frames for several years.
Indeed, according to Labrie, "SGI was an early supporter of the project. They were one of the few who, three or four years ago, actually understood what Lord of the Rings might be, and what it might represent in the market and for us in New Zealand. We found them to be invaluable. It's reflected in the amount of machines that are installed in the building."
To accomplish the visual effects, Weta has put together an impressive array of computer firepower that includes 150 SGI Octane workstations, as well as 80 SGI dual processor 330 and 230 series Linux workstations. Two SGI Origin 2000s serve as the primary file servers for the facility, and Weta has also installed a TP9400 RAID storage array.
"There's 12 terabytes of [online] storage and essentially more than 20 TB of distributed storage if you add the disks on the workstations. In addition, we have a StorageTek Tape Robot. We've got about 40 TB of information already offline on tape with a hierarchical storage management system," said Labrie.
The software lineup is equally extensive, with 51 GUI licenses and 100 render licenses of Nothing Real's Shake compositing system, 60-plus seats of Alias|Wavefront's Maya for character animation, and 10 seats of Side Effects Software's Houdini. In addition, Weta's small army of artists, key frame animators, modelers, digital paint artists, motion editors, compositors and software engineers have seats of Softimage|3D version 3.9, used principally in the camera/matchmove department and the matte painting department, Pinnacle's Commotion for rotoscoping, and discreet's inferno.
Labrie stressed that, "We have the odd license for just about everything."
Shot on Super 35mm, the film is being scanned on two Imagica XE film scanners that Weta has in house. "2k is our nominal working resolution, but we do some shots in 4k when necessary," said Labrie.
The Post House, Germany, has set up a digital color grading facility less than 500 feet from Weta's main facility in Wellington, and the site is rumored to have served as one of the initial beta sites for 5D's new digital grading and finishing system, Colossus. (Representatives from 5D declined to comment).
"A substantial portion of the film is being digitally color graded. Even stuff that is not being treated by Weta directly for visual effects, is seeing some sort of digital color grading," said Labrie. "And a fair bit of the drama work is actually seeing some sort of tweaking by The Post House. Jackson was obviously heavily involved. He's trying to give it a look that you couldn't have on the shoot."
Some of the film's wide aerial shoots were captured with Wescam's stabilized camera systems.
Principal photography for the film wrapped last December. The shoot spanned 274 days spread out over 15 months, with as many as seven units shooting simultaneously at up to 350 locations spread out over New Zealand's North and South Islands. In total, the shoot called for 26,000 extra man days. 50,000 props were built including 900 suits of armor, 2,000 rubber safety weapons, 100 hand-made weapons, 20,000 individual household and everyday items handmade by artisans, and more than 1,600 pairs of prosthetic feet and ears, individually sized and shaped all built in the Weta Workshop.
There are over 60 hero miniatures (which were still undergoing principal photography when Film & Video spoke to Labrie). By the time they are done, two crews will have been shooting almost continuously for three years onto the miniature plates alone.
Dialect and language coaches were brought in to teach the actors to speak Tolkien's Elvish language and to create various accents for each of Middle-earth's nine primary cultures. And swordmaster Bob Anderson was called upon to develop fighting styles for each culture and train actors to fight like orcs, elves and dwarfs. Those fighting styles had to be translated into code for the CG characters.
"At one point, the New Zealand army actually came in to help us move a fair bit of earth so that we could achieve the Hobbiton sets," said Labrie.
He said that the shoot turned out to be one of the worst of Jackson's career. "We had horrendous weather and all kinds of problems in the production... At one point, production had to be stopped when they were shooting in Queenstown, because there was so much torrential rainfall that the entire crew had to help sandbag the town so that it wouldn't be washed away."
At times there were as many as seven different crews shooting simultaneously, and using a satellite video delivery system run by Telecom New Zealand, Jackson was able to remotely manage up to three crews at one time.
"We had a teleconferencing component so we could do talking heads, but more importantly, it was linked to the Video Assist output on the camera, so that he could, in fact, look over the shoulder of a remote DP and be assured that the framing was what he was looking for.
"We had some early teething pains just getting crews to set up a satellite system in a remote location and making sure it was all working, but we had tremendous support from Telecom New Zealand and I can't imagine that anybody would not want to do it that way in the future," he said.
Indeed work on the films followed Jackson wherever he went. "For instance, Peter is going to score in London, and he needs a way to continue to look at our work and do approvals. So we've built our own HD playback system, which is a way for us to review work at 2k on a Mac. When an artist is looking for input or approval, he submits to a system. His frames are transferred and put into the right color space, and we built a custom node to export in a squeezed DV format in 16:9, so we can dump it over an FTP connection to Peter in London," said Labrie. "It's a quick way for us to exchange material and it doesn't require any infrastructure beyond the Internet.
"You can imagine what it's like for Peter, trying to work on the score for film one while he's trying to work on the edit for film two and there are already things hovering for film three," said Labrie. "I can't imagine that anybody who's involved in the project at this point would ever want to tackle three films simultaneously again."
Larger, (and smaller) than life.
One of the biggest challenges that confronted everyone from the set designers, to the DP and the visual effects artists was that of scale. Hobbits are supposed to be about three-and-half feet high, and dwarfs are a little bigger. But from the outset, Jackson didn't want to cast people of small stature in the roles of hobbits and dwarfs.
"We knew we were going to have to figure out multiple clever ways to actually achieve [perspective]," explained Labrie. Those included the old trick of forced perspective, (positioning one actor farther away from the camera than the other), used in Hollywood for years. "The problem with forced perspective... is that traditionally you could not move the camera. If you move the camera, the parallax will give away the gag."
To enable camera movement, the production crew came up with a trick they called "moving camera forced perspective." Basically the shorter actor (farther away) is placed on a moving platform that is carefully choreographed to the camera moves in order to counteract the effect of parallax.
"What's interesting about this particular gag is that it happens on set. There is no compositing associated with it."
In some cases, sets were built as perspective rooms. Sets were often built in two distinct sizes to accommodate the scale issues and most of the props were created in two scales to serve the variety of characters on the project, from hobbit to Gandalf size. Everything from furniture to vegetables in Bilbo Baggins' garden were produced in both small and large sizes.
A Massive Undertaking
Probably the biggest challenge facing Weta was the huge battle scenes they would have to depict in order to do justice to the book. In particular, The Return of the King ends with a climactic battle pitting the forces of good against evil.
"There are literally hundreds of thousands of people fighting and screaming and dying throughout the books. And we knew that we were going to have to achieve large scale battles," said Labrie.
To tackle the problem, Stephen Regelous, crowd supervisor, began work four years ago on a program called Massive.
"Massive is a tool for the creation of artificial ecologies. The way that it generally works is that you build agents that are able to communicate with their environment, basically using sight and sound analogs, and they choose motion capture cycles based on what it is they are trying to achieve -- all based on the brains that have been built inside the tool," explained Labrie.
Developed on Irix, and now ported to Linux, Regelous controls the source code for Massive. Weta has a site license in perpetuity, but eventually, Regelous plans to take Massive to market.
Regelous explained that each creature is actually an artificial intelligence. Each one can see and hear what is around him and will respond to his environment, and other creatures in the area. Each creature is programmed with a range of behaviors which draw from a huge database of motion capture data. Atlanta-based Giant Studios' proprietary motion capture system was used for the huge database of motions that were needed to drive Massive. A motion blending engine within Massive is used to merge motions together.
Labrie reported that the largest battle scene will involve close to 100,000 creatures on the field.
"This was a much better way of getting crowd behavior. What we get is realistic behavior from individuals, and when you put them all together, the way they interact looks realistic," explained Regelous. "Using Massive, we get guys who rush into battle, actually clash and take each other out in the hundreds of thousands. And what we are seeing is pretty realistic looking battle action, which I don't think we would have achieved if we had taken a particle approach. So building the system up from the ground is what we needed to be able to do to tackle those problems.
"There are tons and tons of shots of armies going into battle," said Regelous. "And we can have as many of each type of guy as we want... So far we've done simulations of up to 14,000 guys. We can run about 5,000-8,000 on one processor."
In addition, Massive agents are able to respond to their terrain, walking differently when they are going uphill, than when they go down, for example.
"What we're doing is blending different motion capture cycles and adjusting the angles, all from the brain," said Regelous. "And this is something that we can use, not just for terrain, but for aiming weapons, and grabbing objects. We can manipulate motion capture to get much more intelligent behaviors."
Of course, if you are going to put 10,000 characters in a scene, you don't want them all to look the same. "We have a couple of tools designed to put variation into the actual agents," explained Labrie. "Orc Builder [Weta's in-house software for designing orcs] allows us to define enveloping parameters for a particular agent. You can say `arms are from half-a-meter to a meter long' or `legs are from a meter to a meter-and-a-half long.' And when legs get over a certain length you need to put the large shin guard on instead of the small shin guard. In addition, there are about 20 or 30 different types of armor plates and other things that were done before variance."
When it actually comes to battle, each species had its own martial styles. "We designed different attack, run and defense styles for the five or six army types," said Labrie.
Gollum and the Ents
With 12 hero creatures that appear across the three films, and five in the first film, Labrie reported that, "we knew that we were going to have a very good pipeline in order to build, maintain and render these creatures. So we began work early on on a skeleton, muscle and skinning system done as a plug-in for Maya. The creature department actually builds a skeleton, attaches muscle primitives to it, puts front and secondary dynamics on top of it and puts the skin on top of that."
The Fellowship Of Weta
"The question I seem to be getting a lot is, `how could something as large as Lord of the Rings actually come together in a small country 10,000 km away from Los Angeles?'" said Labrie. "In truth, it's the responsibility of Peter Jackson."
When Labrie joined Weta in 1995, there were seven animators, and they didn't even have e-mail. Labrie who describes himself as the "head geek" at Weta, said that he was originally brought on board to scale the facility to accommodate the 35 artists that were needed to finish the postproduction work on Universal Pictures' The Frighteners.
After The Frighteners, Jackson was scheduled to undertake a remake of King Kong for Universal. But Universal backed out, not wishing to release the film against Mighty Joe Young and Godzilla that year.
It was then that Jackson decided to undertake a far more ambitious project.
"Peter came to me and said, `You know, I'm not interested in working with Hollywood. I want to build the infrastructure in New Zealand to do my work, and would you be interested in sticking around to help me build the next generation facility?'"
Originally, Weta had a production deal with Miramax to make The Lord of the Rings, but Miramax was insisting on two films.
"We did about a year's worth the research and development while we were under the Miramax production deal," explained Labrie. "When Miramax blinked, Peter shopped the project around L.A. New Line Cinema enthusiastically embraced it, saying that it should probably be three films, and they increased the budget to something a bit more realistic.
"We were able to convince New Line early on that it wasn't going to be a traditional relationship, where they come to us and contract Weta to do the work," he said.
Indeed, nothing about Weta is traditional. For the past four years the company has been solely focused on completing this project. In fact, Labrie reported that they haven't yet looked beyond their current project.
"[We] never really sat down and put together any kind of marketing plan or strategy to define the growth of the company, or what kinds of projects [we] might want to take on in the future. The approach that we've taken so far has been less of a `let's build a company to do effects projects'. We've had almost a production sensibility -- `let's build a team of people, and let's size up and get what we need in place to actually achieve this job.' We've had the luxury of doing that because Lord of the Rings is such a long undertaking."
In terms of the artists, perhaps one third of them are New Zealanders, while rest have come from the U.S., Russia, Australia, France, Canada, Japan, Korea and China.
"Some of these people come to the door saying `I've wanted to work on Gollum since I read the book.'"
Indeed to say the book has a huge following of devoted fans might be a bit of an understatement. The novel, which has been read by over 50 million people, in 25 different languages, inspired an entire genre of movies and fiction. When New Line Cinema posted the web site for the film, it was swamped with 41 million hits in its first weekend, averaging close to a million hits per hour, "which was New Line's first indication that they probably had some sort of a phenomenon on their hands.
"I read the book myself six or seven times when I was a kid," said Labrie. "And I've read it a couple times since then. So, I'm a bit of a fan myself. But we've got some people on the production that are real nut jobs. But they're good to have around. They bring a real passion to the work."
Where will Weta be after The Lord of The Rings?
"Weta is 240 odd people right now. 155-160 of them are artists. I don't think it's going to be that size after The Lord of the Rings, because it's pretty difficult to actually run a visual effects facility that size successfully. Personally, as a business model, I can't see us saying that we're going to do effects in New Zealand ad infinitum," said Labrie.
"It's a nomadic business and you are generally pulling from a world market. Even if you hire an [artist] for some incredibly low rate, as soon as they develop the skills that they need to survive in the business, they are going to start looking somewhere else. People are less loyal to facilities than they are to projects. People are working at Weta because of Lord of the Rings."
But in the meantime, the production crew, artists and even the talent have developed a unique family atmosphere. "The artists that come to work at Weta feel like they are part of a larger thing, not just a large project." said Labrie. "Because we are a long way away, there is more of a family atmosphere in terms of the production; there isn't a division between the production and the facility. It's something that I don't think most of us will ever see again. We all knew what it could have been when we first signed up."
So where will Weta be after The Lord of the Rings?
"Because the project will have taken almost five years from beginning to end, that's basically the useful life on a large portion of the infrastructure that we would have purchased and installed over that time. And by the time we're done, most of what we've put in place is going to be obsolete, and the next major round of films are probably going to require technology that is the next generation and beyond. So we don't think in terms of how to use all of this technology after Lord of the Rings.
"What's next? The quick answer is we don't know... the long answer is that sometime, probably nine months before we start to wrap on film three, we'll be starting to look around." said Labrie.
"We are already doodling with the idea of possibly branching out the business into other areas -- possibly interactive development. A lot of people have come to us from the game industry, and that market is becoming increasingly interesting. It's also increasingly cutthroat. We're not going to enter it lightly. We're only going to enter it if we think there will be specific content that we can provide that is going to be compelling."
So is there a video game in the works?
"Electronic Arts is doing a Lord of the Rings game, which will be released in conjunction with film two," said Labrie. "We are working very closely with them to take content that we developed for the film and repurpose it for the game. We are already sending them models and textures and environments and other materials for them to do the polygonal reductions on, and all those sorts of things necessary for them to build their game environment.
"We wouldn't really be able to do justice to it at this stage. It wouldn't be the kind of thing you'd want us to do," he continued. "We wouldn't want to blow that particular piece of it. We are very focused on making sure that the films are right.... We are very focused on making sure that The Lord of the Rings is successful."