Tuesday, June 14, 2011

Pipeline Basics

Pipeline stuff is a topic that TD's and 3D people (in particular) talk about a lot in the real world. The reason is that whatever you're working on, using whatever techniques, at whatever scale you always need to be going through a pipeline (even if it's an almost nonexistent one). Since the state of the pipeline will ALWAYS be affecting the quality of your work and your productivity, just a little bit of effort on this front will affect every job you do, usually positively. And I'm not even talking about the nuts and bolts of networking architecture or any heavily technical stuff, which most successful studios also spend a lot of effort and money on.
The main issue I've come across in studio work in NYC (and in almost all student work) is that, because any work you put into thinking about and developing a pipeline will pay benefits in a kind of diffuse way, it seems that lots of people would rather spend that time (and $) doing production work instead because it seems to pay more immediate dividends. The result is that, at some studios I've worked at, there's almost no pipeline or institutionalized workflow at all. Unsurprisingly, the result is that the same issues keep cropping up on every production.
Some things that I see regularly:
1)No naming conventions at all for either files or even folders. This can lead to mass confusion on a big job or even a smaller job, when a new artist has to step in. Where are the files? Which is the latest version? What does this abbreviation stand for? Which files should I reference?
2) No standard for passing files from one dept to the next. Without some kind of publishing system to create master files, the chances for hiccups (or worse) when using references, etc grow exponentially. What is the plan when you're animating and someone realized a modeling change needs to happen? How do you propagate a change from the front of the pipeline through to the end of the pipeline late in the game?
3) Since there's no standardization in naming or methodology, it's very difficult to develop any tools to help facilitate some tedious tasks. Even little chunks of custom code across a studio can be a huge boon to artists and actually prevent a lot of time/effort-draining errors.
4) it sounds obvious, but I spend a TON of time hunting for files, references, and project spec documents just to figure out what I'm supposed to be doing. When you multiply all that time by all the artists on all the jobs, you realize that some studios spend thousands, sometimes tens of thousands of dollars per year on time that artists aren't actually doing anything productive. Studios that deal well with these issues often have things like production calenders for all the artists to see their deadlines (and their interconnectedness to the pipeline as a whole), areas (on a computer or IRL) to look at the most recent/relevant artwork and ref material, an easy to find collection of tools, shaders, and regular team meetings (where input is heard and addressed) during production, etc.
I know it sounds like I'm slagging the studios, but I don't mean it that way. There are lots things involved here. Yes, in some cases there are studio heads are just ignorant of the business they are in (and remember I work in NYC so most of the studios I deal with are relatively small. I doubt there are many big 3D studios that don't take pipeline issues very seriously, certainly none I've worked at). Mostly I see this ignorance with design and 2D studios that are ramping up to try get 3D work. I worked with a studio a few years back that supposedly lost over $200,000 doing a their first BIG 3D job. They hit almost every point I made above in terms of issues with pipelines. They're not around anymore.
But most of the time it just boils down to not making a concerted effort in developing the infrastructure that's required to do 3D work efficiently. MacDonald's and Coke and Walmart and all other successful companies (whatever your opinion of them) are successful largely because they understand their business model really well and make huge efforts to stamp out inefficiencies. MacDonald's may not make the best burger (I'm pretty sure they don't), but they probably make the most efficient burger and that's part of why they are so successful. On the flip side (no pun intended), many of the producers I've worked with over the years know NOTHING about 3D production specifically. How can they possibly be expected to allot the time and people-power to developing a pipeline when they don't know what that even is? Many studios hire producers on a freelance basis. So where would the motivation be to develop in-house tools and other "invisible" things when you're expected to bring in a specific project on a tight budget. Many 3D leads are overworked and don't have the bandwidth in their day to spend a lot of time working  on this either. Coding for specific tools would probably require hiring a specialist, which can cost a bunch of money. Often, no persuasive argument is made for doing this. And of course, money and time are always tight, so things like pipeline infrastructure often are the last things to be dealt with. But in the end it really is like running a burger joint without knowing where you source your meat or potatoes, it doesn't really make sense to ignore your pipeline when THAT'S THE BUSINESS YOU ARE IN!
Lastly (and, in all honesty, mostly the reason I care) is that most of the "lifestyle" issues in this business that adversely affect us all (by which I mean, crazy late hours, lazy and inefficient scheduling, etc) are directly correlated with all that I've mentioned above. In my experience, studios that have really tight pipelines, have almost by definition, a better understanding of the business. They know how long x,y and z takes, they know where their man-power is going and steer it in productive ways and maybe because of that, tend to have more realistic and "human" schedules. I've been on long jobs, where within 20 minutes of sitting down I can spot a month of all-nighters coming from a mile away. It almost always has to do with the thought and effort put in long before I sit down. Obviously, stuff happens and late nights are required sometimes, but respect for the craft and the people who do it can be manifest in a real and tangible way. When a studio spends the time and energy to figure out how to best do the work that they're paid to do, my experience has been that they're also more likely to respect the people who do that work and to treat them that way (I can't speak for MacDonald's in this regard).
A long rant. Whew. So you know, this is also something that I try to talk about with my master's program students, trying to instill some rigor in the process by which they go about their work, whether on their thesis projects, on personal projects and certainly when get into the working world. Process isn't everything, but it sure helps.
Here's a video talking about some of the basics of this (organization, naming, publishing). Nothing very technical or high level, just the basics to get started.

Maya: Pipeline Basics from zeth willie on Vimeo.


  1. At roughly 40mins into your video you begin discussing cleaning up published versions by importing references and deleting history. How then would an Anim scene with a referenced published-rig (which contains an imported model) get updated if the model is changed? It seems like cleaning up the published file limits file updates to one level deep.

  2. No, no. Actually the opposite is true, it gives you more freedom to go back and forth (albeit with an extra step or two). Sorry if I didn't explain clearly. . .
    Under ordinary circumstances, once you import a rig and begin to animate, you're kinda stuck with the model (unless you've got some system built/scripted to read and replace things from under your anim data, or you want to go make changes to the rig file, in which case you might as well throw out your model files, because they're not up to date anymore).
    But if you publish stuff, you have 2 copies of the latest version of EVERYTHING. One with references, etc that can be updated from the reference files and one to use for downstream referencing (the "master" file). So in the example you mention, you would go back to the last working model file, change that model and save over the "model master". Then you would open the rig working file, which would be referencing the "model master" and hence be updated correctly with the new model. You'd save that over the "rig master" and then you could open your working anim file and viola! Without touching your anim scene, you've updated your model (and the model in every other scene in which the rig appears.) Does that make sense?


  3. I should also mention for the sake of correctness that in the master file, you are always importing references and cleaning up the scene, which avoids "nested" references ugliness. So the master files are just clean copies of the very latest working files, broken out only for the sake of referencing from the same place for artists downstream from you. It breaks the referencing chain from people upstream, while allowing changes to be made (again with the extra step of saving an extra file)


  4. Oh, I missed a step. So for each intermediary phase (in this case 'rigging') I must open the latest working version, which updates its immediate child references and then save over its published version. Once each intermediary phase is updated and published then the model's update trickles downs to the anim scene.

    This workflow is appose to everything getting updating automatically if everything was referenced traditionally. Which we are frowning on.

    Thanks for the super fast reply! Cheers!

  5. This is pretty interesting - to me everything comes down to priority == the known / the certainty of a pipeline, process or workflow.

    For workflow you tend to set priorties of tasks. e.g. i'll do task a first, then task b second etc.. These priorities tend to be based on knowledge of how long it takes. What happens at a td level is that firstly you can get unknown tasks added and with these there priority changes.

    Priorities change on a managerial level and at a personal level - your producer can tell you that this new task has to get done so now you have an unknown task add with its priority set to 1!

    Now this is different to uncertainty - you can still have an unknown task added with it priority set to 1 -but you know that you've done this task before so its certainty is easy and you can keep it at 1.

    Secondly what can happen is something that you know, and was certain before now becomes uncertain as its taking 3 days instead of 1 to do.

    So you basically have the 'knownness' of the task and the 'certainly' of it which dictates it priority.

    Priority is also driven by flow and everything has flow and priority.

    You have pipelines going to pipelines with priorities. You have processes (modelling, rigging) etc which have priorities. And you have those workflows (modelling) which has priorities and flow.

    So priority is essentially order, which is dictated by certainty and knowing.

    Flow, in my mind is like a car production. E.g. Stage A goes to stage B crucially A needs to be set to work with stage B. And likewise stage B needs to query the result of stage A to validate that its right for stage B.

    So you tend to have set, validate and query. So a process needs to validate and set it self correctly so that its result can be queried and validated to work with the next process.

  6. Interesting stuff. I've been reading a few things lately about information theory (about which I am pretty completely and stunningly ignorant). But it's interesting to think of things like this in abstracted ways. And TD/pipeline stuff is sooo broad a topic, that I feel like there are loads of approaches to take to examining a pipeline, from super technical to information parsing/distribution to pretty mundane stuff like I talk about here.
    But I'm feeling what you're putting down Charles :) thanks.
    Good blog, btw, I'll keep checking in on it . . .

  7. So, I've been looking for an answer to this question everywhere, and can't seem to find it.

    At what point do materials/shaders enter the equation?

    Do you put the materials on the master character/props/set, then reference the shaded character into the animation scene?

    What's a good workflow for implementing a shading network?

  8. @BM-

    That's actually a really good question. Give me a day or so and i'll do a little post on it. Probably a bit too long to answer properly as a reply. . .


  9. Little late to the party but I would love to know what BMDela asked as well :D

    Thanks Zeth for all your helpful info!

  10. Same comment as Animaitor and BMDela. Are you still around ? :)