Using Render Layers in Maya 8.5

Posted on Posted in art, mental ray, video games

Something I’ve found that not many game artists are familiar with is the Render Layers found in Maya since version 7.

They can be extremely handy if you want to render your scene in passes, but they can also be very helpful when baking geometry to textures.

Quick Overview

You’ll find the render layers in Maya if you switch from “Display” to “Render”, above your layer panel on the right hand side of your standard Maya layout. By default there are no render layers active, so we’ll have to go ahead and choose Layer > Create Empty Layer.

This will create two layers, one named layer1 and one called masterLayer. The masterLayer is your default layer, containing all the objects in your scene as well as all your scene settings.

In order to make use of layer1 we’ll first have to add some of our objects to it, do this by selecting the objects you want to add and then right clicking on layer1 and choosing Add Selected Objects. Also note that you can remove objects from the layer in the same fasion, just like with regular Display Layers.

If you right click on layer1 you’ll find that there are some set presets which we can apply to our layer, under the Presets-menu. These include Luminance Depth, Occlusion, Normals, Diffuse, Specula and Ambient render presets. I’ll show how to make use of some of these in the next section.

Layer Overrides

You can basically create a layer override on any attribute in Maya, simply by right clicking on it’s name and then choosing Create Layer Override (this requires that you currently have a render layer selected). This will tell Maya to treat that attribute as a unique setting for that layer only, so no matter how you alter that attribute in other layers it will stay the same in this one. Also remember to create the layer override before you change its value, or you will actually be changing that value for every other layer as well.

Attributes that have layer overrides, will show in orange for that specific layer, which is handy when you’re using the layer presets.

You could for example put a light on both layers 1 and 2, and then set a “layer override” that will tell the light not to cast shadows on layer 2.

By default, all layers will use the same render settings, unless you are using a render layer preset. By creating a layer override you could for example tell maya to render a specific layer using Mental Ray and Final Gathering, while using the Maya Software Renderer for your other layers.

Production example: Creating multiple textures using Render Layers

I find it’s always easier to explain things with a real world example, so that’s what I’m going to try here. I created some random poly geometry that I’ll be setting up render layers for, as well as an ortographic camera to render it with.

image: some example geometry for this tutorial.

I started out by creating a polygon plane 1*1 meters big and at an 90 degree angle towards the front-camera. Then set up the rendering resolution to 512*512 pixels.

Lock the XYZ-translation on the front-camera and set the the Orthographic Width (found under Orthographic Views-tab in the camera’s Attribute Editor) to 10 and lock that as well.

Your Orthographic Width may differ depending on your scene’s unit-settings. Just make sure that your front-camera’s render region exactly matches the size of your initial polygon plane, the best way to do this is to open the Attribute Editor, select your front-camera and choose Display Options > Display Resolution. Also set the Overscan to something like 1.3-1.5.

That way you can see exactly what portion of your viewport will be rendered and can easily match up your Orthographic Width to the size of your polygon plane.

As soon as you have your front-camera and base plane set up it’s time to do some modelling, at this stage you can also take the time to apply some basic materials to your geometry, see below. While doing this, feel free to use any shaders, textures and shading networks, even procedural textures. Since the result will be baked to a texture anyway it does not matter what you use to get your final diffuse. Colored speculars would be good to set up at this point as well, since they will show up nicely in the specular-render.

image: some coloured materials applied to the geometry.

When you’re satisfied with the general complexity of your geometry, it’s time to set up our render layers.

To do this, simply follow the instructions in the overview section of this article, creating a layer for diffuse, occlusion, normal and specular and assigning the matching preset for each layer. Don’t forget to assign all your geometry to each layer as well.

Some tweaking may be nessiccary at this point, I would suggest changing the pixel filtering on the normal-layer to Triangular Filtering with a size of 1 – 1.5, in the Render Settings for that layer. Otherwise you may loose quite a bit of sharpness in your normal-maps. I would also suggest increasing the samples on the mib_occlusion-shader that is created when you use the occlusion-preset.

To render your texture maps, simply choose the layer you want to render, then open your Render View and choose Render > Render > Front. These were the resulting textures I got from rendering my example scene from the front-camera.

image: resulting renders from the Diffuse, Occlusion, Normal and Specular render layers.

I then multiplied the Occlusion render on top of the Diffuse in Photoshop and assigned all the textures to a shader on a simple poly plane, see below.

image: Resulting textures applied to a plane and rendered in Maya’s high quality render mode.

You can download this scene below and either use it as a guide when doing your own geometry/material baking or for reference purposes.

Download example maya scene (Maya 8.5)

2 thoughts on “Using Render Layers in Maya 8.5

  1. Hey mate,

    Great tutorial!

    Can I be so bold as to make a Zbrush tutorial request ??? 😀

    It would be great to know how to convert any Zbrush model in to polyz. You know, like sculpt then draw your own poly wireframe!

    Laters fella

  2. Hey!

    I have actually been writing on something like that, I’ll upload it later.

    I also have a couple of videos I recorded at work on that subject, I’ll see if I can get permission to publish them here or on youtube.


Leave a Reply

Your email address will not be published. Required fields are marked *