Com Positing v-Ray Render Layers in Photoshop

Advertise Here

Ahmed Fathi on Jul 15th 2011 with 53 comments

Tutorial Details

Software: 3Ds Max, V-ray, PhotoshopDifficulty: Beginners/IntermediateCompletion Time: 1-2 hrsFiles: Assets.zip

Compositing V-ray Render Layers in Photoshop http://cg.tutsplus.com/tutorials/chaos-group-vray/compositing-v-ray-ren...

1 of 39 11/25/2011 8:36 PM

Twice a month we revisit some of our reader’s favorite posts from throughout the history of Cgtuts+. Today we’retaking a look back at Ahmed Fathi’s tutorial on compositing Vray render layers in Photoshop. This tutorial wasoriginally published back in July of 2010, and generated a lot of interest. So if you missed it the first time, why


2 of 39 11/25/2011 8:36 PM

not check it out now.

In this tutorial Ahmed Fathi takes a look at how to composite together V-ray render layers using blending-modes andmasks in Photoshop. Once completed, this process allows you to change or tweak any aspect of your image inseconds without having to re-render a thing! Ahmed also covers a few extra post production techniques such asChromatic Aberration and Depth Of Field, as well as how to emulate a Cross-processed look.

As this is a compositing tutorial, not a lighting/rendering tutorial, I’ll assume that you have at least a basicknowledge of V-ray, and that you are able to render out your own scenes already. We’re going jump straight aheadinto setting up the different render elements for the compositing process.

In order to make V-ray render out the different layers, we first have to enable them in the V-ray Render Elementstab within the Render Settings window. Once in the tab, we want to enable the following render elements as shown :

VrayDiffuseFilterVrayMtlIDVrayObjectIDVrayRawGlobalIlluminationVrayRawLightingVrayRawShadowVrayReflectionVrayRefractionVraySpecularand VrayZDepth.


3 of 39 11/25/2011 8:36 PM

Most of these elements don t need much work to get them right, but we are going to need to take a few steps to setup the VrayMtlID, VrayObjectID and VrayZDepth layers.

We’ll start with the VRayZDepth element. The ZDepth layer is a black and white map that is used to tell Photoshophow far each object in our scene sits away from the render camera – the further the object is from the camera, thedarker it will appear in the Zdepth layer. Typically the Min value is used to tell the compositor which objects will bein focus.

In order to correctly setup a Zdepth map, we have to adjust the Min & Max distances that V-ray should calculate,and therefore what appears as white (the Min value) and black (the Max value) within our scene. Select theVRayZDepth item in the elements list. At the bottom you’ll see the zdepth min and zdepth max values we need toadjust.


4 of 39 11/25/2011 8:36 PM

To get a sense of what values you should use for the min and max values, you should use a tape helper object tomeasure the distance between the camera and your closest and furthest objects.To simplify things, let s say sphere #2 in the image below is the closest sphere we want to be in focus, whilst Sphere#6 should be slightly out of focus. We would use two tapes to measure those distances and put them in as your Min& Max Values.


5 of 39 11/25/2011 8:36 PM

Here’ s what the VRayZdepth render for this composite looked like after setting the Min to 20 and Max to 70meters


6 of 39 11/25/2011 8:36 PM

The VrayMtlID render layer creates an image with a different solid colour for each material in your scene, and touse it, we need to adjust your material IDs. In your material editor you will find an icon with the number 0 on it (asshown below.) If you click the 0, a grid of numbers from 0 to 15 will appear – this is your material ID. Go throughyour scene and apply a different number to each material you want to have a different color in your MtlID element.As we have 16 numbers, we can have 16 different materials appearing in our MtlID render layer.


7 of 39 11/25/2011 8:36 PM

Here ’s how your VRayMtlID element would look like after setting up the ID’s for each required material. Yourresult may have different colors, but the important thing is that they are separate from one another.


8 of 39 11/25/2011 8:36 PM

VRayObjectID element is just like material IDs element mentioned above, but it outputs a different colour based onthe different objects in your scene – the Materials are irrelevant to it. To set this up, right click your desired object,select Object Properties and give each object a different number in the Object ID field in the G-Buffer section.


9 of 39 11/25/2011 8:36 PM

I only needed to use the VRayObjectID element in order to make selecting the cars and the surrounding buildingseasier in post. Here ’s how it looks after rendering.


10 of 39 11/25/2011 8:36 PM

The last element we need to add in is the ambient occlusion layer, but as you’ll remember, this wasn’t added to ourrender elements list in V-ray. We actually need to render it separately after your initial render has been completed,but don’t worry, there is an easy, fast way to do it!

Apply a VrayLightMtl to your entire scene, and then add a VrayDirt map into the color channel. You’ll then need togo in and tweak the dirt map settings until you get a good, clean result result. For this scene, I got a nice looking AOmap with a Radius of 2 in my dirt map settings.

Note: if you use a V-ray Physical Camera in your scene, (which I highly recommend,) you will need to turn off yourExposure and Vignette options in your camera settings in order to render the AO pass properly.


11 of 39 11/25/2011 8:36 PM

After a couple of trials with the dirt map radius, I ended up with the following AO element. I also increased the dirtmap subdivs from 8 to about 64 which helped to smooth out the final render, and it still rendered out relatively fast.


12 of 39 11/25/2011 8:36 PM

Just a couple more steps and we’ll be ready to fire up Photoshop and start compositing! But first, how do we get allof these elements out of V-ray? For starters, there is the obvious way of saving them to file one at a time, howeverthere are infact two techniques used for saving the different channels to disk and we’ll cover both of them in thenext few steps.

The first method is really helpful, especially for those who use a Linear Workflow, and that is saving all the renderelements out into a single .EXR file. To do that you should first select Enable built-in Frame Bufferfrom theV-Ray::Frame buffer menu, and then turn on the Render to V-Ray Raw image file option below. With that done,we can click browse.


13 of 39 11/25/2011 8:36 PM

Browse to the place where you want to save your file, and enter your file name making sure to add the .EXRextension onto the end of it. Then select All Files(*.*) as your file type. This will allow you to save your render as asingle .EXR file that contains all your different render layers.

If, however, you try to open your new .EXR file in Photoshop, you will only see the first render layer. To see theothers, you’ll need to install a commercial Photoshop plug-in called ProEXR, and that is why I prefer the nextmethod!

Note: EXR files burn in any gamma correction as it assumes you’ll be using a linear workflow. If your image appearswashed out in Photoshop, go to Image > Adjustments > Exposure, and set your Gamma to 0.454, which is theinverse of 2.2 (calculated by dividing 1.0 by 2.2).


14 of 39 11/25/2011 8:36 PM

The second method (my preferred one) is to save all your elements out manually as .TGA files. I prefer TGAs toJPGs as they are 32-Bit, and can hold much more color Information than the normal 8-Bit JPG file. Another bonusis that they support having a built-in alpha channe.

Note: if you have never used the V-Ray Frame Buffer to render out elements before, you can find all of yourelements in the top left drop down menu that says Diffuse by default.


15 of 39 11/25/2011 8:36 PM

Now that you have finished our 3Ds Max and V-Ray Part, fire up Photoshop and let s start playing! First a small tipto help you get all of the different files into layers in one Photoshop document. It is a script built into Photoshop thatwill stack your files as layers and arrange them in alphabetical order. Go to File > Scripts > Load Scripts as Stack,and then browse for your files, select them all and click Ok. Some people prefer bringing in the elements one at atime, however I do prefer this method myself.


16 of 39 11/25/2011 8:36 PM

This is actually your first step in compositing! We’ll start by turning off the visibility on all layers except for theZ-Depth. Select all of the contents of this layer (Ctrl+A), copy them (Ctrl+C) and then switch to your Channels tab.Some people prefer adding a little bit of Gaussian blur to their Z-Depth before using it, however as most of myclients hate DOF, thinking that it’s a needless loss of detail, I do not do this! In the end it is a matter of personalpreference.


17 of 39 11/25/2011 8:36 PM

Once in the Channels tab, first check to see that your original Alpha layer is present. If it isn’t, open up the Diff.tgafile seperately, switch to it’s Channels tab, and drag and drop the Alpha channel from there onto our Z-depth image– an Alpha1 channel should appear. You can then close the Diff.tga file.

Now, click the Create New Channel and paste your Zdepth render into it. You can then switch back to the Layerstab and delete your Z-Depth layer as we don t need it anymore.


18 of 39 11/25/2011 8:36 PM

To follow along with me in the next few steps, first arrange your layers in the order shown. The RGB layer at the topof the stack is the raw render, straight out of V-ray. I’ve kept it there so that we have something to compare to at theend.


19 of 39 11/25/2011 8:36 PM

First, duplicate your Diffuse layer by right clicking it and choosing the Duplicate option, and then move the copyunder your RawLight layer as shown.


20 of 39 11/25/2011 8:36 PM

Turn on visibility for your Diffuse layer and your RawGI layer. Set your Raw GI blending mode to Multiply.


21 of 39 11/25/2011 8:36 PM

Next hold down the Alt key and click between you RawGI layer and your Diffuse layer (the cursor will change intothis two circles and an arrow icon when you’re in the correct place). This little bent arrow that appears indicates thatthis layer (RawGI) is only affecting the Diffuse layer. This is called a clipping mask, and you can find out moreabout it using a simple online search. There are tons of tutorials out there!


22 of 39 11/25/2011 8:36 PM

Using the same technique that you learned in the last two steps. Enable visibility for both Diffuse Copy andRawLight. Set the blending mode for the RawLight Layer to Multiply then use it to only affect Diffuse Copy layer(Alt+click between the two layers).


23 of 39 11/25/2011 8:36 PM

For the sake of better organization, group both the Diffuse & RawGI layers into a group, and the Diffuse Copy &RawLight layers into a different group. To create a group you can click on the small folder icon at the bottom rightof your layers (I should say sorry for people who actually know all this basic stuff. I am just trying not to letanything pass by for beginners too!)


24 of 39 11/25/2011 8:36 PM

Change the blending mode the of Diffuse + Raw Light Group to Linear dodge (Add). This adds the informationcontained in this group to the information contained in the group below . The image should start looking morenatural now.


25 of 39 11/25/2011 8:36 PM

At this point things look much better, but you may be asking yourself – why is my glass black? Well, most of theglass information in the final render is contained in the reflection and refraction passes, and we haven t composedthem just yet.

So, enable visibility for your Reflection Layer and set it’s blending mode to Linear Dodge (Add) and see thedifference. One of the main benefits of having a composite like this is that, for example, on this reflection layer youcan paint/paste in any reflection you want to appear in the windows.

Note: I should explain a little bit what linear Dodge (Add) blending mode does. It adds the color information of thepixels to each other. We know that for example Pure White is 255 and Pure Black is 0. So adding pure black adds 0whilst adding pure white adds 255. Therefore, pure black (0) + pure white (255) = Pure white (255), Mid grey (125)+ Mid grey (125) = White (250) and so on. That is why when you use this mode you no longer see any of yourblack, because it s a zero-value color.


26 of 39 11/25/2011 8:36 PM

To illustrate how easy it is to edit any of your elements (color correct it, adjust exposure, etc) I will assume that Inow want to change the reddish color of my stone texture. If we hadn’t used this multi-pass compositing method, wewould have two options:

Adjust the texture itself in Photoshop and then re-render, perhaps completing a dozen test renders before youwere satisfied with the result, and could render a HQ render. This obviously takes a lot of time and patience!

1.

Using selection tools in Photoshop, we could select the texture and apply the desired corrections to it. Thefirst problem with this is that selection is really a very tedious job, and when you come to add colorcorrections, you ’ll be affecting all of the image – your shadows and highlights will look odd and overallthings won’ t look that good.

2.


27 of 39 11/25/2011 8:36 PM

Well, remember that element called MtlID that we rendered out? It is time to enable that layer. The only use for thislayer is to create MUCH faster selections with only a couple of mouse clicks. Using this layer we will easily selectour stone texture in no time at all.


28 of 39 11/25/2011 8:36 PM

With the layer selected, go to Select > Color Range. Then use the eye dropper tool to sample the magenta color onour materials layer that represents the ID of the stone texture. With that done, press OK and you have yourselection.


29 of 39 11/25/2011 8:36 PM

Now turn off the visibility of your MtlID layer. Then while keeping that selection active, add an Exposureadjustment layer. This will create the adjustment layer and automatically add a mask so that it only affects yourStone. It doesn t matter where you have this layer now, we will move it later on.


30 of 39 11/25/2011 8:36 PM

In the Exposure adjustment layer’s settings, set your Exposure to +0.80 to brighten it up a bit.


31 of 39 11/25/2011 8:36 PM

As we want this layer to affect our diffuse only, you will have to place it right above the Diffuse layer (rememberthat one inside the group?) Move it into position as shown and remember to Alt-click between the layers to make itonly affect your Diffuse channels.

Just as a side note; whether you add it to your Diffuse layer or your Diffuse Copy Layer shouldn t make adifference at all. Some corrections might require you to place them on both Diffuse layers though, and it s just amatter of trial and errors until till you fully understand it all.

Take your Refraction layer one step down so that it sits right above your Reflection layer. Enable both and set theirblending modes to Linear Dodge (Add). I’ve added another Exposure adjustment, with the Exposure value set to+0.80, to my Refraction layer as I wanted it to appear a little brighter.

Note: Editing the Refraction layer is one of the best ways to adjust the tint of clear glass in interior renders.

Enable your Specular layer and once again change its blending mode to Linear Dodge (Add).


32 of 39 11/25/2011 8:36 PM

Duplicate your Specular layer (Ctrl+J) – notice it still has Linear dodge (Add) as its blending mode – and then withthe copy selected, goto Filter > Blur > Gaussian blur.

Set your Gaussian Blur Radius to around 2.5 pixels and press OK. Set your layer Opacity to around 65%. Thisshould help create a specular bloom effect around your specular highlights.

Note: These values are not constant numbers as they depend on your resolution, and of course your taste. Just don tover-do it!

Enable your RawShadow Layer and invert it (Ctrl + I).

You might be surprised that your shadows are bluish but this is in fact normal, they always are during the daybecause V-ray’s GI has a bright blue sky. Set your RawShadow layer’s blending mode to Multiply and it’s Opacity toaround 20%.

This step is not a must but I like enhancing my shadows a bit as it gives the image a bit more contrast. Add a ColorBalance adjustment layer, making it only affect the RawShadow layer, and with the Tones value set to Shadows,give your shadows a reddish tint. Set the opacity for this adjustment layer down to around 20%, although again, thisis just a matter of personal preference.

In other scenes when I find my shadows a little too sharp for my taste, I add a small radius Gaussian blur filter tothe shadow layer, which helps smooth out those hard edged shadows a lot.

Now enable your Ambient Occlusion layer. Set its blending mode to Multiply and its Opacity to around 10%.Ambient Occlusion shouldn’ t be too obvious in your final render, as it just helps you enhance the look of the littledetails in your final image.


33 of 39 11/25/2011 8:36 PM

At this point, we are pretty much done with the compositing! To sum it all up here ’s a simple equation that I useduntil I memorised the different blending modes. If you follow it through, you’ll see it exactly matches what we’vedone in this tutorial!

The only thing missing now is a sky with a few clouds, and that ’s what we are going to add next. You can also use abluish gradient or, if you prefer, you can cut out the V-ray sky from your original render and paste it behind thisimage. To do any of these things however, we first need to access our alpha information.

Go to the Channels pane and Ctrl-click on your main alpha channel’ s thumbnail. This will make an automaticselection of your image ignoring any transparent pieces so that we can easily add in a background image.

Invert your selection (Ctrl + Shift + I) so that we only have the sky background selected.

In my original render I used a cyan-ish gradient with some stock clouds. However as I don t have the rights toredistribute those we will have to find another image. Using Google images search the words sky field or cloudsfield or something similar. Set your search options to Large Images only so you get high resolution pictures only. Ifound this one on the first page.

Save the image and open it in Photoshop. Select all of its contents (Ctrl + A), copy them (Ctrl + C), and as we stillhave that selection we made using our alpha, use Photoshop’ s command Paste Into (Ctrl + Shift + V) command.This should automatically add a mask to our new sky layer. Rename this new layer something like Sky BG.Note: In Photoshop CS5 this shortcut changed to (Ctrl + Alt + Shift + V).

Using Free transform (Ctrl + T) adjust the size of your sky until it fits the image. Remember to hold Shift whileresizing to uniformly scale.

Then add a Color Balance adjustment layer over your sky BG only (again using the clipping mask method) andplay with the settings until you’re happy with the result. I liked mine with a bit of a cyan tint to it.


34 of 39 11/25/2011 8:36 PM

We are now done with the compositing! Now I’m going to cover some post production tips like adding motion blurto cars, adding DOF using ZDepth, and adding chromatic aberration. You have come a really long way, so let’ scompare our composite to the original V-ray output (the RGB layer).

As you can see, apart from the skies, they both look extremely similar, the only difference being that in ourcomposite, anything can be changed or tweaked without having to re-render anything.

Now begins the fun part – post production! There are many different workflows for this part, it all just depends onyour own personal preferences. I will be showing you some of the techniques I use, to help you get started.

First make sure you have the look of your render exactly how you want it. If you want to increase reflections orrefractions, add more specular highlights, now is the time!

Now merge all your visible layers into one single layer by selecting the top-most visible layer and pressing Ctrl + Alt+ Shift + E. This will paste a merged copy of all your layers into a new layer called Layer 1 .

Making sure you’ve got the new Layer 1 selected, use the same select color range technique we learned earlier(steps 26 & 27) but this time on the Object ID layer, to select the cars.

We want to create a new layer containing only the cars, and to do this we press Ctrl + J. Rename the new layer(initially called Layer 2) to Cars Motion Blur then delete the merged Layer 1 .

With the Cars Motion Blur layer selected, go to Filter > Blur > Motion Blur.

Alter the direction of motion so that it matches the direction of the cars, and then choose a medium Distance (I usedaround 25 pixels). Click OK to finalise.


35 of 39 11/25/2011 8:36 PM

Lower the Opacity of this layer to around 30%. The motion blur is complete!

Now to add some chromatic aberration. Please be advised that CA is an effect that should be used as little aspossible, as too much can cause your renders to look blurry and ugly. Before attempting to use this effect on yourown renders I suggest you read more about this phenomenon, so that you know where and when CA should appear.When done right however, this effect can really enhance the realism of your picture.Use the same shortcut as before (step 45) to merge your visible layers together, and rename this new layer Merge .

Making sure the Merge layer is selected, go to Filter > Distort > Lens Correction. We will use this filter again in amoment to add a vignette, so remember where to find it!

Deselect Show Grid and then experiment with the values for Fix Red/Cyan Fringe and Fix Blue/YellowFringe (I used +5 & -5) until you get a suitable, subtle effect. Then set your Edge mode to Edge Extension toprevent your picture from having transparent edges, although if this issue is visible, you’ve likely set your values toohigh already!

Although this is the native way of adding CA in Photoshop, There are many other ways people use. Some people usethree different layers with the red, green and blue data and shift them manually, other people use plug-ins. I decidedto show you the native way as almost all the plug-ins are commercial.

Some of the greatest commercial plug-ins for photoshop post-production are Magic Bullet Photolooks , 55 mmDigital Film Tools and Knoll Light Factory . One of my favorites is Nik Software: Color Efex Pro . If you canafford these then by all means they are worth it, but if you can’t you now know how to add the effect manually!

Now that you have completed your CA effect, it’s time to add some DOF. Remember that we pasted our ZDepthrender element into a new channel? We will use that now! Start by going to Filter > Blur > Lens Blur.


36 of 39 11/25/2011 8:36 PM

In Source choose your Zdepth channel (in my case Alpha 2) and enable the Invert checkbox. Radius controls theamount of blurriness, so experiment with that although don t over-do it! I chose a radius of 4 and then pressed OK.

As a rule of thumb if a filter/effect is really noticeable, it is too much! (Unless of course it is some kind of an artisticapproach). Maybe take some time to surf around in the CG forums and see how the pros use DOF in their images.Try to learn the best ways to add it in without hurting your own visualization. Also, remember that some clients hateDOF, so be careful with it!

Another effect that looks really good on some pictures (and awful on others) is cross processing. Cross processing isbasically playing with your color curves to achieve a more dramatic look. So add a Curves adjustment layer on topof your Merge layer.

Go to the Channel drop down menu in the adjustment layer, and adjust your curves individually by selecting themone at a time. Here’ s the Red channel curve that I used.

Here ’s the Green channel curve.

Finally, here ’s the Blue channel curve. Remember that this is just an example; hundreds of looks can be achievedwith this method, and your curves will very likely vary from one shot to another depending on the look you’ reafter.

I felt the final effect of this adjustment was too much for my taste, so I decreased the Opacity of the adjustmentlayer to around 65%.

The final step is to add a vignette to your render. Some people prefer adding a black layer with an oval shaped soft


37 of 39 11/25/2011 8:36 PM

mask, however I actually prefer adding it using Magic Bullet Photolooks. For this tutorial, we’ll stick with thebuilt-in tools, so let s add it using the Lens Correction Filter.

First we have to merge our Merge layer with the Curves adjustment layer. To do that use the shortcut Ctrl + Alt +Shift + E as before, and then, with the new layer selected, go to Filter > Distort > Lens Correction.

I used an Amount of -20 for this render, however it’s important that you don’ t over-do the vignette, as it justwon’ t look that good!

With that, your image is complete! I really hope you enjoyed this tutorial and learned at least a new trick or two. Ifyou have any comments, questions or criticisms please go ahead and post in the comments below, and I’ ll be morethan happy to answer you.

Don’t miss more CG tutorials and guides, published daily – subscribe to Cgtuts+ by RSS.

Tags: Ahmed FathiCompositingPhotoshopVray

By Ahmed FathiHi I'm Ahmed Fathi, an architect and freelance CG artist from Cairo, Egypt. I have been into CG for nearly 4 yearsnow, and I specialize in architectural visualizations and simple walkthrough animations. I learned most of what Iknow from online tutorials on sites like this one, so I would like to thank everyone who has helped me with anythingalong the way!


38 of 39 11/25/2011 8:36 PM


39 of 39 11/25/2011 8:36 PM

Com Positing v-Ray Render Layers in Photoshop

Documents

Transcript of Com Positing v-Ray Render Layers in Photoshop