Well i’ve just about figured out how to do this, and having found no real help on the forums i thought i’d make a thread and maybe help people do this themselves, this certainly isnt authoritative and i’m sure i’m probably wrong about a lot of things but i’ll gladly take suggestions!
Beast is very good for producing a lightmap within most situations you’d want for games, particularly when you want parity between your baked lights and realtime lights, but for some situations, particularly mobile dev where you won’t be relying much on realtime complex lighting, being able to produce lightmaps that produce a visually pleasing environment with the control you’d like in your lighting, it’s nice to be able to use outside tools
This post is very much focussed on producing lightmaps for objects with tiled textures also
I’ll start from the beginning, so to speak
You’ll need to prepare your model, it’s best to keep the main textures and lightmapping completely apart, a lightmap cannot reproduce the sharpness of a tiled texture unless your lightmap has a huuuge resolution (And i suppose we’re talking basically baking in the entire texture into your diffuse channel here)
(Edit note: make sure your objects use vray materials or you may get strange results)
Assuming your model is broken down into stages where tiling a texture will look good and make sense (Keeping one material per model and sharing materials between models is good, so you can take advantage of Unity’s batching), in max, keep your tiling to 1 on all UVW directions in your material, apply a uvw map modifier to your model, set on channel 1, for however you would like to add mapping to your model, do this for all models, keeping in mind not only several models sharing a material but also, and I suppose this has to be found via experimentation, seeing how large, in screen space, a model will be in-game, and how the resolution of your lightmap will affect the quality of the visuals. Set the tiling via the UVW mapping modifier, I find it very useful to keep your max and unity units as close as possible, so look at your units configuration in max to make sure they are in line with Unity’s (I think 1m in max = 1 unit in Unity, and also set your grids and snaps to relate to this and how you would like to model), it might be useful to to click the fit button in the modifier panel then select ‘Manipulate’ and move the boundaries of the mapping so the top left of the model is in line with the top and left edges of the mapping gizmo and stretch the other two edges until it’s some nice figure you can do the maths easy with, a gizmo of size 100x100 needs tiling of 100x100, easy!
With the mapping of your object’s texture done, add an Unwrap UVW modifier to the object, change the map channel to 2 and click ‘Edit’ - It will probably look like a mess, right click in the window, select ‘face’ then zoom out (middle mouse) and select it all with a selection box, go to the ‘Mapping’ menu and select ‘Flatten’, the settings here, again, rely on experimentation, but the idea is to get your object flattened to 0-1 UV space and distributed evenly around it, a setting worth noting here is ‘Padding’ as it governs the space between each unwrapped portion of your model, if they are too close and the final lightmap is low resolution, part of the texture can ‘bleed’ onto other parts, making strange lighting artifacts appear
You should end up in the Unwrap UVW edit window with a handily arranged bunch of faces of the model, which represents the 3D model in 2 dimensional UV space, which is how you will apply your lightmap, a tiled texture would be very distorted on this mapping which is why map channel 1 is kept to tiling, however as lightmapping doesnt need to be sharp nor rigidly confined by tiling the mapping can be a little more flexible. Be sure that the modifier is using map channel 2, as that is how Unity knows where to apply a lightmap
With your mapping done (I’ve not gone into more here but you might want to look into optimising your model before all this, changing the topology of your model will affect the mapping and rendering lightmaps is lengthy, you don’t want to have to repeat it too many times), it’s time to render it!
Vray does it’s rendering in linear space, which means it renders with a gamma setting of 1.0, which makes sense, except most monitors work with a gamma of 2.5 or so, which causes a huge problem with the brightness of renders if you arent aware of this - Not knowing about this causes people to try and boost their lighting to overcome it inside max’s own frame buffer which isn’t automatically adjusted for it, everythings overly contrasty and light falls off too quickly and worst of all, its completely useless for baking lighting, so you’ll just need to make sure of a few things
In your max preferenced make sure your system gamma is set to 2.2, make sure this is applied to your colour selectors and material editor too, now assuming you’ve set vray as your renderer, in the vray options, go to colour mapping, and set the colour mapping’s gamma to 2.2 also, there! your lightmaps will now render with the correct gamma, however you wont see this in max’s frame buffer, it’ll just appear too bright, which can be misleading but i’ll get onto that
(Edit: with new versions of vray this has changed a little but my settings are to use linear in the colourmapping settings, make sure ‘linear workflow’ is off, if youre getting strange little dots on your render, try enabling clamp, set gamma to 2.2 (As your general max gamma correction should be) and from a nice tip i saw in another thread by Filto, if youre finding your lightmaps overbright and resulting in flat results, apparently lowering the dark and light multipliers in colormapping by 0.5 reduces the ‘exposure’ by 1ev which should hopefully achieve in much smoother graduations in the lightmap (feel free to correct me on this))
Now you’re set to render, make sure you’ve got your vray settings how you’d like them, i’m assuming you know how to use vray and what you like in terms of settings, i think i can just point to a few things that can make a big difference, one is that your lights, assuming you’re using vray lights, have their subdivisions set much higher than default, otherwise they’ll cast very grainy shadows, another is that while the usual irradiance map is good for your primary bounces in indirect illumination, light cache isnt suitable for secondary rays when rendering to a texture, so you might want to change it to brute force, and you’ll find if you use the preset low settings for irradiance map as your primary bounces you’ll get very blotchy lighting on your lightmaps, so be aware that the calculation for your irradiance map can be very lengthy to avoid this
With your vray settings done, time to go to ‘Render to texture’ on the rendering menu. Each object in your scene can have different settings here (Which I quite like compared to some 3rd party offerings) and what I tend to do is have the output folder directly into my Unity assets where i’d have the lightmaps anyway, with the object you want to bake the lighting into, make sure its selected, make sure ‘Use existing channel’ is selected and choose channel 2 (The channel you used to unwrap the model and Unity’s usual lightmap channel). Going down the panel, add an element and make it ‘VrayTotalRawLighting’, set the resolution as you’d like, set the filename to something you’d recognise and it seems I has to find this out the hard way but you want to save your baked lightmap in a 32bit format for full dynamic range, i’d go with EXR, make sure the EXR is set to RGB, you can delete the alpha channel in an image editor if you save as RGBA (If you don’t delete the alpha channel the lightmap will have blue shadows in Unity) but i’ve had very strange behaviour despite doing this if using certain texture compression formats so best try render it without the alpha element - I tend to set it to save without compression and as scanlines too, no idea what this means but it seems to work
You might want to make sure it doesnt create shell materials too, select ‘bake into existing’ or something like that but don’t select a target map slot, its unecessary and if you create shell materials it can be easy to forget and if you have to re-render it’ll render with the shell material and get very confusing
Click render! See what happens. You should have the usual max frame buffer show you it baking the lighting into the model on a 2D plane, 1 you’ll notice it’s probably too bright seeming (Thanks to the gamma setting) and if you have the tiled textures already applied in max, which you should, to create accurate lighting, you’ll see the textures in the render. Don’t worry about this, you arent looking at the lightmap, just max rendering ‘normally’, annoyingly enough you don’t see the output of the render element (in this case ‘VrayTotalRawLighting’) in the frame buffer, just in the resulting file, whcih when the rendering is done, should be an EXR file where you specified it to be saved
Now to get it unto unity! Well, go into your Unity project, assuming you’ve exported the model from max with its new mapping, presumably in fbx format, it should be ready for texturing and a lightmap when it’s added to a scene. Because you did the tiling in the uvw map in max, you shouldnt have to do it in Unity, making sharing materials and therefore batching, straightforward, and now you have the second map channel, or UV2 as Unity calls it, ready for a lightmap. Obviously in the fbx import settings in Unity for your mesh, make sure ‘Generate lightmap UVs’ is unselected
Go to the windows menu, select lightmapping, select your object, in the lighmap windows, select ‘Static’ then set the lightmap index to whatever index you’d like that object to have (Make sure it’s in the lightmap array bounds, easily set in the 3rd tab along the top of the window), make sure the scale is 1, the tiling on both x and y is 1, and the offset on x and y is 0, then go to the 3rd ‘maps’ tab and drag your exr that max should have produced from your assets pane onto the suitable slot, your object should appear to have assumed it’s brand new vray produced lightmap! But it may look strange, make sure in your import settings for the image it’s set to type ‘lightmap’ and you may want to play with the compression type to see what suits you, and you might want to change the shader for the object to something that suits your taste, ive been using the mobile> vertex lit shader
Haha god i must be bored, that was a rant and a half, but anyways if anyone finds something useful in there then i’m glad, just needed to get that out of my system, and if it was completeloy useless then at least i killed some time, cheers!
