Vertex Lighting Versus Lightmaps

On most static game assets, it is necessary for performance reasons to use precomputed lighting. There are two general kinds of precomputed lighting to choose from, vertex lighting and lightmaps. The bulk of this document will describe how to prepare a mesh to use lightmaps. But before we begin, it is worth discussing the pros and cons of the two methods.

Vertex lighting has the benefit of requiring less work on the part of the artist, as it does not require the creation of custom lighting UVs. It is also less prone to errors because it is a much simpler system. Essentially it just gives 3 lighting values (one for each axis) to each vertex. And it should be noted that where smoothing groups are different, creating hard edges on a model, Unreal spits the mesh, creating vertices for each smoothing group. The result is relatively decent, easy lighting in areas where there is a good density of vertices. However, vertex lighting often does a poor job on surfaces where large swatches of geometry exist with no vertices in them. And it is particularly bad for representing edges between shadow and light.

Lightmaps, on the other hand, are much better at representing actual shadows regardless of mesh density. Their resolution can be varied, so that one asset can have very crisp shadows, and another instance of the same geometry can have a very low resolution one to conserve resources. However, in order to use lightmaps, the asset must be prepared with custom lighting UVs. For simple meshes, this is not such an issue, but with more complex mesh, it can be very difficult to make lighting UVs that work well with a low resolution image. Poorly made lighting UVs can introduce many kinds of lighting errors. Some of these can tempt level designers to simply increase the resolution of the lightmap to overcome the problems, and this results in poor memory performance.

Testing in the Unreal 3 engine reveals that vertex lighting costs 23.4 bytes per vertex, regardless of the mesh. Lightmaps, on the other hand, cost about 2-3 bytes per pixel of lightmap. This varies a bit, because lightmaps use DXT1 compression, which has about a 10 or 8 to 1 compression ratio, whereas vertex lighting has no compression. If we estimate an 8:1 ratio between pixel cost versus vertex cost, then:

• A 64×64 lightmap is equivalent in memory to vertex lighting on a mesh with 512 vertices.

• A 128×128 lightmap is equivalent in memory to vertex lighting on a mesh with 2048 vertices

• A 256×256 lightmap is equivalent in memory to vertex lighting on a mesh with 8192 vertices

• What this means is that for meshes without significant vertex count, like walls, vertex lighting is a big performance win, in places where crisp shadows aren’t necessary. But for complex models with many vertices, lightmaps become a more attractive solution.

Ultimately, most game models should be equipped with UVs for lightmaps, so that both methods could be used. But these performance differences should be kept in mind when actually lighting levels.

General Lightmap UV Concepts

The simplest explanation of lightmap UVs is that they give unique texture space to every part of a model that needs unique lighting, which is usually most or all of the model.

Lightmap UVs should always be placed in map channel 2. There are some meshes that require another UV set aside from the primary and lightmap UVs, because of a complex shader, or because the mesh gets a new texture set in some environments that involves shuffling around parts in texture space. Even if an extra channel such as this exists, the lightmap UVs should always be in channel 2 to avoid confusion.

Note that the second UV channel is called 2 in 3DS Max, but since Unreal begins counting with 0, the second UV channel is called 1 there. After importing each mesh into the engine, make sure to open it to the static mesh viewer, and set LightMapCoordinateIndex to 1. Leave LightmapResolution at 0. This makes it default to vertex lighting. Level designers can choose to override this and give lightmaps of varying resolutions to instances in the level.

There is currently a bug in Unreal that whenever a mesh is reimported, it switches to displaying lightmaps using LightMapCoordinateIndex 0 even though it continues to really be set to 1. You can fix this by quickly switching back to 0, then 1 again. Closing and reopening the editor will also fix it. Since lightmaps can be made with varying resolutions, when UV mapping, the artist should plan for the lowest possible resolution by padding the UV parts accordingly. For most assets in Damnation, the default lowest resolution is 64×64. To work with this resolution, it helps to have a 64×64 checker helper image. Apply it to a material, set it to use map channel 2, and then in the Unwrap dialog select it from the pull down menu. It will be blurry until you go into bitmap options, and tell it to render at 64×64.

More padding is necessary for clean lightmaps than might be expected. The image below shows the lighting result in which the far sides of a cube are lit pink, whereas the sides we see are lit blue. The left shows the padding between pieces, and result when lit using a 64×64 lightmap. As we can see, leaving a gap of one pixel that is not covered at all by either pieces still results in a great deal of bleeding, which is most evident with the pink sheen on the right side. Increasing this to 2 pixels solves the problem in some places, but there is still some pink that creeps through along one edge. It’s not until we create a buffer of 3 whole pixels that we eliminate all bleeding. Of course maintaining a 3 pixel padding between every piece is not always viable for complex meshes, but for important seam edges this should be kept in mind.

Aside from padding between individual pieces, padding is also necessary at the edges of UV space, so the lightmap doesn’t bleed over. In Unreal different object lightmaps get packed together into large images, and there is no spacing between them. And every one of those larger images also can wrap around, so even things that happen to get packed by an edge are in danger of bleeding nonetheless. Below we have a simple piece of geometry in which all the sides have a large amount of padding between them, but are placed all the way to the edge. Notice how badly they bleed when lit. What this means is that the edge of UV space for an object should be padded as if it were another UV piece.

Bleeding is also a large problem when pieces are connected. Generally when creating primary UVs, it’s a good idea to connect pieces as much as possible. It eliminates texture seams and reduces the number of parts, making the task of texturing easier. However, for lightmap UVs, it’s usually a bad idea to connect pieces that go around corners, particularly when they are intended to be hard corners. The example below shows the result of doing this. Two of the three edges that face us lose all sharpness and definition, because the lighting data blends from face to face. Furthermore, the upper left rear edge in this example is filled with a line of pink, which has bled over from a side lit with that color. The only edges that retain their sharpness and proper lighting are the ones that are not connected to other pieces in UV space.

Because of this problem the “Unfold Mapping” command is particularly dangerous. Generally it is best to rely more on “Flatten Mapping” and planar projections. Projection can be particularly useful for mapping objects that contain several elements. The usefulness of this will be demonstrated in later examples.

Another rule of primary UV mapping that gets turned on its head with lightmaps is stretching. With primary UVs, it’s usually not a good idea to stretch pieces to very different proportions along one axis compared to another. It can result in texture that looks squashed or stretched on the model. However, for lightmaps this is not an issue. Therefore if an opportunity presents itself to maximize texture space usage by stretching a piece in one direction, that should be done.

Perhaps the most important rule to remember about lightmap UVs however, is that you are not finished creating them until they have been tested in the engine. And even beyond initial artist testing, more lightmap problems may become evident as levels are lit, and assets may need to have their lightmap UVs revisited.

Example • Broken Tribal Door

Here is an example of how to use planar projections well on an object with many elements. The broken tribal wall piece shown below has many bricks and stones that are separate from the main geometry. Because of the problem of bleeding around corners on unfold mapping mentioned above, the different sides of the stones have to be separated from each other. But we see here what a terrible job is done by a generic flatten mapping command. Almost all the space is taken up with innumerable tiny chunks, with only a small corner devoted to the wall itself. And this is with only 1 pixel padding, which is usually insufficient anyhow.

Instead we take the whole mesh and planar project all the major sides. This separates the different angles so they don’t bleed light or shadow around corners, and gives us a limited number of compact chunks of geometry, which should logically be sharing their shadows, that we can space with sufficient padding.

Note however, that when projecting through overlapping geometry, the lighting information will average between all surfaces. And when geometry is inside other geometry, then it gets lit darkly. This will average with the brighter lit visible surfaces, and generally darken down the lightmap. In this case, it doesn’t matter for the most part, because most of the stones are there to contrast with the wall anyhow, so if the doorframe or the broken edge are a bit darker than the rest, then all the better. However, upon testing, the top stones lit poorly from above when overlapped on the main geometry of the wall top, which is why they have been separated here into their own row. Again, testing in engine was the key to knowing which surfaces could safely be overlapped in lightmap UV space, and which ones could not.

Example • Factory Awning Roof

This example shows the projection mapping taken to an even more extreme length. This roof awning has far too many small parts to give them all unique texture space. The important thing is to have a reasonable amount of space given to the top and bottom of the roof itself, where noticeable shadows might be cast. The support pieces are secondary compared to those open areas.

Therefore, in order to conserve texture space we have planar projected most of their surfaces based simply on the direction they are facing, and which row of supports they are in, regardless of their position in space. All the red highlighted faces shown here have been grouped together into the one highlighted section of UV space because they are in the same area of the model, and all face downward. The other UV chunks shown nearby are similar groups based on direction. This sacrifices the possibility of complex shadows on the supports in order to gain enough texture space to put the entire asset on a small resolution lightmap, which is necessary for good performance.

Conclusion

Creating lightmap UVs presents entirely different challenges than primary UVs. An artist must throw out many preconceptions about mapping, and think about how best to optimize texture space for very small resolution lightmaps, such as 64×64, while avoiding bleeding between pieces, or around corners. Often approximations based simply on lighting angle must be made using projection mapping. Assets must be tested with in-engine lighting. Ultimately if all these considerations are taken into account, the result is an asset with excellent lightmap UVs.