Baking and procedural patterns

I am actually interested in baking pastries too, but I think I shall tackle that in the fall maybe, when it gets cooler. Let's talk about baking procedural patterns instead!

A few months back I wrote about the challenges and solutions of Material Layering. The post was quite useful for a lot of old and new friends. More questions came up, and since I joined Allegorithmic I learned much more about extra-pixarian approaches. In this post I'll focus on patterns and baking.

The composition of a Material

Taking a step back, Materials like a "painted dusty metal" are effectively plug-ins to a renderer (like RenderMan). In a physically based renderer, the Integrator is responsible to answer to the question "what color is this pixel"? To do that, the Integrator asks questions to the Materials and to the Lights.

A material can be conceptually split into Patterns and Bxdf, or shader. The shader is responsible for the illumination response component, and it has a number of parameters. Some are constants (presented as color swatches, sliders, etc...). The ones that vary across the surface can be connected to Patterns,  and are hence called "varying". Patterns define how those properties change over the surface and the volume of the objects being rendered.

Patterns are implicitly organized into logical node graphs. You don't always see that graph though. Sometimes the graph is exposed in tools such as Substance Designer, Slim (yeah! remember that one?), Maya's Hypershade, ShaderFX. Often times though, the graph is displayed in a simpler manner, as layers (Mari, Substance Painter, Photoshop), or hidden altogether behind a bunch of presets and sliders (similar to Lightroom and the upcoming Substance Alchemist).

Patterns: Procedural and Data

Patterns are not all the same in nature. I like to split them into Data and Procedurals.
Procedurals are a type of pattern that computes values on the fly, with math and/or any applicable algorithm. If you are familiar with 2d graphics, this split is rather analogous to Raster versus Vector Graphics. Procedural networks have many advantages:

• Flexible at all resolutions.
  You can "zoom" into a procedural as much as you want. It will in general always look good, within the limits of your parametrization floating point precision.
• Dynamically adjustable.
  Procedurals have parameters and changing them will simply make the procedural evaluate to a different result. This can happen during the render, at runtime.
• Little memory Footprint
  Procedurals are in general extremely cheap in terms of memory.
• Procedurals can use any input: they don't always need UV coordinates, as they can key off any geometric, be it 1D (e.g. height, facing ratio, time), 2D (UVs), 3D (point, normals, uvw, reference positions) and 4D (arbitrary, time) coordinates.

There are some cons too:

• Not always practical at render time
  While the memory usage is low, the cpu/gpu load is another story. Computing a ton of procedurals in real time at each sample of your renderer can really kill the performance.

Example of aliasing (top portion)

• Aliasing.
  While zooming in gives good results, zooming out may result in artifacts that look like there are patterns where there shouldn't be. This is due to under-sampling, and called aliasing. This can be fixed if the procedural can be correctly filtered (or blurred) analytically, that is, using math rather than over-sampling it. Pixar's wavelet noise is an elegant example of a modified procedural that behaves well at variable distances, without exhibiting aliasing.

On the other hand, flat data is a type of pattern that outputs values that are read in form some form of file, be it a texture or a volume, or a point cloud, etc...

Flat data has advantages too:

• It is fast to compute. In fact it hardly has any computation at all.
• Filters well at a distance.
  Quite the opposite of procedurals, textures are usually well equipped to provide pre-computed filtered results to be used when "zooming out" on the pattern.

And some cons:

• The quality and detail of data based patterns is limited to the resolution of the data itself. If you zoom in too much, you may end up with grainy results
• In a data based pattern, you trade off memory for speed. Loading, or streaming in, data will take some of your memory and bandwidth, and if the data is not available locally in a fast location (e.g. a local drive, or the RAM, or the GPU memory), it will affect the speed too.
• To read data, you need some parametrization. This is typically, but not limited to, UVs. Having a good parametrization is key to generating and using data successfully.
• Not all files behave the same way. If a renderer doesn't know how to handle large amounts of a certain type of file efficiently, you may lose more than you gain in speed. For example, we had to be judicious and figure out when it was actually worth baking to Ptex in our Renderman projects at Pixar, because Renderman does not have an architecture that is as optimized for Ptex files as much as Disney's Hyperion. Most of the times, UV-textures performed better (for those assets where we had good UVs)

Baking

Fortunately, any amount and combination of patterns can be converted into fast, performing data, as a pre-process. This step is what happens when you are loading a level of a game, or sometimes when a render is starting. This process is named Baking.

As cool as baking sounds, it is a lossy process, in two important ways:
Static data: once a procedural is baked to flat data (i.e. an image), it is no longer, well, procedural. That means you can no longer change its parameters and change it dynamically. Speaking of baking, this is just like you cannot change the ratio of butter after a croissant is baked.

Resolution: the baked result is fixed in resolution, and inherits all the properties of static data, including its limits in quality when zooming in. So one needs to be careful about the size of the bakes. This consideration becomes even more important when dealing with less trivial file formats, such as volumetric data, or Per Polygon Textures (Ptex).

In general, baking trades speed for memory: save on computation cycles and use pre-computed data instead, which will require space. This trade-off varies a lot depending on the incoming network. If it was cheap, it may not be worth it at all. But if the original network included a lot of data too, this process ends up saving both memory and time.

You also don't have to bake the whole pattern network. Placing strategically baking points to get the most savings (bake most procedural costly nodes) without losing all the quality and tweak-ability will get you the best of both worlds.
For example, keeping a high frequency noise to add variation where the texture resolution is not sufficient will save you from having to bake humongous textures. Adding a live HSV adjustment after the texture read will save you from having to bake multiple textures for each variation of a model, saving both time and memory.

Pattern graphs and Compositing graphs

I am going to talk a bit about Substance, not only because that's where I work, but because it's super relevant to illustrate the topic of patterns and baking. The nodes in Substance Designer are for the most part procedurals. Many of them - especially FxMaps  and Pixel Samplers - are computationally intensive and are processed by the Substance Engine which is, at its heart, a compositor, and its outputs are images. That is why it can really be seen as a baking engine on steroids. Of course, leave it to the French to show us how baking is done!

Samples vs Buffers

To fully grasp the difference one needs to visualize what a compositing node is working on, as opposed to a GL or OSL pattern.


Live pattern graphs work on individual samples: a small set of values that sometimes includes their derivatives, such as in OSL.
Sure, that operation is then run on batches, often by the thousands, but each variable can only access its corresponding input, on the same pixel/texel/sample.
Concatenating operation of this type can be reduced to a code generation problem, as the output of one pattern is used as the input of the next one.
This type of patterns does not have implicit spatial limits, so it can create infinite variations (at least theoretically, you may still run into floating point precision issues, but that's not a very common problem).

A compositing graph, on the other hand, works on buffers. Each texel at coordinates [u, v] in the output buffer may optionally depend on any or all of the texels in the input buffer, rather than just on its corresponding input at coordinates [u, v]. This allows for much more powerful effects. You can have random access nodes such as blurs and warps and sub-procedural bombing which are often prohibitive features in a live render. To be fair, bombing is a technique that can be kept live (RenderMan ships with a texture bombing OSL), but you need to bomb pre-existing textures, while Substance can bomb live procedurals.
Compositing graphs are spatially limited to their output buffers and resolution, so their variation is limited to that space. In the texturing case, that can be attenuated by making them seamless when tiling.

In Random Access patterns, the result of a given output texel can be affected by any (or all) input texels

Combining the two

So we can further decompose a material into a pre-baking (data based) and a post-baking (procedural) network:

The two types of graphs, are really not in competition, and it is important to understand how to make the best of both in advanced material design that can scale in complex productions. Scanning, processing, and rendering is nice to make tile-able materials, but that's only good for some categories of assets.

One of my favorite pastries here in France is called Demi-Cuit, or half-baked. It's sort of a molten lava cake that was not frozen before baking. It also conceptualizes nicely the essence of getting the best of baked and procedural, still fluid patterns.

Disregarding the difference

The semantics of the two types of networks are quite different, and there are operations (e.g. blur) that are only practical in one of them.

But that may not be true in 20 years on in a completely new type of renderer. There are a lot of "nodes" or conceptual operations that are pretty much identical across the two types of graphs (add, multiply, texture read, perlin noise), which makes it tempting to go ahead and describe both networks in one, and let the renderer figure out where to insert the baking points as appropriate.

This is not a solved problem today, but the MaterialX project put forwards by Industrial Light and Magic is certainly trying to do just that.

For more info about this topic, check out my older post about Portable Material Descriptions.