UselessGameDev logo UselessGameDev title

Painterly Realtime Rendering

Fri 20 October 2023 #UselessGameDev
This text is a written version of the video above, with future plans to incorporate graphics and figures for illustration.
As I'm working my way through the backlog of videos that were published before this website, I suggest reading the article alongside the video for a comprehensive experience.

In this project we're (attempting to) build a real-time painterly rendering filter.

Let's talk about the state of the art first. There are two main sources I used for this project, one is a talk by 11-11 Memories Retold developer Alexander Birke, and the other is a paper on Painterly Stylization.

These two implementations are way more advanced than mine, as you can see in the video mine lacks time consistency and is very... bubbly. That's partly because these guys know what they're doing and I don't, and also because of the time constraint: I can't afford to let the scope of these videos get out of hand and so I can only dedicate a few weeks to coding each project, while a proper game production will have months if not years of work to perfect their result.

But enough excuses, let's see how I did, and hopefully you'll get inspired to make something even better.

Compute Shaders

This project uses Compute Shaders. If you've been following the channel, you know I've been wanting to learn to use compute shaders for a while, as they are a great way to process many small things at once. Basically you can think of compute shaders as a way to run general purpose code on the GPU.
Compute shaders are super fast if you want to process a lot of tiny things. But not just pixels. Here's a good video by Game Dev Guide on how to get started with compute shaders.

Why do we need them though? Well we're still going to be processing colors but we'll use compute shaders to quickly do a lot of processing on our image, and generate secondary images that will be helpful to determine where to draw brush strokes and how, such as a Sobel filter, a box blur, greyscale...

Prep Work

So we have a base image. Whether it's a regular texture or if it's fed from the camera, we will need to cover it with brush strokes. The first thing I did was to set up a "coverage" texture that keeps track of what pixels have been painted. So after a few strokes have been added, the coverage looks like this. To keep going we can select points where we have no coverage and keep adding brush strokes until the coverage texture is fully white.

The issue now is determining where to place the strokes. Especially at the beginning of the process, as the coverage texture is fully black, every pixel could be eligible to receive a stroke. We're going to generate a random noise for the image, and only retain the values that are above a certain threshold. If the noise changes every frame, once combined with the coverage texture we get random points to apply the strokes onto. That's enough for now, let's apply strokes.

Strokes

Our compute shader has found a list of points that could use a brush stroke. At each of these points, we can sample the color of the camera texture to know what color the stroke should be.

foreach (var item in stampsPositions)
{
    Vector2Int texturePos = item.position;
    Color color = new Color(item.color.x, item.color.y, item.color.z, 1f);
    material.SetColor("_Color", color);

    Graphics.DrawTexture(new Rect(texturePos, brushScale), stampTexture, material);
}

All strokes are then applied sequentially using Graphics.DrawTexture, which, as its name implies, stamps a texture onto the active render texture. You can provide a material so I made a simple one that only uses the alpha of the brush texture, but the color provided by the compute shader.

This is good, but we have a long way to go still. What if we want to use a non-circular brush for instance, we'll need to figure out a way to orient each stroke. This means the compute shader needs to be able to provide an angle value, on top of the position and color it already does. I had no idea how to do this, but then I remembered using a Sobel filter for the Moebius video. We used it to detect outlines, but it actually detects changes in values along the x and y axes, which gives us a vector that says "in this direction, the color changes a lot". So I figured one other application could be to take this vector and draw the stroke perpendicular to it, in hope that we will draw along the outline. It worked okay.

// Get Sobel vector
float2 sobel = SobelSinglePixel(id, cameraTexture);
float2 sobelNormalized = normalize(sobel);
// use atan2 to find the angle of the vector
float angle = atan2(sobelNormalized.y, sobelNormalized.x);
angle = degrees(angle);

There's probably a more accurate way to find an angle though.

Moving on to the scale of the stroke. My very basic understanding of how painting works is, if you have a large surface with no details, you'll use a larger brush, and for the smaller details you'll need a smaller brush. We can still use our Sobel filter for this, because the higher the norm of the sobel vectors in a region, the smaller the details, because colors are changing a lot of in these regions.
We can thus map the sobel value, which goes from 0 to 1, to a range of scales for our brush, and we should get strokes or various scales, provided we sort them beforehand, placing the larger strokes behind the smaller ones. Neat!

By the way, if you want to use Graphics.DrawTexture with rotation and scale you can provide a TRS matrix. Don't put the scale both in the matrix and the Rect, otherwise you will have a squared scale.

foreach (var item in stampsPositions)
{
    Vector2Int texturePos = item.position;

    Color color = new Color(item.color.x, item.color.y, item.color.z, 1f);
    material.SetColor("_Color", color);

    Vector3 pivot = new Vector3(texturePos.x + (brushScale.x / 2f), texturePos.y + (brushScale.x / 2f));
    Matrix4x4 newMat = Matrix4x4.TRS( pivot, Quaternion.Euler(0, 0, angle), Vector3.one)
                     * Matrix4x4.TRS(_pivot, Quaternion.identity, Vector3.one);
    GL.MultMatrix(newMat);

    Graphics.DrawTexture(new Rect(texturePos, brushScale), stampTexture, material);
}

You might have noticed strokes take a few frames to cover the entire image, that's because I preemptively limited the number of strokes per frame for performance reasons. This is especially noticeable at the start of the process because there's a lot of work to do to achieve full coverage. However this cap might not be needed because, as most processing runs on the GPU, the framerate stays above 60 fps when I increase the limit.

Now what can we do to improve our results. Well, I arrived at the same conclusion as the research paper: some larger strokes keep ending up in the detailed zones, even with rescaling the strokes based on contrast. What I resorted to do was to copy and paste the entire process to create three layers. One layer with large strokes, one with mid-sized strokes, and one with small strokes. When running these layers sequentially we get a better result. We can also vary the brush texture from layer to layer.

But my issue now is that the bottom layer still tries to draw strokes everywhere including in places where there's a lot of details an we know we'll be placing smaller strokes on the smaller levels.

The solution for that was to set a range in the sobel value to apply the bottom layer on low sobel contrast regions, the mid layer on mid regions, and the top layer on regions with high details.

When combining the three, we get an OK result. It's time to make the scene move. To do this we're going to come back to the coverage texture. When the camera moves, the existing coverage should be invalidated, however we're just going to reset the coverage for pixels that have significantly changed since the last frame. This way if the camera moves only a little or if the camera stays still but an object moves, resetting the coverage for the corresponding pixels will trigger a repaint for the affected regions.

It's already not looking great but the result when the camera moves is catastrophic. Ok.
I didn't go any further than this because this project was already dragging, but one thing both the paper and the 11-11 game do, is keeping track of each stroke and its parameters, in order to get be able to move them along with the camera, allowing for more consistency. My version just stamps the strokes and then forgets about them, and that's why when the camera moves we need to update the entire screen instead of just displacing strokes.

Conclusion

All in all, it was a very interesting project, very time-consuming project too and even after spending hours trying to get things right the result is a bit lackluster, which makes you really appreciate the work of actual artists, but I'm glad this video is out.

And there's so much more we could do if we had more time for this project! For instance making sure that some specific objects are always high details because they're important to the gameplay.

There's also bonus content on the patreon about making a lit version of the shader.

Have a good one and don't forget to grab the full source code for this Unity project on the Patreon page.

Credits

Assets

Music