Add blurred Element support to OutputDamageTracker

[mrhh69]

The problem

I'm working on getting blurred backgrounds for windows to work with smithay on my toy compositor, but have had some trouble when trying to integrate with the damage tracking in smithay.
From my POV adding this feature would be pretty important, but I could also understand if it's outside the scope of the module or too complicated to implement. It seems niri is looking to implement blur though: YaLTeR/niri#54 and this is clearly a roadblock to that.

Rambling about a potential solution

My Implementation

I'll start by explaining how I've approached the blur problem: I have a custom WaylandSurfaceRenderElement that modifies the code to render with rounded borders as well as gaussian blur. To blur I first blit the current render target into a texture and then blur the parts I care about.

All of this worked fine w/o damage tracking but with damage tracking only the pixels directly underneath the damaged elements are redrawn, so when the blur window is moved the edges will try to access parts of the texture that have not been re-rendered into the current target, so edges appear darker after being damaged.

Further, eventually this will clearly run into the problem that when there is damage beneath a blurred window its contents won't correctly update. For example: if a square beneath a blurred window changes from red to green, only the pixels directly above that square will re-render, so if the blur is in a 9px square, parts of the blurred window in the 9px border around the damaged square will be incorrect.

Ideas

• First, there will have to be a way to tell blurred elements from non-blurred elements. The way I've implemented this is to just add a diffuse function to the Element trait (diffuse meaning the size of the blur). I haven't thought about this part too much because I've been working in my local fork, but it is clearly important.
• Second: as the current damage algorithm keeps track of opaque areas, it will now have to keep track of these blurred areas and damage/redraw elements accordingly

Observations

The current damage tracking algorithm has a Vec<Rectangle> to keep track of damage (ignoring at what height the damage is). This is fine because it doesn't need to know the z-index of the damage: if there is damage under a semi-transparent element, it will redraw in that exact same damage area. (with blur this is more complicated)

It does, however need to know the z-index of any opaque regions in order to not cut out any elements on top of opaque regions.

Another oddity: my first idea was to just force each element underneath a blur element to redraw in a 17px by 17px (for 8px blur) square around the damage, but when the buffer is submitted that is actually drawn to the screen even though these redrawn areas aren't part of the damage. Why is that? How does the GbmBofferedSurface know which parts of the buffer to draw to the screen? Am I thinking about this wrong?

Any thoughts are appreciated!