Tips on handling a site with potentially hundreds of images/textures

[cswkim]

I'm extremely new to the WebGL world so hopefully I can try to explain my scenario well enough to elicit some advice. The basic gist of the site is a very long carousel, where at most you see one image/texture at a time. Then there are previous and next buttons that will run a GL transition to the previous or next image. I don't want any lag/delay in transition during carousel change. What I'm noticing is that on Android browsers and Chrome for macOS, the transitions run very well. However, on iOS devices (iPhone and iPad) and Safari for macOS, there is often an unpleasant effect during transition where the screen goes black for a brief moment. I'm guessing this is due to the texture not being "ready" before the plane starts to render the transition? So if you went from imageX to imageY, you often see that black screen, however if you then went back to imageX from imageY, the transition runs smooth. Is that because the textures have been properly uploaded to the GPU?

All examples I've seen use just a handful of images and can preload the textures easily. But what if you have potentially hundreds of images? You can't preload them all at once, but maybe every time you move to a new image, you preload the previous and next ones ahead of time?

My questions:

1. Do I need to remove all Planes from the Curtains object and then create a new one for every image change?

2. If I keep the same Plane object for the entire carousel, I shouldn't use addTexture or createTexture, but rather have 2 fixed texture objects with my sampler names (uTextureFrom and uTextureTo) and use setSource()? But setSource needs the images to already be loaded into the Plane?

3. Should I keep track of every texture I preloaded using TextureLoader, so I have a giant array of all these "loose" textures? Then for every image change I check this global array to see if a texture has already been preloaded to the GPU with the matching image source and if so, use that loose texture in a createTexture call?

4. How long does a texture "live" in the GPU? Will it be a problem if you are navigating through this long carousel and loading all these textures into the GPU? How does curtains behind the scenes keep track of textures in the planes and what has been sent to the GPU? When do the textures get removed from the GPU?

5. Can you actually preload let's say 200 textures? I'm assuming this would be a bad idea. An image could be anywhere from ~100KB - ~500KB. For my device I checked <myCurtainsObject>.gl.getParameter(<myCurtainsObject>.gl.MAX_TEXTURE_IMAGE_UNITS) and it returned 16. But is that value the maximum number of textures drawn to the canvas at a time?

I suppose the closest example I encountered was the ajax navigation with plane removal.

[martinlaxenaire]

Hey @cswkim,

I'm gonna try to answer all your questions, but first you may want to understand a few things about WebGL textures and how curtainsjs handles them.

Uploading textures to the GPU using gl.texImage2D could be quite expensive, especially with large images. That is why you'd want to upload all your textures beforehand, or at the very least when you're in an idling state.
Once uploaded, textures take space in the GPU memory. If you try to upload 200 images weighing between 100~500ko, you might very well run out of memory and crash your browser tab.
Games engines for example often use what is called texture atlases - big textures containing multiple images each.
You can read a bit more about that here if you want: https://webglfundamentals.org/webgl/lessons/webgl-3d-textures.html

The library caches every image texture under the hood. This means that every time you use loadImage() or setSource() methods, it will check if there's already a texture using that image in the cache and use it if possible, avoiding a call to gl.texImage2D and the need of creating a new texture.

I know that @kekkorider has been dealing with a fair amount of textures when working on this website: https://rupert-rothschildvignerons.com/gallery
If I remember well, he had to load the textures by chunks of a dozen or so to avoid performance issues. Maybe he could tell us a bit more about that.

So, with that in mind, I'll try to answer your questions:

1. I don't see why you would remove and create new planes here. You should basically remove / create planes only on route changes.

2. Yes, I think you should preload all the textures and use setSource() in this case. Remember the textures will be cached, you shouldn't notice any delay here. This is what's done here: https://github.com/martinlaxenaire/curtainsjs/blob/master/examples/multiple-textures/js/multiple.textures.setup.js

3. You could keep track of them yes, or use your plane's textures array.

4. Textures are cached untill the curtains instance is destroyed, so they are usually cached during the whole time your website is running.

5. gl.MAX_TEXTURE_IMAGE_UNITS returns the maximum textures that could be used in a single draw call (ie while drawing 1 plane). curtainsjs only bind the textures actually used in your fragment shaders, which means that unless you're using 16 different textures or more in your fragment shader, you shouldn't worry about that. Altho, as stated above, I don't think you should consider using 200 different textures all in all.

I hope this has been helpful.

Cheers,

[cswkim]

First of all, thank you for the help and advice @martinlaxenaire, I really appreciate your time. This current site of mine is the first major WebGL project I've worked on and curtainsjs has made it really easy to get started. Thanks for your patience, I know a lot of the questions were more about WebGL fundamentals.

Yeah I figured that the logic to manage the textures, it's cache, whether or not to create a new one - was all handled behind the scenes by curtains. It's nice to know that the complex task of managing the state is not something I need to worry about.

I just have a few follow-up questions:

1. If you navigate away from the page/route that contains the curtains set-up and also has the canvas and plane DOM elements, in the React/Vue/Svelte, etc. world those DOM elements get removed/destroyed. So the curtains object would then get garbage collected and the textures would be "released" from GPU memory right? I suppose there would never be an instance where the canvas and plane DOM elements no longer exist and the textures cache is gone but the textures in GPU memory remain and become orphaned?

2. I figured trying to preload hundreds of textures at one time was a bad idea. But even if you preload just a small chunk at a time, maybe a dozen like you mentioned, sooner or later you are going to run into memory issues and potentially crash the browser tab. If you sit there and keep navigating through image after image after image, they keep getting stored in memory, the performance would keep getting degraded the more you click through the carousel until eventually there isn't enough GPU memory and a crash happens right?

3. Is there a way to remove individual textures besides removing the entire plane itself? I didn't see any method on the Texture class. And does removing a plane free up GPU memory? I know you mentioned that creating new textures is an expensive process, but I'm trying to think of a way to manage the textures cache to only what is needed for a transition to occur. In my scenario, if I'm on image 42, the GL transition is only going to happen with image 41 or 43, I don't need to know about image 74 yet. Is there some way I can only have let's say the current image, the 5 previous images and the 5 next images stored in the textures cache at any one time? For example:

Navigate to image 54, some async process preloads textures for image 49, 50, 51, 52, 53, 55, 56, 57, 58, 59. Click through to next image which is 55, then remove texture for image 49 and create a new texture for image 60.

In your honest opinion, do you think a site with hundreds of image textures will work at all?

UPDATE to Question 3
In my mind I had the plane.textures[] as equal to the global textures cache, when in fact they are 2 separate objects. When you use loader.loadImage(), that creates a new texture or returns an existing one, but they do not have parents. You have to manually plane.addTexture() or texture.addParent(). So editing the textures of a plane has no effect on the global textures cache. Managing plane.textures[] is simple enough, that property is not read-only so you can use Array methods like pop or slice or splice right?

Is there a way to interact with the global textures cache? When you use loadImage(), does that push to some internal array? Is there a way to delete individual textures from the GPU? Would CurtainsObject.gl.deleteTexture()work?

[martinlaxenaire]

Hey @cswkim,

When using a SPA such as React/Vue/Svelte or any custom AJAX navigation, you should instanciate your curtains object once on initial page load and most of the time you should keep it running as long as your web app is opened.

Whenever a texture image source is loaded, the texture is added to the cache (if not already in there). The cache isn't cleared until you destroy your curtains instance, which means it's living the whole time your app is up. That way if you ever recreate a plane that uses an already uploaded image (like when navigating back to a page that first contained the plane), the texture will be taken from the cache instead of recreated from scratch. This is a huge performance boost.

So yeah, you could potentially clog your GPU memory when creating hundreds of textures. But once again, this is not something you should be doing with WebGL, whatever the library used.

As for your third question, you could try to implement this algorithm, but I'm pretty sure that uploading images to the GPU at runtime like that will impact the performances. If you want to permanently delete a texture (ie remove it from the cache and delete the GL texture), you could use texture._dispose(true) as seen here in the source code:

curtainsjs/src/core/Texture.js

Lines 1226 to 1269 in bab814c

	/*** DESTROYING ***/
	/***
	This is used to destroy a texture and free the memory space
	Usually used on a plane/shader pass/render target removal
	params:
	@force (bool, optional): force the texture to be deleted even if cached
	***/
	_dispose(force = false) {
	if(this.sourceType === "video" || this.sourceType === "image" && !this.renderer.state.isActive) {
	// remove event listeners
	if(this._loader) {
	this._loader._removeSource(this);
	}
	// clear source
	this.source = null;
	}
	else if(this.sourceType === "canvas") {
	// clear all canvas states
	this.source.width = this.source.width;
	// clear source
	this.source = null;
	}
	// remove its parent
	this._parent = null;
	// do not delete original texture if this texture is a copy, or image texture if we're not destroying the context
	const shouldDelete = this.gl && !this._copiedFrom && (force || this.sourceType !== "image" || !this.renderer.state.isActive);
	if(shouldDelete) {
	this._canDraw = false;
	// if the texture is in our textures cache array, remove it
	this.renderer.cache.removeTexture(this);
	this.gl.activeTexture(this.gl.TEXTURE0 + this.index);
	this.gl.bindTexture(this.gl.TEXTURE_2D, null);
	this.gl.deleteTexture(this._sampler.texture);
	}
	}

Cheers,

[cswkim]

Do you know roughly the space an image of 500KB would take in GPU memory? Does the texture add any more data on top of the image itself when being uploaded to GPU? Is there compression happening? I'm trying to figure out a good number to be able to preload at once without degrading performance or worse, crashing your browser. I know this also depends on the users GPU size, but I just want to get a feel for a number, 10? 50? Can you recommend a GPU/WebGL profiler so I can monitor all of this? Would the Chrome heap snapshot be a useful tool?

[martinlaxenaire]

Hey @cswkim ,

I don't know exactly how the GPU memory is handled.
There are a few debug tools that can help you monitor that however.

There's the Chrome devtools frame rendering tool:

stats.js can also be of some help: https://github.com/mrdoob/stats.js/

Cheers,