How should Umbraco v9 handle runtime minification of App_Plugins?

[Shazwazza]

Umbraco version

v9

Description

There have been some questions raised on Twitter about how v9 does runtime minification on JS/CSS files in App_Plugins. This is different to how v8 did it but that is for specific reasons. However, the settings implemented are naively configured based on past history and experience with how JS worked. In recent times, these settings do not work for some scenarios but the discussion should take this further by determining what should actually be done. I have listed a description and options below.

Debug mode

Debug mode is when the debug switch is configured in the config. This should never be used in production, only pre-prod/testing.

v8 Cache busting

The cache busting value was directly taken from ClientDependency.config file, it was a hard coded value and requires a developer to update this.

v8 Composite files

(Composite files mean files that are joined into 1 and processed - like minified and other processes)

in debug mode, v8 wouldn't do anything with the files in app_plugins, it would serve them as-is in separate files. It would also append the current cache buster value.

Problems would arise sometimes when a site was pushed to production with the debug flag set to false and then composite files were enabled since the files hadn't been previously tested with composite files.

No source maps are ever created in v8.

v9 Cache busting

Cache busting in v9 is controlled by an abstraction: IRuntimeMinifier.CacheBuster. This abstraction is implemented by Smidge. And this is determined by a few settings that Umbraco configures as seen here in SmidgeRuntimeMinifier.CreateJsBundle. Umbraco sets the cache buster type to either:

• ConfigCacheBuster - Used by default, means the cache buster value is determined by the config value (like in v8).
• TimestampCacheBuster - Used by default when in dev mode with the source code. This should be used by default for Umbraco extensions developers too but that will require setting this value in config. This value is currently set in the appsettings.Development.json config file. You can set it like: "RuntimeMinification"/"cacheBuster" = "Timestamp". This will mean that your files are always cache busted on every refresh because the value is based on the current timestamp. NOTE: I realize on twitter I said this would be set if in Debug mode, but that is not the case, it's only set in "Development" mode when working with the source code. Developers should also use the appsettings.{Environment}.json switch to customize their development experience by have Development specific config settings and in which case you should probably use the Timestamp setting.
• AppDomainLifetimeCacheBuster - Used if configured. Will generate a new cache buster value on every web app restart.

v9 Composite files

In v9, composite files for app_plugins are enabled all of the time. (NOTE: on twitter I said that it wouldn't minifiy in debug, but i was wrong). The settings for that are here.

In debug mode a file watcher is enabled which means that no matter what type of cache buster setting you use, changes to your app_plugins files should auto-cache bust.

Composite files are enabled in debug mode because Source Maps are created in v9 so it's still easy to debug and step through code even though composite files are enabled. This also mitigates the issues in v8 of devs not testing what happens to their files when composite files are enabled.

In v9, any .min.js or .min.css files are not included in minification since it's assumed they are already minified. In this case, they are just added to the bundle as-is.

Release mode

v8

composite files are enabled

v9

file watching is not enabled

What do we actually want?

Source maps have been raised as an issue on Twitter if folks are creating custom JS bundles with their own source maps and then having those included in the app_plugins bundle. In this case, the custom source maps won't work because the source map declaration will be embedded within the bundle and the declaration won't get used. I don't think it's (easily) possible to have a bundle of a bundle with nested source maps. In this case though, it would be worth testing that these custom bundles have the .min.js extension to tell Smidge to not JS process the file.

Another alternative for folks that don't want any runtime minifier interaction is to just load in your scripts/css lazily in the back office using the umbraco assetsService and don't register them in the package.manifest.

We could make Umbraco's bundler ignore certain files and have them loaded individually in the back office - like any .min.js files? But this may end up with all developers making .min.js files and then all app_plugins JS files are loaded individually, and in which case, we might as well remove composite file processing all together. This is another option... But this could lead to having tons of individual JS files loaded when the back office loads.

The good news is that v9 is super flexible in how it handles this so we should be able to configure the right approach.

How can you help?

If you would like to suggest a change in behavior, please ensure you have read and understood the above and then leave your suggestions below :)

[JasonElkin]

Loading many small files vs fewer larger ones should probably be configurable in itself. Over HTTP2/3 browsers perform better with multiple smaller bundles, over HTTP1.1 fewer larger bundles are better.

That's a great point about assetsService @Shazwazza

An important question to ask then is not just what we want to load, but when do we need to actually load it.

To my mind, the JS referenced in app-manifest is better if it's just a lightweight file that's then responsible for bootstrapping any other assets in the most efficient way for the user (ideally as they need them and only if they do) - this could be via the assetService or "native" dynamic import.

To that end, I think the min.js approach would work well for me

Presumably that has the added benefit that the min.js files are cached separately from the rest of the bundle? i.e. if another package is installed and the bundle(s) rebuilt, that doesn't actually affect my package's min.js file and the browser will load from its cache?

If that's the case then I'd have thought the extra hit of fetching a few more files on the first backoffice load is far outweighed by the more efficient caching from that point on. In fact, if I think about it that way, why would we want to bundle/minify different packages JS together at all? Some packages are updated frequently, some are not and not everyone installs updates as frequently as they should. With that in mind wouldn't we want to only ever load the JS that has actually changed?

[nathanwoulfe]

@JasonElkin I think that makes sense - there's definite benefits in shipping multiple smaller bundles compared to one large, catch-all bundle. It does shift the responsibility onto the developer to ensure they're providing minified/optimized bundles, but that's not an unfair expectation if that developer is also wanting to do things outside of the expected Umbraco patterns.

I've tried building my bundle as plumber.min.js, which strangely then doesn't show any trace of the file in dev tools. It still loads as part of the Umbraco backoffice bundle (as expected), but isn't referenced in the sources panel (it was previously when not built as .min.js). Bit strange.

I'm trying an approach with a separate loader referenced in the manifest, which then injects the script tag for the main bundle. Benefit of doing it this way is that I can then specify type="module" on the script, and potentially remove the browserify step from my build, ship the bundles, and let the browser manage loading. Obviously would need to test that this doesn't introduce any issues, but could be an interesting experiment at the very least.

Would be interesting too to know what the new backoffice UI approach will be - I've had a poke around the repo, the source is all using JS modules, but that doesn't mean the backoffice necessarily does the same. Does make sense though to try to align approaches so that the package dev experience can be as pleasant as possible.

[KevinJump]

I think the main affect i would be looking for from a package development point of view is that the JS in the package is presented to the user as is. be that minified or not by the package dev.

While there are lots of us with build processes and minifications (that we would probably like to keep) - there are also many property editor style packages where there is a couple of files added to the package.manifest and then sent over - i don't think we should be introducing a new set of build steps for people needing to develop a simple package like this.

The consequences of packages effecting each other in the one large concatenated file could be reduced by only combining files from any given package manifest (e.g per package) but i don't think its a massive deal to have them all in one to be honest.

[Shazwazza]

i don't think we should be introducing a new set of build steps for people needing to develop a simple package like this.

I agree. We need to cater for all scenarios somehow - or at least document how it should/could work for these scenarios.

I'm trying an approach with a separate loader referenced in the manifest, which then injects the script tag for the main bundle.

@nathanwoulfe any feedback on this? did it work?

To that end, I think the min.js approach would work well for me

@JasonElkin The hard part here is what happens if a package has a mixture of .min.js and just .js ?

I wonder if another option here is to be able to specify bunding behavior as part of the package.manifest file. We could be default continue to do what we are doing and bundling all package assets in one bundle but if you were to specify a new flag in the package.manifest like: "bundling": "none" which would just deliver all assets in the manifest as-is, or "bunding": "standalone" which would make one CSS and one JS bundle for that particular package that is separate from the main package bundle.

Thoughts? or other ideas?

[JasonElkin]

be able to specify bunding behavior as part of the package.manifest file.

@Shazwazza that would be ideal.

[Shazwazza]

For the upcoming beta what I'll do is:

• Use latest Smidge version which contains some cache buster fixes along with some source map fixes. Although the source map fixes are better, it's not perfect and that may be very difficult to debug due to the complexities of source maps themselves and how nuglify puts them together.
• Disable composite files for package files when in debug mode. This means that the files will be delivered as-is without being processed but will contain the cache buster value. This is more like how v8 worked. This just makes it simpler for all package source debugging since you'll just be working with the raw files. I realize that the way it is now is not great for various reasons: Even if source maps worked, etc... the minify engine will remove debugging statements like debugger; or possibly even logging statements which is not good for debugging. I'm sure nuglify could be tweaked for debug scenarios but IMO it's not worth the effort. Lets just let devs use their raw files to debug with cache busting enabled and then in non-debug we'll process all package files like we did in v8. This will mean that package devs will still need to test their package in non-debug mode to make sure that the minification/composite processing doesn't have any issues. (this will still be much better than v8 since nuglify is a much smarter js/css processor).

We can still implement other ideas (see above) if anyone thinks they are important.

[Shazwazza]

I've created the PR for that here #10503 in theory this will be shipped for beta4

[nathanwoulfe]

Hey @Shazwazza, these changes sound perfect - a light touch for development is ideal.

I like the idea of additional manifest options too, giving additional opt-in control means devs can chose to be a bit cleverer in how they provide their assets. I'd certainly be interested in lazy loading parts of Plumber (eg if users don't have access to the backoffice section, they don't need to load the JavaScript).

I did try loading via the assets service, but would need to do it in the application boot pipeline, to make sure it runs before any views are rendered and potentially referencing yet-to-be loaded scripts. Hadn't got back around to trying that change, but wouldn't be necessary with these changes.

[JasonElkin]

Yeah, I forgot about how views are loaded which means it's not as easy an approach as it should be.

Hopefully this will be a lot easier in the new backoffice.

[Shazwazza]

Here's the plan

The latest (beta4) changes will be helpful to package devs but we should go a step further by allowing having the package.manifest dictate how a packages assets will be bundled.
The package.manifest JSON key could be "bundleOptions:" The values could be:

• "Default" - which is the default value if none is specified and means that the package's assets will be processed within the main package bundle like it does today. (in debug, files will not be processed)
• "None" - which means that the assets in the package will not be processed at all and will all be requested as individual assets. This will essentially be a bundle that has composite processing turned off for both debug and production.
• "Independent" - which means the packages assets will be processed as it's own separate bundle. (in debug, files will not be processed)

I think this should cater for all cases. Thoughts?

[nathanwoulfe]

This sounds wonderful. Since there are an ever increasing number of ways to approach building a package, and handling the compilation/transpilation/bundling etc of the package assets, it makes sense to give devs more control at the Umbraco end too.

I know the way I'm approaching my build is not really 'normal', so will be awesome to be able to ask Umbraco to trust that I'm shipping optimised files.

[Shazwazza]

Hi all,

I've submitted a PR for this functionality here #10691

If anyone has time for feedback/testing that would be most appreciated!

[nathanwoulfe]

Hey @Shazwazza this looks awesome, really appreciate your efforts (I'm sure I'm not alone)!

Will try to find some time to test the PR, but this sounds like a great solution.