Providing a better experience with JSON storage in property editors

[DanDiplo]

More and more property editors are storing content as blobs of JSON. I've no argument against this, as it's a fast and very flexible way of storing complex models simply - it is easily referenced client-side in the back-office and can be de-serialised to strongly typed models for back-end code. Nested Content, Block List, Grid and even new Media Picker 3 now all store content as JSON.

However, there are a number of issues this causes and I'd like to open up a discussion in how best they can be solved. For me, the main issues are these:

Lack of Referential Integrity

If, for instance, an Element Type is serialised to JSON within (say) the Block List editor there is currently no way of knowing which elements are stored where. There's no referential link between the element and the property editor.

To illustrate this, imagine I create a simple Element Type called Person with one property with an alias of name using a Textstring. I then use this as a block within a Block List editor datatype. I then have a number of pages that I create where I add the Person block and populate the Name field content. If I subsequently go into the back office and edit the Element Type and change the name alias to surname I will find all the content for every instance of the block will have gone in the backend. If I'd used this on every page in a site and it was a required field, it would mean I couldn't publish a single page until I repopulate all that content.

Actually, and this can be even worse, the content hasn't gone until the page is republished - it's still stored in the published content JSON as name - but it is no longer mapped back to the property alias name in the Umbraco UI. If in future I add another property with alias of name to the Element the content will be resurrected! But what happens if my new name property is using a datatype that doesn't expect a text string but, say, a UDI? I can get JSON errors which prevents me publishing the page at all.

Which leads to the second problem...

JSON Conversion Errors

One of the nice things about Umbraco is you can swap the property editor used by a property at any time. This works great when the property editor stores the content as a string - you can swap a rich text editor for a textbox and the worst that can happen is your textbox will contain some HTML tags. However, when the property editor stores content as JSON this can cause issues if the property editor expects the content to be serialised in a specific format.

For instance, if you have a Block List editor and decide you want to swap it for Nested Content (which doesn't seem unreasonable, as you might expect them to work in a similar way) then not only do you lose all the content but when you attempt to publish the page you'll be greeted to a full-screen error dialogue similar to the one below:

This will completely prevent you from publishing the page. The only way to fix it is to remove the property or at least change the property editor to something that doesn't parse JSON.

I understand why Umbraco does it, but is this the most friendly experience? After all Umbraco doesn't warn you when changing the property editor and then leaves you in a state where your entire editing experience is completely broken. Is there a way that this could be handled more gracefully?

• Could there be a warning if you try and swap to an incompatible property editor?
• Could the above error dialogue just be a publish error notification with the full error logged?
• When swapping property editors could there be an option to "blank out" the existing value, so they are nulled, removing any existing JSON?
• Could the property editor be smart and try and convert from similar formats? For instance, if it can't be parsed as as BlockListModel could it be parsed as a Nested Content model and converted on save? This would have been handy for the new Media Picker 3, for instance, which suffers from this same issue.

I don't know all of the answers, which is why I'm opening this for discussion, as I'm sure we have some smart people who can give this consideration.

As more and more complex property editors are developed then I think we need to find a way to solve these core problems which can at best be annoying and at worst break entire sites or end up with large swathes of content getting lost.

[skttl]

Could be interesting to do a concept of data editor mappers, where you could define which data editor you are migrating from, and which you are migrating to. When you attempt to change the data editor of a data type, the UI could warn you if no mappers exist, and give you the option to null existing values.

I think that would be very friendly.

Could also help you migrate from older editors in old projects, eg grid to block list. That would require you to add your own mapper though.

But core could come with mappers for media picker 2 to media picker 3, nested content to Block list, content picker to url picker etc.

[skttl]

The first one, which is why I see challenges with deploy/usync.

[patrickdemooij9]

But wouldn't deploy/usync be able to use the IsConverter function to see what converter to run?

[skttl]

@patrickdemooij9 not in my example, where multiple converters can be chosen from.

[patrickdemooij9]

@skttl, ah I understand now. The only way I could see this working for uSync or deploy is that they track what converters are used. So if you select the converter from NestedContent to BlockEditor, it'll save it (maybe key, alias, something?) in the uSync file, so that it can execute that on importing.

Or we have to offer the same option when importing the data, but then you'll have to remember what converter you chose.

[ronaldbarendse]

All property/data editors can already return a custom IDataValueEditor that allows changing the value when it's sent from the database to the editor and vice versa.