A new pattern for translating metadata in tabular data packages

[augusto-herrmann]

Frictionless already has a couple of patterns for translation of tabular data. The first involves creating additional columns with a language tag in the end, e.g. "name@en". The other one suggests creating different files in the same path, one CSV file for each language.

Both of these require duplicating a lot of data. Sometimes you have a lot of numeric columns, so it makes no sense to translate the data (which is just numbers after all), but you still want to translate the metadata (i.e. column names and descriptions). Other times, you could translate the data, but the volume is so great that it isn't feasible in the short term, but you still want to release the untranslated data with translated metadata.

With that in consideration, I would like to suggest a third pattern for translating just the metadata in a tabular data package.

1. Leave the CSV files untouched.
2. Create multiple datapackage.xx.json files, replacing xx with the language tag for the file. The strings in those file should all be in the respective language. All other definitions in the tabular data package (e.g. column type, missing values, restrictions, etc.) should be exactly the same in every file.

We are already using this approach in a data package we publish as open data. A challenge of this pattern is keeping the non-textual information in sync among all those files. To tackle this challenge, we create the datapackage.xx.json files in Python, defining all the data validation properties in there. All text-based strings (titles and descriptions) are read from a separate yaml file. If we want to add a new language, just copy the datapackage-strings.xx.yaml file to a new one, translate it, run the Python script and we get the new respective datapackage.xx.json file. If we want to change any of the data validation properties, we edit the Python script and it will generate all of the datapackage.xx.json files, keeping data validation in sync.

Any comments on alternate ways to translate just the metadata (titles and descriptions) in tabular data packages would be appreciated. If we find out this pattern I describe is really the best way to do it, perhaps we should add it to the translation support patterns.

[lwinfree]

I really like this idea @augusto-herrmann!

[augusto-herrmann]

Here is an example of a data package using this idea:

• in English
• in Portuguese

Should I open an issue for this in frictionless/specs?

[AFoletti]

I have a similar problem with my datapackages, and I solved it in another way.
I am not duplicating data in the CSV nor proliferate datapackages. Rather, I add the @ tag to the needed elements in the datapackage itself.
Example:

{
    "resources": [{
    "name": "ogd10_energieforschungstatistik_ch.csv",
    "languages": ["de", "fr"],
[...]
    {
        "name": "finanzquelle",
        "type": "string",
        "format": "default",
        "title@de": "Finanzquelle",
        "title@fr": "Source de financement",
        "description@de": "In- oder ausländische Stelle, welche die Forschung finanziert",
        "description@fr": "Entité nationale ou étrangère qui finance la recherche"
    }
[...]
}

The Data Package Validator on DataHub tells me it's valid, but I am not really confident that I am using the datapackage the way it's meant to be used.
What does the community think about that?

[augusto-herrmann]

I think your pattern looks similar to the current inline pattern, which is intended for l10n of data, but applied to metadata instead, so that you store the column only once.

IMHO it is a sensible approach and is also a possible solution to specs#355 for only translating fields names for which a translation is available. It would need to be supported in all metadata that takes a string, including at the data package level (which I have seen that your data package file does have but the snippet does not show).

How about the use case where you have both the data and metadata multilingual? I think the @ tag could be interpreted to mean the language of the data when used as the value of a field name, and to mean the language of the metadata when used as the key of other properties. For instance, a possible way to combine the inline pattern with the approach you suggested:

{
    "title@en": "Multilingual dataset example",
    "title@fr": "Exemple de jeu de données multilingue",
    "resources": [{
    "name": "example.csv",
    "languages": ["en", "fr"],
[...]
    {
        "name": "description",
        "type": "string",
        "format": "default",
        "title@en": "The description",
        "title@fr": "La description (en anglais)",
    },
    {
        "name": "description@fr",
        "type": "string",
        "format": "default",
        "title@en": "The description (in French)",
        "title@fr": "La description",
    }
[...]
}

id,description,description@fr
1,A sentence describing item 1 in English.,Une phrase qui décrite ligne 1 en français.
2,Yet another row.,Une ligne de plus.

[AFoletti]

Yes, it is actually the inline pattern approach applied to metadata instead of data, both at the package and resource level. It works in my opinion quite well and allows me to only translate those elements where I actually have more than one language.

And sure, you can combine it with multlingual data as you suggested.