feat: New command `lang:sync` #9023

neznaika0 · 2024-07-05T17:49:58Z

Description
A simple command to copy existing translation files to a new locale.
From the file Language/en/Example.php:

<?php

return [
    'title' => 'Default title',
    'status' => [
        'error'    => 'Error!',
        'done'     => 'Done!',
        'critical' => 'Critical!',
    ],
    'nullableKey' => null,
    'numericKey' => 100500,
    'more' => [
        'nested' => [
            'key' => 'More nested key...',
        ],
    ],
];

After the command php spark lang:sync --locale en --target ru we get Language/ru/Example.php .

<?php

return array (
  'more' => 
  array (
    'nested' => 
    array (
      'key' => 'Example.more.nested.key',
    ),
  ),
  'nullableKey' => 'Example.nullableKey',
  'numericKey' => 'Example.numericKey',
  'status' => 
  array (
    'error' => 'Example.status.error',
    'done' => 'Example.status.done',
    'critical' => 'Example.status.critical',
  ),
  'title' => 'Example.title',
);

applied cs-fix:

<?php

return [
    'more' => [
        'nested' => [
            'key' => 'Example.more.nested.key',
        ],
    ],
    'nullableKey' => 'Example.nullableKey',
    'numericKey'  => 'Example.numericKey',
    'status'      => [
        'error'    => 'Example.status.error',
        'done'     => 'Example.status.done',
        'critical' => 'Example.status.critical',
    ],
    'title' => 'Example.title',
];

And so it is with all other translations

=====

When there are translations in the target folder Language/ru, their keys are saved only if they exist in the Language/en folder:

Language/ru/Example.php:

<?php

return [
    // Deleted keys
    // 'more' => [
    //     'nested' => [
    //         'key' => 'Example.more.nested.key',
    //     ],
    // ],
    'nullableKey' => null,
    'numericKey'  => 100000,
    'status'      => [
        'error'    => 'ru.status.error',
        'done'     => 'ru.status.done',
        'critical' => 'ru.status.critical',
    ],
    'title' => 'ru.title',
    'mistakeKey' => 'ru.mistake_key',
];

Language/ru/Example.php after sync:

<?php

return [
    'more' => [
        'nested' => [
            'key' => 'Example.more.nested.key',
        ],
    ],
    'nullableKey' => null,
    'numericKey'  => 100000,
    'status'      => [
        'error'    => 'ru.status.error',
        'done'     => 'ru.status.done',
        'critical' => 'ru.status.critical',
    ],
    'title' => 'ru.title',
];

Checklist:

Securely signed commits
Component(s) with PHPDoc blocks, only if necessary or adds value
Unit testing, with >80% coverage
User guide updated
Conforms to style guide

neznaika0 · 2024-07-05T17:51:14Z

There are some questions:

Does anyone else need it? 😄
I have indicated the reason for using it as a more precise translation control.
I need help with the documentation. I don't know English
Formatting new arrays requires the use of php-cs-fixer. This was discussed in another PR feat: Language translations finder and update #7896 (comment) . How can we take this out for general use?
phpstan. I don't quite understand what he wants from me. I have specified a common type for the values

neznaika0 · 2024-07-05T18:00:39Z

Userguide:

Changelog:

kenjis · 2024-08-01T02:21:55Z

I don't get the description "When there are translations in the target folder Language/ru, their keys are saved only if they exist in the Language/en folder:". Is that correct?

If app/Language/en/Example.php is:

return [
    // Deleted keys
    // 'more' => [
    //     'nested' => [
    //         'key' => 'Example.more.nested.key',
    //     ],
    // ],
    'nullableKey' => null,
    'numericKey'  => 100000,
    'status'      => [
        'error'    => 'ru.status.error',
        'done'     => 'ru.status.done',
        'critical' => 'ru.status.critical',
    ],
    'title' => 'ru.title',
    'mistakeKey' => 'ru.mistake_key',
];

Then, app/Language/ru/Example.php will be:

return [
    'mistakeKey'  => 'Example.mistakeKey',
    'nullableKey' => 'Example.nullableKey',
    'numericKey'  => 'Example.numericKey',
    'status'      => [
        'error'    => 'Example.status.error',
        'done'     => 'Example.status.done',
        'critical' => 'Example.status.critical',
    ],
    'title' => 'Example.title',
];

That is, deleted keys are gone.

kenjis · 2024-08-01T02:26:50Z

If app/Language/en/Example.php is:

return [
    'title'  => 'Default title',
    'status' => [
        'error'    => 'Error!',
        'done'     => 'Done!',
        'critical' => 'Critical!',
    ],
    'nullableKey' => null,
    'numericKey'  => 100500,
    'more'        => [
        'nested' => [
            'key' => 'More nested key...',
        ],
    ],
];

Then, app/Language/ru/Example.php will be:

return [
    'more' => [
        'nested' => [
            'key' => 'Example.more.nested.key',
        ],
    ],
    'nullableKey' => 'Example.nullableKey',
    'numericKey'  => 'Example.numericKey',
    'status'      => [
        'error'    => 'Example.status.error',
        'done'     => 'Example.status.done',
        'critical' => 'Example.status.critical',
    ],
    'title' => 'Example.title',
];

That is, the key order changed.
This makes comparison of the two files, original and translated, difficult.

kenjis · 2024-08-01T02:36:37Z

user_guide_src/source/outgoing/localization.rst

+.. versionadded:: 4.6.0
+
+After working on your translation for the current language for a long time, in some cases you will need to create files for another language.
+The ``spark lang:find`` command can be used, it is not prohibited. But it cannot fully detect absolutely all translations. Especially if the parameters are dynamically set as ``lang('App.status.' . $key, ['payload' => 'John'], 'en')``.


I don't know why you added , it is not prohibited. What do you want to say by it?

Ah, I don't get the following whole explanation.

The spark lang:find command can be used, it is not prohibited. But it cannot fully detect absolutely all translations. Especially if the parameters are dynamically set as lang('App.status.' . $key, ['payload' => 'John'], 'en'). In this case, the best solution is to copy the finished language files and translate them. This way you can save your unique keys that the command did not find.

"to copy the finished language files and translate them"
What do I need to do exactly? If I copy the file, there is no need of lang commands.

I also don't get why the explanation is there. What's the relation with the lang:sync command?

I warned that my English is bad.

I wrote that the lang:find helps you find NEW keys. What if you don't need to search? Or lang:find couldn't find complex translations as lang('App.title', [], $locale)?
In this case, you must manually add the translation.

What do I need to do exactly? If I copy the file, there is no need of lang commands.

You will copy a identical array. After that, you need to manually search for and change existing keys (if available). When there is no file for --target locale, yes it is logical (copy-paste).
Command is more suitable for merging.

Is it embarrassing? Then why do the make:controller and others commands, is it possible to copy an available controller?

remove , it is not prohibited. I think it is not needed.

move the whole explanation to the section "Generating Translation Files via Command". Because it is an explanation for lang:find, not lang:sync.

user_guide_src/source/outgoing/localization.rst

kenjis · 2024-08-01T02:47:17Z

This command would be useful.
However, I would like the order of the keys to match the original.

neznaika0 · 2024-08-01T05:12:09Z

I don't get the description "When there are translations in the target folder Language/ru, their keys are saved only if they exist in the Language/en folder:". Is that correct?

If app/Language/en/Example.php is:
...

No. Look at the order of the files. We from EN combine RU. There are translations in RU ru.title, so the keys are saved and not overwritten Example.title. Old keys remain.

That is, the key order changed.
This makes comparison of the two files, original and translated, difficult.

Yes. The order has changed. I'm sorting a new array. I think it can be fixed. But the keys will be chaotic when combined.

kenjis · 2024-08-02T07:28:29Z

This PR branch is too old. Can you rebase?

kenjis · 2024-08-02T07:34:49Z

No. Look at the order of the files. We from EN combine RU. There are translations in RU ru.title, so the keys are saved and not overwritten Example.title. Old keys remain.

Okay, I got what you say.

app/Language/en/Example.php:

return [
    'title'  => 'Default title',
    'status' => [
        'error'    => 'Error!',
        'done'     => 'Done!',
        'critical' => 'Critical!',
    ],
    'nullableKey' => null,
    'numericKey'  => 100500,
    'more'        => [
        'nested' => [
            'key' => 'More nested key...',
        ],
    ],
];

app/Language/ru/Example.php:

<?php

return [
    // Deleted keys
    // 'more' => [
    //     'nested' => [
    //         'key' => 'Example.more.nested.key',
    //     ],
    // ],
    'nullableKey' => null,
    'numericKey'  => 100000,
    'status'      => [
        'error'    => 'ru.status.error',
        'done'     => 'ru.status.done',
        'critical' => 'ru.status.critical',
    ],
    'title' => 'ru.title',
    'mistakeKey' => 'ru.mistake_key',
];

$ php spark lang:sync --locale en --target ru
$ composer cs-fix

app/Language/ru/Example.php:

return [
    'more' => [
        'nested' => [
            'key' => 'Example.more.nested.key',
        ],
    ],
    'nullableKey' => null,
    'numericKey'  => 100000,
    'status'      => [
        'error'    => 'ru.status.error',
        'done'     => 'ru.status.done',
        'critical' => 'ru.status.critical',
    ],
    'title' => 'ru.title',
];

kenjis · 2024-08-02T08:04:30Z

Yes. The order has changed. I'm sorting a new array. I think it can be fixed. But the keys will be chaotic when combined.

I think the key order should not be changed.

Can't you rewrite with recursive calls?
This is PoC sample code by ChatGPT:

<?php

function syncTranslationsRecursive($original, $translations) {
    // Initialize an array to store the results
    $syncedTranslations = [];

    // Loop through the original message array
    foreach ($original as $key => $message) {
        if (is_array($message)) {
            // If the message is an array, recursively synchronize it
            $syncedTranslations[$key] = syncTranslationsRecursive(
                $message,
                array_key_exists($key, $translations) ? $translations[$key] : []
            );
        } else {
            // If the message is not an array, check if the translations array contains the same key
            if (array_key_exists($key, $translations)) {
                $syncedTranslations[$key] = $translations[$key];
            } else {
                // If the key doesn't exist in the translations array, set an empty string
                $syncedTranslations[$key] = '';
            }
        }
    }

    return $syncedTranslations;
}

// Original message array
$original = [
    'title' => 'Default title',
    'status' => [
        'error'    => 'Error!',
        'done'     => 'Done!',
        'critical' => 'Critical!',
    ],
    'more' => [
        'nested' => [
            'key' => 'More nested key...',
        ],
    ],
];

// Translated message array (partially translated)
$translations = [
    'title' => 'Título predeterminado',
    'status' => [
        'error' => '¡Error!',
        'done'  => '¡Hecho!',
    ],
];

// Synchronize the translations array with the original message array
$syncedTranslations = syncTranslationsRecursive($original, $translations);

// Print the results
print_r($syncedTranslations);

neznaika0 · 2024-08-02T10:14:35Z

What is the disadvantage of the current array merge?

https://github.com/neznaika0/codeigniter-dev/blob/19f9440bd9deb5131079ff427781f6f14de99584/system/Helpers/Array/ArrayHelper.php#L327-L341

I can just remove the sorting and everything looks like you wanted.
https://github.com/neznaika0/codeigniter-dev/blob/19f9440bd9deb5131079ff427781f6f14de99584/system/Commands/Translation/LocalizationSync.php#L136-L137

But, if the target file has keys, then they are added to the end.

kenjis · 2024-08-02T11:05:08Z

I would like the key order is never changed in any case.
If so, I don't care about the implementation.

neznaika0 · 2024-08-05T05:46:47Z

Results based on the data above:

// translation RU is empty
return [
    'title'  => 'Example.title',
    'status' => [
        'error'    => 'Example.status.error',
        'done'     => 'Example.status.done',
        'critical' => 'Example.status.critical',
    ],
    'nullableKey' => 'Example.nullableKey',
    'numericKey'  => 'Example.numericKey',
    'more'        => [
        'nested' => [
            'key' => 'Example.more.nested.key',
        ],
    ],
];

// translation RU exist
return [
    'nullableKey' => null,
    'numericKey'  => 100000,
    'status'      => [
        'error'    => 'ru.status.error',
        'done'     => 'ru.status.done',
        'critical' => 'ru.status.critical',
    ],
    'title' => 'ru.title',
    'more'  => [
        'nested' => [
            'key' => 'Example.more.nested.key',
        ],
    ],
];

kenjis · 2024-08-05T07:11:58Z

When translation RU exist, the key order will be different from EN.
In any case, the key order should be the same in all lang files.

neznaika0 · 2024-08-08T14:56:15Z

// Before: app/Language/en/Example.php
return [
    'title'  => 'Default title',
    'status' => [
        'error'    => 'Error!',
        'done'     => 'Done!',
        'critical' => 'Critical!',
    ],
    'nullableKey' => null,
    'array'       => [],
    'numericKey'  => 100500,
    'more'        => [
        'nested' => [
            'key' => 'More nested key...',
        ],
    ],
];

// Before: app/Language/ru/Example.php
return [
    // Deleted keys
    // 'more' => [
    //     'nested' => [
    //         'key' => 'Example.more.nested.key',
    //     ],
    // ],
    'nullableKey' => null,
    'status'      => [
        'critical' => 'ru.status.critical',
        'done'     => 'ru.status.done',
        'error'    => 'ru.status.error',
    ],
    'mistakeKey' => 'ru.mistake_key',
    'numericKey'  => 100000,
    'title' => 'ru.title',
];

// After: app/Language/ru/Example.php
// 1. Right sort as in EN
// 2. Delete 'mistakeKey' (in EN language not exist)
// 3. Untranslated keys have a placeholder ''Example.more.nested.key'
// 4. Previously translated keys are saved
return [
    'title'  => 'ru.title',
    'status' => [
        'error'    => 'ru.status.error',
        'done'     => 'ru.status.done',
        'critical' => 'ru.status.critical',
    ],
    'nullableKey' => null,
    'array'       => [
    ],
    'numericKey' => 100000,
    'more'       => [
        'nested' => [
            'key' => 'Example.more.nested.key',
        ],
    ],
];

The array intersection search function ArrayHelper::intersectKeyRecursive is no longer needed.
Combining arrays should now initially not have extra keys (current mistakeKey) from the target array.

datamweb · 2024-08-09T00:29:25Z

I prefer the following result. To make it easier for the developer to translate the file.

// After: app/Language/ru/Example.php
return [
    'title'  => '(To be translated) Default title',
    'status' => [
        'error'    => '(To be translated) Error!',
        'done'     => '(To be translated) Done!',
        'critical' => '(To be translated) Critical!',
    ],
    'nullableKey' => null,
    'array'       => [
    ],
    'numericKey' => 100000,
    'more'       => [
        'nested' => [
            'key' => '(To be translated) More nested key...',
        ],
    ],
];

kenjis · 2024-08-09T02:25:50Z

tests/system/Commands/Translation/LocalizationSyncTest.php

+            'b' => 2000,
+            'c' => null,


I think we do not accept int or null as lang messages.
So should show an error message when processing?

Maybe. These values are more for the translation search test.
I can filter out the bad lines for RU keys, if it's important.

Please do one of the following:

show errors (or just throw Exception?) if there are invalid values in any lang files.

add comments in the test code that "values other than string is invalid, but we use invalid values just for testing".

neznaika0 · 2024-08-09T05:46:41Z

@datamweb I think it's wrong. If the current locale is RU and the translations are the same (EN), it is unclear on the page whether the translation was made or not. Because the fallback translation is EN.

Additional text interferes with the preview of the page (too long for some places).

Placeholders clearly show untranslated lines.

It is possible to make it configurable --replace-old

ddevsr added the 4.6 label Jul 6, 2024

kenjis added the enhancement PRs that improve existing functionalities label Jul 6, 2024

kenjis reviewed Aug 1, 2024

View reviewed changes

datamweb reviewed Aug 1, 2024

View reviewed changes

user_guide_src/source/outgoing/localization.rst Outdated Show resolved Hide resolved

neznaika0 force-pushed the feat-lang-sync branch from cbd2d0f to 19f9440 Compare August 2, 2024 09:49

neznaika0 force-pushed the feat-lang-sync branch from 19f9440 to cf93a62 Compare August 5, 2024 05:37

neznaika0 force-pushed the feat-lang-sync branch from cf93a62 to 6bc2c49 Compare August 8, 2024 14:55

kenjis reviewed Aug 9, 2024

View reviewed changes

neznaika0 added 6 commits November 4, 2024 10:08

feat: Add lang:sync command

9ead16f

test: Add tests for lang:sync command

0c7a010

feat: Add array helper intersectKeyRecursive

e5bb6cf

docs: Add changelog

e8f6549

docs: Add description for command

c67616d

fix: Improve condition

e1e8931

neznaika0 added 6 commits November 4, 2024 10:08

fix: docs title underline too short

0b89adc

fix: typo

eeef980

fix: Delete debug info

81e570e

fix: Remove sorting lang keys

6460df0

fix: key sorting

e1d5d1c

Remove array helper intersectKeyRecursive

435d8d6

neznaika0 force-pushed the feat-lang-sync branch from 6bc2c49 to 435d8d6 Compare November 4, 2024 07:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: New command `lang:sync` #9023

feat: New command `lang:sync` #9023

neznaika0 commented Jul 5, 2024 •

edited by kenjis

Loading

neznaika0 commented Jul 5, 2024 •

edited

Loading

neznaika0 commented Jul 5, 2024

kenjis commented Aug 1, 2024 •

edited

Loading

kenjis commented Aug 1, 2024

kenjis Aug 1, 2024

kenjis Aug 1, 2024

neznaika0 Aug 1, 2024

kenjis Aug 9, 2024

kenjis commented Aug 1, 2024

neznaika0 commented Aug 1, 2024

kenjis commented Aug 2, 2024

kenjis commented Aug 2, 2024

kenjis commented Aug 2, 2024

neznaika0 commented Aug 2, 2024

kenjis commented Aug 2, 2024

neznaika0 commented Aug 5, 2024 •

edited

Loading

kenjis commented Aug 5, 2024

neznaika0 commented Aug 8, 2024

datamweb commented Aug 9, 2024 •

edited

Loading

kenjis Aug 9, 2024

neznaika0 Aug 9, 2024

kenjis Aug 10, 2024

neznaika0 commented Aug 9, 2024 •

edited

Loading

feat: New command lang:sync #9023

Are you sure you want to change the base?

feat: New command lang:sync #9023

Conversation

neznaika0 commented Jul 5, 2024 • edited by kenjis Loading

neznaika0 commented Jul 5, 2024 • edited Loading

neznaika0 commented Jul 5, 2024

kenjis commented Aug 1, 2024 • edited Loading

kenjis commented Aug 1, 2024

kenjis Aug 1, 2024

Choose a reason for hiding this comment

kenjis Aug 1, 2024

Choose a reason for hiding this comment

neznaika0 Aug 1, 2024

Choose a reason for hiding this comment

kenjis Aug 9, 2024

Choose a reason for hiding this comment

kenjis commented Aug 1, 2024

neznaika0 commented Aug 1, 2024

kenjis commented Aug 2, 2024

kenjis commented Aug 2, 2024

kenjis commented Aug 2, 2024

neznaika0 commented Aug 2, 2024

kenjis commented Aug 2, 2024

neznaika0 commented Aug 5, 2024 • edited Loading

kenjis commented Aug 5, 2024

neznaika0 commented Aug 8, 2024

datamweb commented Aug 9, 2024 • edited Loading

kenjis Aug 9, 2024

Choose a reason for hiding this comment

neznaika0 Aug 9, 2024

Choose a reason for hiding this comment

kenjis Aug 10, 2024

Choose a reason for hiding this comment

neznaika0 commented Aug 9, 2024 • edited Loading

feat: New command `lang:sync` #9023

feat: New command `lang:sync` #9023

neznaika0 commented Jul 5, 2024 •

edited by kenjis

Loading

neznaika0 commented Jul 5, 2024 •

edited

Loading

kenjis commented Aug 1, 2024 •

edited

Loading

neznaika0 commented Aug 5, 2024 •

edited

Loading

datamweb commented Aug 9, 2024 •

edited

Loading

neznaika0 commented Aug 9, 2024 •

edited

Loading