Localization support and transifex integration

[NaitLee]

This contributes to progress of #54 by the following additions:

• Script lang/makepot.sh for gathering & updating strings from source code (of all branches) for translation, and it excludes vanilla strings from this codebase
• Script lang/merge.sh for “merging” translations: merge vanilla strings when packaging for distribution as a whole as a pk3 (will also be run by build.sh by default)
• Script lang/updatepo.sh for updating existing po files manually, useful for offline translation
• Transifex configuration .transifex.yml for integration with the service
• Some initial Chinese translations are also included

Before merge, check if the scripts work on your machine… :)
NOTE: merge these code to a local branch of official repository (not my fork) and then test. it needs other gameplay branches to work

After merging, consider:

• Create your own team on transifex, set this project up with provided configuration, and put the project link to readme
• (or probably I can manage a team, if you want)
• Run lang/makepot.sh, to have new strings (those from other branches) included for translation; consider running it routinely
• Make more strings translatable; move server side strings to client if necessary; some short strings may be better to have prefix for precious meaning
• After some time, make an optional content pack with newest translations for the mod by running lang/merge.sh, and consider to distribute it

If I missed something just ask… :)

[NaitLee]

@LegendaryGuard just have added you to the testing transifex team. I was busy for a very long time and didn't take time to manage that.

[LegendaryGuard]

@LegendaryGuard just have added you to the testing transifex team. I was busy for a very long time and didn't take time to manage that.

Nice! Right now, I just entered into the testing transifex project and I translated few japanese strings to see if it updates in the next time.
How long does that last to be updated? Every (2, 3, 4, ...) days, a week, a month, or...? I don't have clue how it's configured.

[NaitLee]

How long does that last to be updated? Every (2, 3, 4, ...) days, a week, a month, or...? I don't have clue how it's configured.

I just modified the config to 1% translated before pushing to repo. It's been 50%, though I noted that the updated pot contains so many strings... :)

I see a push was issued after I added a string afterwards, with some minutes delay. (I set a 5 mins cooldown as for now)

[LegendaryGuard]

Great!
I tested your branch, the translated strings in-game work as expected. All ok!
Details: merge.sh outputs dirty errors when there's no directory or file:

mkdir: created directory 'compiled/zzz-smb-l10n.pk3dir'
rm: cannot remove 'compiled/zzz-smb-l10n.pk3dir/*': No such file or directory
Merging localization files to compiled/zzz-smb-l10n.pk3dir
rm: cannot remove 'compiled': Is a directory
rm: cannot remove 'compiled/zzz-smb-l10n.*.pk3': No such file or directory
Making localization package compiled/zzz-smb-l10n.a0491c0c.pk3
  adding: common.ja_JP.po (deflated 80%)
  adding: common.zh_CN.po (deflated 78%)
  adding: common.zh_HK.po (deflated 78%)
  adding: common.zh_TW.po (deflated 78%)
  adding: zzz-smb-l10n.pk3.serverpackage (stored 0%)

The console/terminal output should be clean, so I made some adjustments to check the directories and pk3 file in merge.sh:

#!/usr/bin/env bash
cwd="`pwd`"
cd "`dirname $0`/.."

base=xonotic
if [ ! -d "$base" ]; then
    echo "RTFM (README.md)"
    cd $cwd
    exit 1
fi

pk3name=zzz-smb-l10n
dst=compiled/${pk3name}.pk3dir

if [ ! -d "$dst" ]; then
    mkdir -vp $dst
fi
if [ -d "$dst" ] && [ "$(ls -A $dst)" ]; then
    rm -v $dst/*
fi

echo "Merging localization files to $dst"
for i in $(ls lang); do
    if [ -f lang/$i ] && [ -f $base/$i ] && [[ "$i" != "common.pot" ]]; then
        msgcat --to-code UTF-8 $base/$i lang/$i > $dst/$i
    fi
done

if [ ! -n "${ZIP:-}" ]; then
    if [ -e "compiled/${pk3name}.*.pk3" ]; then
        rm -v compiled/${pk3name}.*.pk3
    fi
    # unique id for different versions of localization
    sha256=$(cat compiled/${pk3name}.pk3dir/* | sha256sum)
    pk3path=compiled/$pk3name.${sha256:0:8}.pk3
    echo "Making localization package $pk3path"
    echo "localization package for smb server" >$dst/${pk3name}.pk3.serverpackage
    zip -9 -j $pk3path $dst/*
fi

cd $cwd

It would be ok to update the script in the PR.

Suggestion: that could be interesting is that you can see the percentage of each language of how much it's translated, an example would be something like this:
https://gitlab.com/xonotic/xonotic-data.pk3dir/-/blob/master/languages.txt
https://gitlab.com/xonotic/xonotic-data.pk3dir/-/commit/567ff7590cb21d3910d74a966a9563f0bb86b369#a436ade39b3ccbf92fb013034c399e506c1d7d92

That could save time going to check the completion percentage in transifex. But now I see that the percentage here is included inside commits like these:
13007fa
efc9171

P.D. Going to add some translated japanese strings in transifex to see if it's going as expected besides than adding a new .po file.

[LegendaryGuard]

How long does that last to be updated? Every (2, 3, 4, ...) days, a week, a month, or...? I don't have clue how it's configured.

I just modified the config to 1% translated before pushing to repo. It's been 50%, though I noted that the updated pot contains so many strings... :)

I see a push was issued after I added a string afterwards, with some minutes delay. (I set a 5 mins cooldown as for now)

Ok. We think that the transifex bot should check once a week, what do you think, @MarioSMB ? Or should be once a month? If not, how long do you prefer for the bot to check the updates in transifex of some languages and make a commit?

Once all that is done, let's rename the transifex project name by replacing test-smbmod with xonotic-smbmod (or whatever @MarioSMB names it) and add the updated link to the README in this pending line:
https://github.com/MarioSMB/modpack/pull/55/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R57

P.D. Transifex bot commit from right now: Nice, it checks and updates correctly as expected!

[LegendaryGuard]

I took a look to some details, these words aren't from the main branch:

• https://app.transifex.com/naitlee-soft/test-smbmod/translate/#ja_JP/0623e4e9ca81c758d238c3c848be8892/492711740?q=text%3ACyber
• https://app.transifex.com/naitlee-soft/test-smbmod/translate/#ja_JP/0623e4e9ca81c758d238c3c848be8892/492711747?q=text%3AZ-Saber
• https://app.transifex.com/naitlee-soft/test-smbmod/translate/#ja_JP/0623e4e9ca81c758d238c3c848be8892/492711726?q=text%3ADetective

Why do these words from another branch (cyber) appear there?

[NaitLee]

It would be ok to update the script in the PR.

done

Why do these words from another branch (cyber) appear there?

makepot.sh gathers translatable strings from all mod branches, so that the translation can be used for all variation of mods.

[LegendaryGuard]

Understood. You need to clarify that requires installed this package to execute the translations (msgmerge and xgettext commands):

• Linux:

sudo apt-get update
sudo apt-get gettext

• Windows MSYS:

pacman -S gettext

Ok, so the manual process is:

• Go to lang/ directory
• Execute: ./makepot.sh
• Execute: ./merge.sh
• Copy/Move zzz-smb-l10n*.pk3 into xonotic-data.pk3dir/data, Xonotic/data (or the PATH in README)
• Run the game and enjoy with the translated languages from the mod!
• Execute if there are new translations: ./updatepo.sh

updatepo.sh should remove the *.po~ files if needed, I think.

Btw, I created a new organization to attach the modpack: https://app.transifex.com/team-xonsmb/
Here the empty created project to attach: https://app.transifex.com/team-xonsmb/xonotic-smb-modpack/dashboard/

So, I think you can replace that line in the README to the URL I just shared: https://github.com/MarioSMB/modpack/pull/55/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R57

If you do, consider updating the documentation I just shared.

And I guess the instructions to execute successfully from the organization are there, right? #55 (comment)

[NaitLee]

Things are updated.

./updatepo.sh are intended for those who translate via editing .po files directly (such as me)
(and they just upload .po files to Transifex)
Code maintainers don't have to run it, because updates to .po files are done by Transifex.

To avoid pull request spams, you may configure Transifex to directly push translations to master branch.

[MarioSMB]

Thank you for continuing to work on this - I've given LegendGuardian permission to set it up at the earliest convenience.

[LegendaryGuard]

@NaitLee Sorry for asking you some questions, but I want to make sure I apply everything correctly. Since the team and project have already been created, I'm not sure if I missed any steps in Transifex before running these scripts via GitHub Actions:

1. Is this ok?

2. Does GitHub integration need to be enabled? I think this step requires the owner's permission and the installation of the Transifex GitHub Actions.

[NaitLee]

@LegendaryGuard

The project url should be ok.

The "integration" should be installed, and should be confirmed by the GitHub account owning this repository, then probably configured with target .pot file(s):

Note that this integration isn't GitHub Action but GitHub "App" for a repository:

(Nevertheless you may create a GitHub Action to automatically build translation pk3s. I'm not experienced to this yet.)