Internationalization for non-OS error messages #77

New Issue

Open

opened 2024-03-01 23:47:19 -07:00 by emma · 9 comments

emma commented 2024-03-01 23:47:19 -07:00

Owner

Copy Link

No description provided.

emma added the

bug

help wanted

question

labels 2024-03-01 23:47:19 -07:00

silt changed title from i18n for non-OS error messages to Internationalization for non-OS error messages 2024-03-28 01:42:56 -06:00

silt commented 2024-03-28 04:37:05 -06:00

Owner

Copy Link

alright this turned into a nearly three-hour rabbit hole but i've crawled back out of it with a something that might hopefully be of some use:

#[macro_use]
pub mod tstrutils {
    macro_rules! translated_string {
        ( $name:ident, $( $lang:ident: $tstr:expr ),+ $(,)?) => {
            #[allow(non_camel_case_types)]
            pub struct $name {
                $(
                    pub $lang: &'static str,
                )+
            }

            pub const $name: $name = $name {
                $(
                    $lang: $tstr,
                )+
            };
        };
    }
}

mod s {
    translated_string!(
        HELLO_WORLD,
        en: "Hello, world!",
        fr: "Salut le Monde!",
        jp: "こんにちは、 世界！",
    );

    translated_string!(
        VOCALIZATION,
        dog: "woof",
        bird: "chirp",
        cat: "meow",
        cat_fr: "miau",
    );
}

fn main() {
    println!("{}", s::HELLO_WORLD.en);
    println!("{}", s::HELLO_WORLD.fr);
    println!("{}", s::HELLO_WORLD.jp);

    println!("{}", s::VOCALIZATION.dog);
    println!("{}", s::VOCALIZATION.bird);
    println!("{}", s::VOCALIZATION.cat);
    println!("{}", s::VOCALIZATION.cat_fr);
}

$ cargo run
Hello, world!
Salut le Monde!
こんにちは、 世界！
woof
chirp
meow
miau

no clue how idiomatic any of this is, never used macros before either so there are almost certainly issues here but it at least seems like a reasonable enough starting point. thoughts, praise, criticism, personal attacks, etc are all welcome.

alright this turned into a nearly three-hour rabbit hole but i've crawled back out of it with a something that might hopefully be of some use: ```rust #[macro_use] pub mod tstrutils { macro_rules! translated_string { ( $name:ident, $( $lang:ident: $tstr:expr ),+ $(,)?) => { #[allow(non_camel_case_types)] pub struct $name { $( pub $lang: &'static str, )+ } pub const $name: $name = $name { $( $lang: $tstr, )+ }; }; } } mod s { translated_string!( HELLO_WORLD, en: "Hello, world!", fr: "Salut le Monde!", jp: "こんにちは、 世界！", ); translated_string!( VOCALIZATION, dog: "woof", bird: "chirp", cat: "meow", cat_fr: "miau", ); } fn main() { println!("{}", s::HELLO_WORLD.en); println!("{}", s::HELLO_WORLD.fr); println!("{}", s::HELLO_WORLD.jp); println!("{}", s::VOCALIZATION.dog); println!("{}", s::VOCALIZATION.bird); println!("{}", s::VOCALIZATION.cat); println!("{}", s::VOCALIZATION.cat_fr); } ``` ``` $ cargo run Hello, world! Salut le Monde! こんにちは、 世界！ woof chirp meow miau ``` no clue how idiomatic any of this is, never used macros before either so there are almost certainly issues here but it at least seems like a reasonable enough starting point. thoughts, praise, criticism, personal attacks, etc are all welcome.

silt commented 2024-03-28 04:44:48 -06:00

Owner

Copy Link

and yes i'm aware of how disgusting it is to reuse the identifier like that, but it works and i don't really see any practical issue with it?

and yes i'm aware of how disgusting it is to reuse the identifier like that, but it works and i don't really see any practical issue with it?

silt commented 2024-03-28 05:52:02 -06:00

Owner

Copy Link

one potential issue that i do see is that, because of my choice to do str_id.lang instead of lang.str_id, there's seemingly no simple way to get and store a usable language identifier once at program init and then not incur additional overhead with language selection each time a string is used later on in the program. i made that choice for the sake of keeping related things in the same place, and i stand by it. i believe that this can be elegantly worked around with yet another macro, but unfortunately my only clear idea on that front so far would involve bringing in Syn for its ability to turn strings into identifiers. it's a hefty dependency for smth this minor, but the only alternative i can see at this point is using a hashmap so that we can use strings as identifiers, which i feel only moves that heftiness into runtime.

one potential issue that i do see is that, because of my choice to do `str_id.lang` instead of `lang.str_id`, there's seemingly no simple way to get and store a usable language identifier *once* at program init and then not incur additional overhead with language selection each time a string is used later on in the program. i made that choice for the sake of keeping related things in the same place, and i stand by it. i believe that this can be elegantly worked around with yet another macro, but unfortunately my only clear idea on that front so far would involve bringing in [Syn](https://docs.rs/syn/latest/syn/index.html) for its ability to turn strings into identifiers. it's a hefty dependency for smth this minor, but the only alternative i can see at this point is using a hashmap so that we can use strings as identifiers, which i feel only moves that heftiness into runtime.

silt commented 2024-03-28 06:30:23 -06:00

Owner

Copy Link

upon closer inspection, it appears that i can do what i want without Syn, but it will still require a proc macro. not a huge deal, but it does add some complexity to the repo

upon closer inspection, it appears that i can do what i want without Syn, but it will still require a proc macro. not a huge deal, but it does add some complexity to the repo

trinity commented 2024-03-28 06:53:37 -06:00

Owner

Copy Link

I've been interested in GNU gettext(3) which is the GNU solution to localization. The underscore macro as a shortcut to gettext(3) is sprinkled throughout the coreutils (here it is in their true(1) abomination).

gettext(3) looks up the given message in a catalog and returns localizations. We could use UNIX locales and implement a gettext(3), mindful of the classic complaints about locales (reimplementing some C standard library functions, at least <ctype.h>). Or have an environment variable BONSAI_LOCALE or something.

The tstrutils solution is a lot easier for programmers but a massive source file with all the combinations of locale and string might be somewhat difficult for non-programmer translators and it's harder to audit big files. I would prefer at least a bonsai/i18n/lang database or something.

Another thought I have is somehow adding errnos and strerrors, but that wouldn't be useful for non-errors (e.g. Usage:).

I haven't considered efficiency for any of this and am out of time to write this comment but will come back to this thread later with ideas.

I've been interested in [GNU gettext(3)](https://www.man7.org/linux/man-pages/man3/gettext.3.html) which is the GNU solution to localization. [The underscore macro as a shortcut to gettext(3)](https://www.linuxquestions.org/questions/programming-9/c-underscore-before-string-870510/#post4300788) is sprinkled throughout the coreutils ([here it is in their true(1) abomination](https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/true.c#n36)). gettext(3) looks up the given message in a catalog and returns localizations. We could use UNIX locales and implement a gettext(3), mindful of [the classic complaints about locales](https://github.com/mpv-player/mpv/commit/1e70e82baa9193f6f027338b0fab0f5078971fbe) (reimplementing some C standard library functions, at least `<ctype.h>`). Or have an environment variable `BONSAI_LOCALE` or something. The `tstrutils` solution is a lot easier for programmers but a massive source file with all the combinations of locale and string might be somewhat difficult for non-programmer translators and it's harder to audit big files. I would prefer at least a bonsai/i18n/lang database or something. Another thought I have is somehow adding errnos and strerrors, but that wouldn't be useful for non-errors (e.g. `Usage:`). I haven't considered efficiency for any of this and am out of time to write this comment but will come back to this thread later with ideas.

silt commented 2024-03-28 15:55:27 -06:00

Owner

Copy Link

after banging my head against proc macros for the past 8 or so hours, i've decided to call it quits on that front.
here's a more reasonable solution with similar ease-of-use to GNU's _():

mod s;
use std::io::Result;

fn main() -> Result<()> {
    println!("\nField accessors:");
    println!("{}", s::HelloWorld.en_US);
    println!("{}", s::HelloWorld.fr_FR);
    println!("{}", s::HelloWorld.ja_JP);

    println!("\ntranslated_string()!:");
    println!("{}", s::HelloWorld.in_locale("en_US")?);
    println!("{}", s::HelloWorld.in_locale("fr_FR")?);
    println!("{}", s::HelloWorld.in_locale("ja_JP")?);

    // psst hey look over here this is the cool part
    println!("\ntranslated_string()! with LANG var:");
    println!("{}", s::HelloWorld);

    println!("\ntranslated_string()! error on untranslated:");
    println!("{}", s::HelloWorld.in_locale("blah")?);

    Ok(())
}

#[macro_export]
macro_rules! translated_string {
    ( $name:ident, $( $lang:ident: $tstr:expr ),+ $(,)?) => {
        #[allow(non_snake_case)]
        #[derive(Debug)]
        pub struct $name {
            $(
                pub $lang: &'static str,
            )+
        }
        
        #[allow(non_upper_case_globals)]
        pub const $name: $name = $name {
            $(
                $lang: $tstr,
            )+
        };

        use std::io::{Result, Error};
        impl $name {
            pub fn in_locale(&self, lang: &str) -> Result<&'static str> {
                match lang {
                    $( stringify!($lang) => Ok(self.$lang), )+
                    _ => Err(Error::other(format!(
                        "Untranslated string for current locale: {}", 
                        stringify!($name)
                    )))
                }
            }
        }

        impl std::fmt::Display for $name {
            fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
                write!(f, "{}", self.in_locale(&std::env::var("LANG").unwrap().split('.').take(1).collect::<String>().replace("-", "_")).unwrap())
            }
        }
    };
}

use trans::translated_string;

translated_string!(
    HelloWorld,
    en_US: "Hello, world!",
    fr_FR: "Salut le Monde!",
    ja_JP: "こんにちは、 世界！",
);

it isn't very good or optimized code, but it does work!
this assumes that LANG is set to a standard RFC5646 language tag (or the common-but-nonstandard variant where subtags are instead delimited by an underscore to work around programming language syntax), but it does gracefully ignore any .encoding specifiers in order to maintain compatibility with normal Linux (*nix?) systems

after banging my head against proc macros for the past 8 or so hours, i've decided to call it quits on that front. here's a more reasonable solution with similar ease-of-use to GNU's `_()`: ```rust mod s; use std::io::Result; fn main() -> Result<()> { println!("\nField accessors:"); println!("{}", s::HelloWorld.en_US); println!("{}", s::HelloWorld.fr_FR); println!("{}", s::HelloWorld.ja_JP); println!("\ntranslated_string()!:"); println!("{}", s::HelloWorld.in_locale("en_US")?); println!("{}", s::HelloWorld.in_locale("fr_FR")?); println!("{}", s::HelloWorld.in_locale("ja_JP")?); // psst hey look over here this is the cool part println!("\ntranslated_string()! with LANG var:"); println!("{}", s::HelloWorld); println!("\ntranslated_string()! error on untranslated:"); println!("{}", s::HelloWorld.in_locale("blah")?); Ok(()) } ``` ```rust #[macro_export] macro_rules! translated_string { ( $name:ident, $( $lang:ident: $tstr:expr ),+ $(,)?) => { #[allow(non_snake_case)] #[derive(Debug)] pub struct $name { $( pub $lang: &'static str, )+ } #[allow(non_upper_case_globals)] pub const $name: $name = $name { $( $lang: $tstr, )+ }; use std::io::{Result, Error}; impl $name { pub fn in_locale(&self, lang: &str) -> Result<&'static str> { match lang { $( stringify!($lang) => Ok(self.$lang), )+ _ => Err(Error::other(format!( "Untranslated string for current locale: {}", stringify!($name) ))) } } } impl std::fmt::Display for $name { fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result { write!(f, "{}", self.in_locale(&std::env::var("LANG").unwrap().split('.').take(1).collect::<String>().replace("-", "_")).unwrap()) } } }; } ``` ```rust use trans::translated_string; translated_string!( HelloWorld, en_US: "Hello, world!", fr_FR: "Salut le Monde!", ja_JP: "こんにちは、 世界！", ); ``` it isn't very good or optimized code, but it does work! this assumes that `LANG` is set to a standard [RFC5646](https://datatracker.ietf.org/doc/html/rfc5646) language tag (or the common-but-nonstandard variant where subtags are instead delimited by an underscore to work around programming language syntax), but it does gracefully ignore any `.encoding` specifiers in order to maintain compatibility with normal Linux (*nix?) systems

silt self-assigned this 2024-03-28 15:56:01 -06:00

silt commented 2024-03-29 21:52:16 -06:00

Owner

Copy Link

a lot easier for programmers but a massive source file with all the combinations of locale and string might be somewhat difficult for non-programmer translators

i've been pondering this issue since you mentioned it, and the conclusion i've come to is this:
ultimately, the goal of this isn't translation—it's localization. localization isn't just blindly mapping words onto other words. it requires context, not only for the locale for which the text is being translated, but for the actual text itself. non-technical people who get scared off by fairly lightweight syntax (fwiw i could make the macro even lighter on this; the quotes are not strictly necessary) aren't going to provide localized strings that are of any more use than what google translate could give us.

it's harder to audit big files

i've seen the localization files for other large projects, and they aren't any better. since these are rust source files, as long as everything is imported under the s module, splitting them up should not actually be an issue.

> a lot easier for programmers but a massive source file with all the combinations of locale and string might be somewhat difficult for non-programmer translators i've been pondering this issue since you mentioned it, and the conclusion i've come to is this: ultimately, the goal of this isn't translation—it's *localization*. localization isn't just blindly mapping words onto other words. it requires context, not only for the locale for which the text is being translated, but for the actual text itself. non-technical people who get scared off by fairly lightweight syntax (fwiw i could make the macro even lighter on this; the quotes are not strictly necessary) aren't going to provide localized strings that are of any more use than what google translate could give us. > it's harder to audit big files i've seen the localization files for other large projects, and they aren't any better. since these are rust source files, as long as everything is imported under the `s` module, splitting them up should not actually be an issue.

emma commented 2024-03-29 21:53:25 -06:00

Author

Owner

Copy Link

this assumes that LANG is set to a standard RFC5646 language tag (or the common-but-nonstandard variant where subtags are instead delimited by an underscore to work around programming language syntax), but it does gracefully ignore any .encoding specifiers in order to maintain compatibility with normal Linux (*nix?) systems

The standard is a combination of ISO 639 and ISO 3166 language and country identifiers, not IETF BCP 47 language tags.

> this assumes that `LANG` is set to a standard [RFC5646](https://datatracker.ietf.org/doc/html/rfc5646) language tag (or the common-but-nonstandard variant where subtags are instead delimited by an underscore to work around programming language syntax), but it does gracefully ignore any `.encoding` specifiers in order to maintain compatibility with normal Linux (*nix?) systems The standard is [a combination of ISO 639 and ISO 3166 language and country identifiers](https://wiki.archlinux.org/title/Locale), not IETF BCP 47 language tags.

silt commented 2024-03-29 21:57:00 -06:00

Owner

Copy Link

oops, tysm for the correction :D

oops, tysm for the correction :D

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

getopt-formatting

optimizations

libfileis

style-c

stris

sleep

pg

sysxs

0.14.1

0.14.0

0.13.23

0.13.22

0.13.21

0.13.20

0.13.19

0.13.18

0.13.17

0.13.16

0.13.15

0.13.14

0.13.13

0.13.12

0.13.11

0.13.10

0.13.9

0.13.8

0.13.7

0.13.6

0.13.5

0.13.4

0.13.3

0.13.2

0.13.1

0.13.0

0.12.0

0.11.0

0.10.3

0.10.2

0.10.1

0.10.0

Labels

Clear labels   

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

joke

you are silly!

  

question

More information is needed

  

wontfix

This won't be fixed

No Label

bug

duplicate

enhancement

help wanted

invalid

joke

question

wontfix

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

silt

3 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: bonsai/harakit#77

No description provided.