kv(1) - key/value #148

New Issue

sashakoshka · 2024-07-17T06:08:12Z

sashakoshka commented

2024-07-17 06:08:12 +00:00

Rationale

A lot of scripts read configuration data by sourcing other scripts which are disguised as key/value config files, and I don't even need to explain why that's a bad idea. The reason they do this is because its generally the path of least resistance. Even some software published by freedesktop (specifically xdg-user-dir(1)) does this.

kv(1) would offer a more robust way of doing this by allowing key/value files of a specified format to be queried and modified with simple one-liners.

Invocation

The program would be invoked roughly like this:

kv set|get FILE [KEY[=VALUE]...] [OPTION...]

The program would have two modes of operation. set would change values in the file, and get would retrieve values from the file. Following the mode would be the file to operate on, and then a list of key/value pairs. Specifying the value of a pair in get mode would return that value (as a default) if the pair was not found within the file. Not specifying the value of a pair in set mode would delete that pair from the file.

getting a pair would cause the program to output the value of that pair to stdout.

FILE could be specified as -, which would cause the program to read from stdin. set mode could still be used in this scenario in conjunction with some option to redirect the output to stdout, another file, etc.

File

The file would be a list of lines, each containing either a key/value pair or a comment. The key would be separated from the value with a = character, where everything before is part of the key and everything after is part of the value.

Caveats

Multiple pairs can be specified. I still haven't decided how these would be output in get mode. Perhaps, since the file is separated line by line, each value could be output on its own line.

A malformed file might specify multiple values for the same key. Perhaps the program could just operate on the first one and ignore the rest.

This might be a better task for the already planned/implemented field-oriented utilities (fop, fis, tex, fit, etc). If it is then tell me and I won't make the program.

## Rationale A lot of scripts read configuration data by sourcing other scripts which are disguised as key/value config files, and I don't even need to explain why that's a bad idea. The reason they do this is because its generally the path of least resistance. Even some software published by freedesktop (specifically `xdg-user-dir(1)`) does this. `kv(1)` would offer a more robust way of doing this by allowing key/value files of a specified format to be queried and modified with simple one-liners. ## Invocation The program would be invoked roughly like this: `kv set|get FILE [KEY[=VALUE]...] [OPTION...]` The program would have two modes of operation. `set` would change values in the file, and `get` would retrieve values from the file. Following the mode would be the file to operate on, and then a list of key/value pairs. Specifying the value of a pair in `get` mode would return that value (as a default) if the pair was not found within the file. Not specifying the value of a pair in `set` mode would delete that pair from the file. `get`ting a pair would cause the program to output the value of that pair to stdout. FILE could be specified as `-`, which would cause the program to read from stdin. `set` mode could still be used in this scenario in conjunction with some option to redirect the output to stdout, another file, etc. ## File The file would be a list of lines, each containing either a key/value pair or a comment. The key would be separated from the value with a `=` character, where everything before is part of the key and everything after is part of the value. ## Caveats Multiple pairs can be specified. I still haven't decided how these would be output in `get` mode. Perhaps, since the file is separated line by line, each value could be output on its own line. A malformed file might specify multiple values for the same key. Perhaps the program could just operate on the first one and ignore the rest. This *might* be a better task for the already planned/implemented field-oriented utilities (fop, fis, tex, fit, etc). If it is then tell me and I won't make the program.

sashakoshka added the

enhancement

label 2024-07-17 06:08:12 +00:00

sashakoshka commented

2024-07-17 06:18:17 +00:00

As for what this could look like in a script:

config=~/.config/greeter.conf
timeOfDay=`kv get "$config" timeOfDay=morning`
name=`kv get "$config" name="$USER"`

echo good $timeOfDay, $name.

As for what this could look like in a script: ```sh config=~/.config/greeter.conf timeOfDay=`kv get "$config" timeOfDay=morning` name=`kv get "$config" name="$USER"` echo good $timeOfDay, $name. ```

trinity commented

2024-07-17 17:37:31 +00:00

ASV (#19) might be better than =. Multiple fields would also be nice, but may be overcomplicating the matter.

ASV (#19) might be better than `=`. Multiple fields would also be nice, but may be overcomplicating the matter.

emma commented

2024-07-17 18:11:16 +00:00

I definitely think this is a little too high-level to implement here as its own tool. We should aim to support this via other tools in harakit, though.

sashakoshka commented

2024-07-17 18:36:24 +00:00

ASV (#19) might be better than =. Multiple fields would also be nice, but may be overcomplicating the matter.

The idea is that you can also edit the file by hand

> ASV (#19) might be better than `=`. Multiple fields would also be nice, but may be overcomplicating the matter. The idea is that you can also edit the file by hand

sashakoshka commented

2024-07-17 18:41:00 +00:00

I definitely think this is a little too high-level to implement here as its own tool. We should aim to support this via other tools in harakit, though.

Perhaps it could be a shell builtin (#8), something like source except for only reading variables?

> I definitely think this is a little too high-level to implement here as its own tool. We should aim to support this via other tools in harakit, though. Perhaps it could be a shell builtin (#8), something like source except for only reading variables?

trinity commented

2024-07-18 14:44:11 +00:00

I definitely think this is a little too high-level to implement here as its own tool. We should aim to support this via other tools in harakit, though.

Could you elaborate on this?

> I definitely think this is a little too high-level to implement here as its own tool. We should aim to support this via other tools in harakit, though. Could you elaborate on this?

emma commented

2024-08-04 03:40:47 +00:00

This would be more idiomatic:

$ config=~/.config/greeter.conf
$ mm -i "$config"
timeOfDay=morning
name="$USER"
$ mm -i "$config" | fit -d"\n" kv # sets timeOfDay and name
printf 'good %s, %s\n' "$timeOfDay", "$name".

fit(1)

This would be more idiomatic: ```sh $ config=~/.config/greeter.conf $ mm -i "$config" timeOfDay=morning name="$USER" $ mm -i "$config" | fit -d"\n" kv # sets timeOfDay and name printf 'good %s, %s\n' "$timeOfDay", "$name". ``` [fit(1)](https://git.tebibyte.media/bonsai/harakit/issues/68)

emma commented

2024-08-04 03:44:23 +00:00

it should be noted that -d"\n" does not work in POSIX shell as it passes a literal \n to the program. Here, it represents a literal newline.

it should be noted that `-d"\n"` does not work in POSIX shell as it passes a literal \n to the program. Here, it represents a literal newline.

emma commented

2024-08-04 03:45:32 +00:00

In qi(1), \n will expand to a literal newline.

In [`qi(1)`](https://git.tebibyte.media/bonsai/harakit/issues/8), \n will expand to a literal newline.

emma commented

2024-08-04 03:46:54 +00:00

ASV (#19) might be better than =. Multiple fields would also be nice, but may be overcomplicating the matter.

The idea is that you can also edit the file by hand

The assignment symbol makes sense for this, anyway, as that is exactly what it is doing.

> > ASV (#19) might be better than `=`. Multiple fields would also be nice, but may be overcomplicating the matter. > > The idea is that you can also edit the file by hand The assignment symbol makes sense for this, anyway, as that is exactly what it is doing.

👍 1

emma commented

2024-08-04 03:48:45 +00:00

Setting values in the file could be achieved through manual editing and in scripts by existing (or not yet existing) tools.

sashakoshka commented

2024-08-04 04:10:04 +00:00

This would be more idiomatic:

$ config=~/.config/greeter.conf
$ mm -i "$config"
timeOfDay=morning
name="$USER"
$ mm -i "$config" | fit -d"\n" kv # sets timeOfDay and name
printf 'good %s, %s\n' "$timeOfDay", "$name".

Trying to understand the role of kv in this context. Is it a shell builtin that sets a variable based on a key=value string? Because if so, it may have undesirable effects (setting an internal variable or overriding env variable) unless all variables set by it are prefixed in some way. Also, I don't see how fit(1) could be capable of running a shell builtin within the current shell as if it were a command.

> This would be more idiomatic: > ```sh > $ config=~/.config/greeter.conf > $ mm -i "$config" > timeOfDay=morning > name="$USER" > $ mm -i "$config" | fit -d"\n" kv # sets timeOfDay and name > printf 'good %s, %s\n' "$timeOfDay", "$name". > ``` Trying to understand the role of `kv` in this context. Is it a shell builtin that sets a variable based on a `key=value` string? Because if so, it may have undesirable effects (setting an internal variable or overriding env variable) unless all variables set by it are prefixed in some way. Also, I don't see how fit(1) could be capable of running a shell builtin within the current shell as if it were a command.

sashakoshka commented

2024-08-04 04:15:13 +00:00

Setting values in the file could be achieved through manual editing and in scripts by existing (or not yet existing) tools.

In my opinion this kind of defeats the purpose of the tool, the idea was to allow scripts to read and write to flat files to store and recall arbitrary information. In order to be useful I think it would have to at least be capable of formatting an entry given a key and a value.

> Setting values in the file could be achieved through manual editing and in scripts by existing (or not yet existing) tools. In my opinion this kind of defeats the purpose of the tool, the idea was to allow scripts to read and write to flat files to store and recall arbitrary information. In order to be useful I think it would have to at least be capable of formatting an entry given a key and a value.

emma commented

2024-08-04 04:51:52 +00:00

This would be more idiomatic:
$ config=~/.config/greeter.conf
$ mm -i "$config"
timeOfDay=morning
name="$USER"
$ mm -i "$config" | fit -d"\n" kv # sets timeOfDay and name
printf 'good %s, %s\n' "$timeOfDay", "$name".
Trying to understand the role of kv in this context. Is it a shell builtin that sets a variable based on a key=value string? Because if so, it may have undesirable effects (setting an internal variable or overriding env variable) unless all variables set by it are prefixed in some way. Also, I don't see how fit(1) could be capable of running a shell builtin within the current shell as if it were a command.

The qi(1) shell is going to have as few builtins as possible, therefore this probably wouldnt be implemented as one. The kv(1) invocation here would simply export variables into the environment, which is what I thought its purpose was from your description.

> > This would be more idiomatic: > > ```sh > > $ config=~/.config/greeter.conf > > $ mm -i "$config" > > timeOfDay=morning > > name="$USER" > > $ mm -i "$config" | fit -d"\n" kv # sets timeOfDay and name > > printf 'good %s, %s\n' "$timeOfDay", "$name". > > ``` > > Trying to understand the role of `kv` in this context. Is it a shell builtin that sets a variable based on a `key=value` string? Because if so, it may have undesirable effects (setting an internal variable or overriding env variable) unless all variables set by it are prefixed in some way. Also, I don't see how fit(1) could be capable of running a shell builtin within the current shell as if it were a command. The `qi(1)` shell is going to have as few builtins as possible, therefore this probably wouldnt be implemented as one. The `kv(1)` invocation here would simply export variables into the environment, which is what I thought its purpose was from your description.

emma commented

2024-08-04 04:59:18 +00:00

I see where I misunderstood. The kv(1) command in your original proposal returns the variable value from the store. Altogether, though, I’m starting to wonder what the benefit of this is over having environment variables set in the shell environment.

I see where I misunderstood. The `kv(1)` command in your original proposal *returns* the variable value from the store. Altogether, though, I’m starting to wonder what the benefit of this is over having environment variables set in the shell environment.

sashakoshka commented

2024-08-04 06:30:46 +00:00

I’m starting to wonder what the benefit of this is over having environment variables set in the shell environment.

Configuring things via environment variables is generally very clumsy. Changing a configuration value requires setting the variable(s) in your profile, and logging out and back in again. If you want to access these new values without doing that, you need to manually export the variables in each new shell and/or specify them before running the program which reads them. It's not as simple as changing a line in a file and re-running the program, or changing a value using some CLI/GUI tool. The way environment variables are stored, loaded, and reset needs to be rethought somewhat if they are to be used as a configuration system. Additionally, environment variables are only good for configuration parameters and can't be used as a general data storage/retrieval mechanism.

> I’m starting to wonder what the benefit of this is over having environment variables set in the shell environment. Configuring things via environment variables is generally very clumsy. Changing a configuration value requires setting the variable(s) in your profile, and logging out and back in again. If you want to access these new values without doing that, you need to manually export the variables in each new shell and/or specify them before running the program which reads them. It's not as simple as changing a line in a file and re-running the program, or changing a value using some CLI/GUI tool. The way environment variables are stored, loaded, and reset needs to be rethought somewhat if they are to be used as a configuration system. Additionally, environment variables are only good for configuration parameters and can't be used as a general data storage/retrieval mechanism.

emma commented

2024-08-04 15:30:34 +00:00

Configuring things via environment variables is generally very clumsy. Changing a configuration value requires setting the variable(s) in your profile, and logging out and back in again

This is a misuse of the .profile file as it is for login shells only. For configuration data which can be read by non-login shells, .env is the preferred location. This still does require a shell reload but, to quote @trinity, we're working on an ecosystem so we can actually solve these issues (for ourselves). For example, we can make the shell watch the .env-equivalent and reload for any changes.

The way environment variables are stored, loaded, and reset needs to be rethought somewhat if they are to be used as a configuration system. Additionally, environment variables are only good for configuration parameters and can't be used as a general data storage/retrieval mechanism.

Please elaborate.

> Configuring things via environment variables is generally very clumsy. Changing a configuration value requires setting the variable(s) in your profile, and logging out and back in again This is a misuse of the `.profile` file as it is for login shells only. For configuration data which can be read by non-login shells, `.env` is the preferred location. This still does require a shell reload but, to quote @trinity, [we're working on an ecosystem so we can actually solve these issues (for ourselves).](https://git.tebibyte.media/bonsai/harakit/issues/19#issuecomment-3257) For example, we can make the shell watch the `.env`-equivalent and reload for any changes. > The way environment variables are stored, loaded, and reset needs to be rethought somewhat if they are to be used as a configuration system. Additionally, environment variables are only good for configuration parameters and can't be used as a general data storage/retrieval mechanism. Please elaborate.

sashakoshka commented

2024-08-04 23:29:34 +00:00

Please elaborate

Well, you half elaborated for me. Watching .env as you described is the kind of "rethinking" I was talking about. As for the general data storage/retrieval mechanism, I was sort of envisioning it, along with directory hierarchies, being used to store and manipulate structured data in a way that can't really be expressed with env variables. I'll list some examples here:

Desktop icons/launchers
Init services
Virtual web hosts/reverse proxying (i.e. nginx and tor)

Of course, these all imply complicated software, and the main consumer of such files would likely not be a shell script. However, ideally, kv(1) and the programs would speak the same language here, and it would be possible to write scripts that manage and interact with these files.

Maybe this would be more useful as some sort of library that kv(1) is a shell interface to.

> Please elaborate Well, you half elaborated for me. Watching `.env` as you described is the kind of "rethinking" I was talking about. As for the general data storage/retrieval mechanism, I was sort of envisioning it, along with directory hierarchies, being used to store and manipulate structured data in a way that can't really be expressed with env variables. I'll list some examples here: - Desktop icons/launchers - Init services - Virtual web hosts/reverse proxying (i.e. nginx and tor) Of course, these all imply complicated software, and the main consumer of such files would likely not be a shell script. However, ideally, kv(1) and the programs would speak the same language here, and it would be possible to write scripts that manage and interact with these files. Maybe this would be more useful as some sort of library that kv(1) is a shell interface to.

emma commented

2024-08-05 04:52:58 +00:00

Please elaborate

Well, you half elaborated for me. Watching .env as you described is the kind of "rethinking" I was talking about. As for the general data storage/retrieval mechanism, I was sort of envisioning it, along with directory hierarchies, being used to store and manipulate structured data in a way that can't really be expressed with env variables. I'll list some examples here:

Desktop icons/launchers

Init services

Virtual web hosts/reverse proxying (i.e. nginx and tor)

Of course, these all imply complicated software, and the main consumer of such files would likely not be a shell script. However, ideally, kv(1) and the programs would speak the same language here, and it would be possible to write scripts that manage and interact with these files.

Maybe this would be more useful as some sort of library that kv(1) is a shell interface to.

This seems more like something you’d build using the harakit and not something to be included in it.

> > Please elaborate > > Well, you half elaborated for me. Watching `.env` as you described is the kind of "rethinking" I was talking about. As for the general data storage/retrieval mechanism, I was sort of envisioning it, along with directory hierarchies, being used to store and manipulate structured data in a way that can't really be expressed with env variables. I'll list some examples here: > > - Desktop icons/launchers > - Init services > - Virtual web hosts/reverse proxying (i.e. nginx and tor) > > Of course, these all imply complicated software, and the main consumer of such files would likely not be a shell script. However, ideally, kv(1) and the programs would speak the same language here, and it would be possible to write scripts that manage and interact with these files. > > Maybe this would be more useful as some sort of library that kv(1) is a shell interface to. This seems more like something you’d build *using* the harakit and not something to be included in it.

trinity commented

2024-08-05 21:15:48 +00:00

It would be better to read from standard input only and use file redirection.

Sign in to join this conversation.