Load user datas

User datas are values setup by user for configuration variables.

There is differents types of user datas for differents sources types.

We can cumulate user datas loader.

For this section, we will use dict/00-base.yml a structure file:

%YAML 1.2
---
version: 1.1

my_variable: my value   # My first variable

my_boolean_variable: true  # My boolean variable

my_integer_variable: 1  # My integer variable

my_secret_variable:
  description: My secret variable
  type: secret

my_hidden_variable:
  description: My hidden variable
  hidden: true
...

Here is the first script which is load this file:

the script.py file content

from rougail import Rougail, RougailConfig

RougailConfig["main_namespace"] = None
RougailConfig["main_structural_directories"] = ["dist/"]
rougail = Rougail()
config = rougail.run()
print(config.value.get())

Let’s execute script.py:

$ python3 script.py
{<TiramisuOption path="my_variable">: 'my value', <TiramisuOption path="my_boolean_variable">: True, <TiramisuOption path="my_integer_variable">: 1, <TiramisuOption path="my_secret_variable">: None}

YAML

We want to load this YAML file with value define by user:

---
my_variable: a new value

my_boolean_variable: false

my_integer_variable: 10

my_secret_variable: MyVeryStrongPassword

Here is the script which is load user data from the YAML file:

the script.py file content

from rougail import Rougail, RougailConfig
from rougail.user_data_yaml import RougailUserDataYaml

RougailConfig["main_namespace"] = None
RougailConfig["main_structural_directories"] = ["dist/"]
RougailConfig["step.user_data"] = ["yaml"]
RougailConfig["yaml.filename"] = ["dist.yml"]
rougail = Rougail()
config = rougail.run()

user_datas = RougailUserDataYaml(config).run()
rougail.user_datas(user_datas)
print(config.value.get())

Let’s execute script.py:

$ python3 script.py
{<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: 'MyVeryStrongPassword'}

Set a secret in clear text file is not always a good idea. This is why the yaml.file_with_secrets parameter allows you to define whether files define in yaml.filename can contain a secret and which one:

• all: all file can contains secret

• first: only the first file can contains secret

• last: only the last file can contains secret

• none: no file can contains secret

the script.py file content

from rougail import Rougail, RougailConfig
from rougail.user_data_yaml import RougailUserDataYaml

RougailConfig["main_namespace"] = None
RougailConfig["main_structural_directories"] = ["dist/"]
RougailConfig["step.user_data"] = ["yaml"]
RougailConfig["yaml.filename"] = ["dist.yml"]
RougailConfig["yaml.file_with_secrets"] = "none"
rougail = Rougail()
config = rougail.run()

user_datas = RougailUserDataYaml(
     config,
).run()
rougail.user_datas(user_datas)
print(config.value.get())

Let’s execute script.py:

$ python3 script.py
{<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: None}

Environment variables

We can define use data from environment variables. The environment name is a “prefix” (ROUGAIL by default) with “_” and variable name in uppercase format.

For example:

• my_variable has ROUGAIL_MY_VARIABLE as a environment variable name

• my_family.my_variable has ROUGAIL_MY_FAMILY.MY_VARIABLE as a environment variable name

Some shell doesn’t allow dot in environment file. In this case use the command “env”.

For example: env ROUGAIL_MY_FAMILY.MY_VARIABLE="value" ./script.py.

Here is the script:

the script.py file content

from rougail import Rougail, RougailConfig
from rougail.user_data_environment import RougailUserDataEnvironment

RougailConfig["main_namespace"] = None
RougailConfig["main_structural_directories"] = ["dist/"]
RougailConfig["step.user_data"] = ["environment"]
rougail = Rougail()
config = rougail.run()

user_datas = RougailUserDataEnvironment(config).run()
rougail.user_datas(user_datas)
print(config.value.get())

Let’s execute script.py:

env ROUGAIL_MY_VARIABLE="a new value" ROUGAIL_MY_BOOLEAN_VARIABLE="False" ROUGAIL_MY_INTEGER_VARIABLE=10 ROUGAIL_MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py
{<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: 'MyVeryStrongPassword'}

We can redefine the prefix with environment.default_environment_name (prefix is always uppercase characters):

the script.py file content

from rougail import Rougail, RougailConfig
from rougail.user_data_environment import RougailUserDataEnvironment

RougailConfig["main_namespace"] = None
RougailConfig["main_structural_directories"] = ["dist/"]
RougailConfig["step.user_data"] = ["environment"]
RougailConfig["environment.default_environment_name"] = "EX"
rougail = Rougail()
config = rougail.run()

user_datas = RougailUserDataEnvironment(config).run()
rougail.user_datas(user_datas)
print(config.value.get())

Let’s execute script.py:

env EX_MY_VARIABLE="a new value" EX_MY_BOOLEAN_VARIABLE="False" EX_MY_INTEGER_VARIABLE=10 EX_MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py
{<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: 'MyVeryStrongPassword'}

If you define a main_namespace or extra_namespaces, the environment.default_environment_name is automaticly define with the name of the namespace in uppercase. And the separator is no more “_” but “.”:

the script.py file content

from rougail import Rougail, RougailConfig
from rougail.user_data_environment import RougailUserDataEnvironment

RougailConfig["main_namespace"] = "main"
RougailConfig["main_structural_directories"] = ["dist/"]
RougailConfig["step.user_data"] = ["environment"]
rougail = Rougail()
config = rougail.run()

user_datas = RougailUserDataEnvironment(config).run()
rougail.user_datas(user_datas)
print(config.value.get())

Let’s execute script.py:

env MAIN.MY_VARIABLE="a new value" MAIN.MY_BOOLEAN_VARIABLE="False" MAIN.MY_INTEGER_VARIABLE=10 MAIN.MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py
{<TiramisuOption path="main">: {<TiramisuOption path="main.my_variable">: 'a new value', <TiramisuOption path="main.my_boolean_variable">: False, <TiramisuOption path="main.my_integer_variable">: 10, <TiramisuOption path="main.my_secret_variable">: 'MyVeryStrongPassword'}}

Set a secret in clear variable environment is not always a good idea. This is why the environment.with_secrets parameter allows you to reject secret from environment variable:

the script.py file content

from rougail import Rougail, RougailConfig
from rougail.user_data_environment import RougailUserDataEnvironment

RougailConfig["main_namespace"] = None
RougailConfig["main_structural_directories"] = ["dist/"]
RougailConfig["step.user_data"] = ["environment"]
RougailConfig["environment.with_secrets"] = False
rougail = Rougail()
config = rougail.run()

user_datas = RougailUserDataEnvironment(config).run()
rougail.user_datas(user_datas)
print(config.value.get())

Let’s execute script.py:

env ROUGAIL_MY_VARIABLE="a new value" ROUGAIL_MY_BOOLEAN_VARIABLE="False" ROUGAIL_MY_INTEGER_VARIABLE=10 ROUGAIL_MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py
{<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: None}

Comand line parser user data

Value can be define directly with command line arguments:

the script.py file content

from rougail import Rougail, RougailConfig
from rougail.user_data_commandline import RougailUserDataCommandline

RougailConfig["main_namespace"] = None
RougailConfig["main_structural_directories"] = ["dist/"]
RougailConfig["step.user_data"] = ["commandline"]
rougail = Rougail()
config = rougail.run()

user_datas = RougailUserDataCommandline(
     config,
).run()
rougail.user_datas(user_datas)
print(config.value.get())

Let’s execute script.py to display help:

$ python script.py -h
usage: script.py [-h] --my_variable [MY_VARIABLE] --my_boolean_variable --no-my_boolean_variable --my_integer_variable [MY_INTEGER_VARIABLE] --my_secret_variable MY_SECRET_VARIABLE

options:
  -h, --help            show this help message and exit
  --my_variable [MY_VARIABLE]
                        my_variable (My first variable) (default: my value)
  --my_boolean_variable
                        my_boolean_variable (My boolean variable) (default: True)
  --no-my_boolean_variable
  --my_integer_variable [MY_INTEGER_VARIABLE]
                        my_integer_variable (My integer variable) (default: 1)
  --my_secret_variable MY_SECRET_VARIABLE
                        my_secret_variable (My secret variable)
{<TiramisuOption path="my_variable">: 'my value', <TiramisuOption path="my_boolean_variable">: True, <TiramisuOption path="my_integer_variable">: 1, <TiramisuOption path="my_secret_variable">: None}

And now with modified value:

$ python script.py --my_variable "a new value" --no-my_boolean_variable --my_integer_variable 10 --my_secret_variable MyVeryStrongPassword
{<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: 'MyVeryStrongPassword'}

Boolean variable has a special behavour. To set False you need to add –no-VARIABLE, to set True you need to add –VARIABLE parameter.

Combine user datas

You can combine user datas, for example if you want to load datas from environment and/or command line argument:

the script.py file content

from rougail import Rougail, RougailConfig
from rougail.user_data_environment import RougailUserDataEnvironment
from rougail.user_data_commandline import RougailUserDataCommandline

RougailConfig["main_namespace"] = None
RougailConfig["main_structural_directories"] = ["dist/"]
RougailConfig["step.user_data"] = ["environment", "commandline"]
rougail = Rougail()
config = rougail.run()

user_datas = []
user_datas.extend(RougailUserDataEnvironment(
     config,
).run())
user_datas.extend(RougailUserDataCommandline(
     config,
).run())
rougail.user_datas(user_datas)
print(config.value.get())

Let’s execute script.py with environment variable and commandline arguments:

$ env ROUGAIL_MY_VARIABLE="a new value" ROUGAIL_MY_BOOLEAN_VARIABLE="False" python script.py --my_integer_variable 10 --my_secret_variable MyVeryStrongPassword
{<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: 'MyVeryStrongPassword'}

If the value of a variable is define with an environment variable and commandline argument, the value is the value of the last user data define:

 $ env ROUGAIL_MY_VARIABLE="not a new" python script.py --my_variable "a new value" --no-my_boolean_variable --my_integer_variable 10 --my_secret_variable MyVeryStrongPassword
{<TiramisuOption path="my_variable">: 'a new value', <TiramisuOption path="my_boolean_variable">: False, <TiramisuOption path="my_integer_variable">: 10, <TiramisuOption path="my_secret_variable">: 'MyVeryStrongPassword'}

Manage errors and warnings

Recreate a script with environnement variable support which is display the return of user_datas function:

the script.py file content

from rougail import Rougail, RougailConfig
from rougail.user_data_environment import RougailUserDataEnvironment

RougailConfig["main_namespace"] = None
RougailConfig["main_structural_directories"] = ["dist/"]
RougailConfig["step.user_data"] = ["environment"]
RougailConfig["environment.with_secrets"] = False
rougail = Rougail()
config = rougail.run()

user_datas = RougailUserDataEnvironment(
     config,
).run()
print(rougail.user_datas(user_datas))

Try to load the value an unknown variable:

$ env ROUGAIL_UNKNOWN_VARIABLE="a value" python script.py
{'errors': [], 'warnings': ['variable or family "unknown_variable" does not exist, it will be ignored when loading from environment variable']}

As you can see, a warnings is return.

Try to load the value of an hidden variable:

$ env ROUGAIL_MY_HIDDEN_VARIABLE="a value" python script.py
 {'errors': [], 'warnings': ['variable "my_hidden_variable" (My hidden variable) is hidden, it will be ignored when loading from environment variable']}

Finally if a try to change the value of a secret, which is not allowed:

$ env ROUGAIL_MY_SECRET_VARIABLE="MyVeryStrongPassword" python script.py
{'errors': ['the variable "my_secret_variable" contains secrets and should not be defined in environment variable'], 'warnings': []}

An error is generated.