How to use pydantic to read environment variables and secret files in Python

Learn to manage your environment variables in a better way

In our Python code, we often need to specify some sensitive information such as database username, password, API keys, JWT tokens, etc. We should not store any sensitive data as plain texts in our source code repository because they can get leaked easily. A common practice is to store the credentials as environmental variables or secret files on the machine on which the application is running. The machine is a generic concept and can be a bare-metal machine, a virtual machine, a docker container, a Cloud Run service, a Cloud Function, etc.

Photo by Kristina Flour on Unsplash.

For simplicity, suppose that we have set up two environment variables for database username and password.

$ export DB_USERNAME=some_username
$ export DB_PASSWORD=some_password

To read environment variables, a common way is to use the os module:

If the environment variable does not exist, using os.environ will raise KeyError. In this case, we can use os.environ.get to avoid KeyError. We can also pass a default value if the environment variable does not exist.

This classical way of dealing with environment variables is convenient when we just have one or two environment variables to deal with. However, it can be cumbersome and there would be a lot of duplicate code if we have more than a few environment variables. Besides, sometimes we will also have secret files for our code. In this example, the username can be set as an environment variable. However, the password should preferably be put in a secrete file, not as an environment variable. This is because the environment variable can be easily found by the env…