Dalia's Blog

If you're anything like me, environments are just one of those weird quirks of computer systems that you've ignored for a long time. Sometimes you need to put something in there, sometimes you need to retrieve a value. When they work they're useful, if a bit weird. When they don't work (i.e. when the values you need are just mysteriously missing), they're a downright pain to use.

If that describes you, this article is going to blow your mind. Environments are a far more useful concept than you might have realised, and they come up all the time. This article is going to teach you about a concept that touches every aspect of computing.

What is an Environment?

Alright so that might have sounded like a bit of hyperbole. I need to back up those claims a bit. Before I go into more depth about why this concept is so important, I'm going to give you a really quick and basic overview of what an environment is.

At a really high level, an environment is a collection of key-value pairs that gets loaded into memory when you enter a particular scope, and gets removed when you leave that scope. This applies to shells, running processes, and programming languages. That's it. That's the whole concept.

Shells

Now, what does this really look like in practice? Chances are, you've only come across the term 'environment' in the context of a shell, or of setting external variables for a program, so let's start there. If you've worked with a UNIX shell (and I believe PowerShell and CMD on Windows work the same way, although maybe with different syntax), you have probably set variables, such as my_var="random string". You can then access the value with $my_var.

What's happening when you define this? The environment is a data structure, loaded into memory. Exactly what that data structure is depends on the program in question. For simplicity's sake, we can assume for the time being, that it consists of a key and a value. So when you define a variable, such as my_var="random string", the data structure gets a new entry, with the key my_var, and the value "random string". When you reference my_var to access the value, it looks up that key in the data structure, and resolves to a value if it exists, or nothing if it doesn't (at least Bash, which controversially made this design choice. It's up to you to check whether the value is what you expect or not, although there are ways to change this behaviour).

This should already explain a lot of shell behaviour to you. This is why an undefined variable in the shell doesn't resolve to a value. It also means that when you unset a variable, you are deleting it from the current environment data structure. It means that when you run source with a script, aside from running any extra commands in that script, you are primarily adding a bunch of variables to the current environment data structure.

Scope plays a big role in shells too. When you run a program with a variable assignment prefixed to it, as in my_var=1 ./my-program, you are setting $my_var to that particular value only within the scope of that program. As soon as your program finishes, or when you run commands in another tab or window (any other shell process in other words), or if you use CTRL-Z to put your program in the background, my_var will not have the value you gave it in that line. That value is set in the environment structure that your program sees, but all other programs see the original value.

When you export a variable, you are setting it for every program that runs within the current shell process. When you add the variable to a .bashrc, .zshrc, .profile or other local configuration file that gets run by the shell on startup, you are ensuring that the variable will be set to that value in every shell session you run. The variable will be created in the environment data structure that is visible to all the programs you run in your shell.

The concept of scope is what explains the values accessed by your programs. If you set a variable in your configuration files, but then give it a different value when you run one single command, the program you run will first check its local environment data structure. When it finds that value, it will stop looking and use that one, completely unaware that a different value exists in the outer scope. If it doesn't find a variable in the local scope, it will check the outer one, and keep checking until it finds it. This is how you shadow an outer environment variable to give it a different value.

Program Config Files

The same principle applies, fairly obviously, to program configuration files. You can see examples of these in any widely used enterprise language. You find them in the Spring Boot property files, in Python and JavaScript config files, in Docker files that set environment variables.

They often follow the same principle (although this is implementation dependent, and actual behaviour may vary). A general configuration file will be overridden by a file specific to the current "environment", i.e. the collection of servers associated with dev, testing, or prod. A specific config file will be overridden in turn by locally set environment variables. The term environment creates a little bit of ambiguity here, but it refers in the first case, to the infrastructure the program is running on, and it is distinguished by the environment values which are set in it, which go into the environment data structure which is in scope. (Some languages and tools behave differently, but if you understand this principle, it will at help you see connections between these concepts. Just remember to check the behaviour of your tools so you know what will be in scope.)

Applying This To Programming Languages

So far this has been a (hopefully) useful overview of how shell environments work, but it hasn't offered mind-blowing insights into how programming works. That's up next.

If you've studied any programming languages in much depth, the discussion of scope above probably sounded familiar, and for good reason. The concept of scope in a programming language is the same concept. I owe much of my understanding of how scope is implemented in a programming language to this chapter of Crafting Interpreters. It's a great book for understanding these ideas more deeply. I recommend it if you want to learn more.

A programming language implements multiple scopes, all the way from global, down to the local block of code being executed. And these scopes contain environments, which are data structures that contain keys and values. This is how variables are implemented. It's also how function and class definitions will be stored in an interpreted language, in an abstract syntax tree, so that the program knows where to go when a particular function name is invoked.

When a name is used in a language, the runtime or the compiler searches for it in the local scope. If it's there, it resolves to its value. If it's not, it searches the next scope outwards, going all the way to global scope if necessary, and throwing an error if the name is not found.

That's why you are able to shadow a variable name in an inner scope, by giving it a different definition from the outer variable. It's also why your IDE probably warns you when you do this, since it might not be the behaviour you want, and forces you to check every reference and definition in the scope to work out which value it will get. This is particularly problematic in dynamically typed languages, where there is no visual distinction between initialisation and reassignment. my_var = 1 can be giving a new value to an existing variable, or it can be creating an entirely new one, and you don't know which is which.

This is also why outer scopes are unable to access the values of variables that are only defined within inner scopes. Once you leave that inner scope, the environment associated with it gets destroyed, and is no longer visible. If the variable doesn't exist in the outer scope, you will get an error when you try to access it, as in the following C code:

#include <stdio.h>

int main() {


  {
    int num = 1;
    printf("Num inside block: %d\n", num);
  }

  printf("Num outside block: %d\n", num);

  return 0;
}
    

This will produce an error if you try to compile it, as num is not defined outside of that scope.

It's important to note, however, that if the variable does exist in the outer scope, then assigning a new value to it in an inner scope will often overwrite the outer value. This makes sense if you remember how scope works. When you define and assign a variable in an inner scope, the program will first check if that variable exists in an outer environment. If it does, then it will add your value to that key, overwriting the old one. If it doesn't, then it will define a new variable in a dynamically typed language, or throw an error in a statically typed one, as you are attempting to assign a value without declaring the variable.

Another point that programming languages bring up, especially a statically typed one like C, is that the data structure for an environment is not just going to be a key and a value. It will often also contain the type information, which will tell the runtime how to handle this variable, whether it's an int, a double, or a pointer to an array. Depending on language implementation, it could be some other sort of data structure. It most likely won't just be a key and a value, even in a shell. As long as you understand that it's a structure that stores the names and the values, or the addresses where the values are stored, the concept of an environment should be useful to you.

For some more on how scope is handled in a fairly representative language, this is a great source (sorry in advance for making you look at C++).

References

You now understand enough to make sense of another quirk of lower level programming languages that can cause considerable confusion for learners, namely automatic variables. This term refers to variables which are initialised within a function. In a language like C, variables inside a function go out of scope as soon as the function returns, and their values are destroyed (this actually applies to high level languages too, we just don't notice, because we don't have to manipulate pointers). When you attempt to return a pointer to an automatic variable, the pointer will refer to a place in memory that has been freed (because it's in the stack, not the heap). That's why you have to return pointers instead. You can return values from functions, but not the variables themselves.

This can lead to a variety of issues. My understanding is that this is one of the issues the borrow checker in Rust was invented to solve (among a number of others referring to memory and pointers, which are just generally a headache for humans to keep track of).

Again, the mental model of a data structure as an environment can help you make sense of the issue. The variable inside your function was defined in the local environment, along with its value. You then returned the memory address where that value was stored, but the memory got cleaned up, as soon as the function exited, along with the variable itself. The address now points to memory which does not belong to the process, and the kernel rejects the request.

This also illustrates the point that the environment data structure of course exists in memory, like everything else on the computer, which is why pointers can be passed around to them. But that discussion centres more around pointers than environments.

Security Challenges

Here's another issue that arises due to the nature of environments: they can hold sensitive values which are sometimes visible to people who shouldn't see them. Environments are visible to any process within their scope. In the case of a shell environment, they are typically visible to a whole range of processes. Anyone who gets access to that shell can read those values. This can include the values of passwords and other credentials.

This is still less of a concern than, say, writing out a credential in a plain text config file and committing it to source control. But it is still an issue that needs to be considered, depending on your threat model.

To mitigate this issue, we can take advantage of the nature of environments to reduce the scope of visibility of those variables. That's where defining a variable on the same line that the program is run on comes in handy. A line like:
SECRET=$(get-secret-from-remote-vault) ./my-program will ensure that the result of a command to get a secret is only visible to the process using it, rather than the entire shell. And once that process finishes, that variable will go out of scope and lose its value. You would probably still want to understand, though, what happens to the memory that that variable was stored in to ensure that it can't be read, showing, again, why understanding this mental model is important for ensuring that you detect potential vulnerabilities. Other methods of ensuring proper credential security on top of this are usually required. If you are in a position to make security decisions, be sure to research this topic thoroughly.

Coding Is All About Environments

Now we finally get to the insight which I find the most transformative: computing is all about manipulating environments. A necessary disclaimer before I go on to explain this: this is only one way to frame computing, and it doesn't consider side-effects, which are mostly related to I/O. Obviously those I/O effects are also essential for actually showing results to a user.

Nevertheless, in the logic portion of a program, most of what you are doing is changing the values in environments. When you define a variable, you add a value to an environment. When you run a loop, you repeatedly reassign certain values to the variables in the block environment. When you call a function, you add values to the parameters in that function's environment, and when you return from the function, some of the variables are deleted, while return values are added to the outer scope. (For more about how functions work, see my article on the topic.) You manipulate the data structures which live in variables inside of the environment. Sometimes you take advantage of the persistent nature of an environment to allow a variable to be manipulated or visible within more than one scope. When you think about it, it's all manipulating environments.

A big part of the complexity I touched on near the end of this essay surrounds the way functions are handled in code. My next essay is going to be about understanding functions. Keep an eye out for that one.

Obviously there's still an awful lot of other things to understand about computing, but I think these insights about environments can really tie a lot of concepts together in a way that makes it more memorable. I hope it does that for you too!