Dalia's Blog

Promises are notoriously difficult to understand. They are used everywhere in asynchronous code in JavaScript, but not often well-explained. Many developers just learn to write a .then() call and maybe a .catch() if they're feeling especially diligent.

If you've ever struggled to understand promises, or how to use them, my hope is that this article will change that for you. I have struggled with callbacks and promises a lot myself, but I recently had some breakthroughs that allowed me to understand how to create my own promises and work effectively with asynchronous code in JavaScript. I'll explain those breakthroughs, and show some practical, easy examples to follow.

Asynchronous Code

What is Asynchronous Code?

Asynchronous means happening out of synchronisation. In other words, one thing can happen at a different time to another thing. Synchronous code means everything happens in a specific order, processes generally wait for each other process to complete in sequence, only one thing can happen at a time per thread or process.

Asynchronous code allows you to continue working while something else happens in the background. One of the reasons this concept trips new developers up, is that we don't perceive what is happening in the background. When you write fs.readFile(filename) you think of this as a short, single operation, that is completed before the next line of your program is executed.

If you were writing in Java, Python, or C, that assumption might be correct. But in JavaScript, that is basically never the case. Reading a file is actually a complex operation, and in some cases it can take quite a bit of time, depending on where the file is stored. In languages like Java, you can perform other operations while this is being done, using different forms of asynchronous or concurrent programming. But in JavaScript it is essential.

To understand why, we need to consider what JavaScript is typically used for -- graphical interfaces that users interact with. Displaying the interface and allowing users to interact with it is one of JavaScript's main jobs. If it devotes all its energy to opening and reading a file, the interface will freeze until this is complete, which leads to a very bad user experience. Even the act of displaying a spinning icon requires the JavaScript runtime to dedicate some of its energy, so it can't be focusing on opening the file.

This also applies on the backend. Node is a JavaScript runtime which supports the same style of asynchronous programming as the browser. Asynchronous programming is especially important when handling user requests on a server. If the server freezes every time a user request comes in, while it updates a server or fetches an HTML page to return, any other user requests which come in in the meantime will sit waiting.

This might not sound like a big deal when you are writing practice servers on your own computer, but when you are supporting a production server which handles thousands of requests per second, a one second pause to fetch a file will lead to disaster. (It should be noted that it is possible to use synchronous APIs in many JavaScript libraries, which will pause everything and wait, but it's usually not a good idea, for the reasons mentioned above. If you find yourself in the rare situation where it's needed, you'll probably know.)

Languages like Java and Python can manage multiple requests at the same time using threads, and sometimes they will even provide interfaces for asynchronous programming in JavaScript style. But in JavaScript, asynchronous programming is the default, as a user interface can never pause to fetch a file or make a network request.

In JavaScript, this isn't primarily achieved using threads. Threads entail separate lines of execution, which work without talking to each other (communication is possible, but not default). In JavaScript, the main thread of execution usually does need to be interrupted once the result of reading a file or making a request is ready, but it needs to avoid waiting for that result. For that reason, event-based, non-blocking, asynchronous programming is the default in JavaScript.

Event-based Asynchronous Programming

That was a lot of words to describe this style of programming. What do they all mean?

The JavaScript runtime does not just handle asynchronous programming using threads. Event-based programming entails a different style of execution entirely.

In order to ensure that the main thread of execution can return to a task as soon as the required data is ready, the runtime runs an event loop in the background. There is an infinite while loop running, which checks, for every cycle of its iteration, if there are any pending tasks. Events are used to queue up new tasks as soon as they are ready, and ensure that they get the attention of the event loop.

So what's an event? It's essentially just a notification. It might be a message dropped into a queue, or an item appended to an array. A task such as reading a file will emit an event. Once it's complete, the event goes into the queue. The runtime sees the new event and performs the action associated with the event. (That explains asynchronous and event-based. We'll get back to non-blocking later.)

Now this is where promises and callbacks come in. The action associated with an event is a callback. Callback-based programming involves triggering a process which emits an event, and passing in a callback, which is a type of function reference.

Function References

What's a function reference? It's a way of invoking a function via a variable. This is a concept that exists in Python, and barely exists in modern Java, but really makes more sense if you come from a background in Lisp. (You can also create pointers to functions in C and Assembly, it just hurts your brain a lot more to try.) If you don't know Lisp, that's ok. Think of a function reference as a way of asking a function to call another function.

Here's a really simple example. (As with many examples in technical explanations, it will be very trivial, but hopefully also very easy to follow.)

Let's suppose we define a function which can modify a number, but we don't specify how it will modify it, only that it can accept different ways of modifying it:

function change_number(x, fn) {
  return fn(x);
}
  

Notice that fn is a parameter. It's acting just like the other parameters, as a piece of data. If we wrote this in TypeScript, with type annotations, the signature would look like this:

function change_number(x: number, fn: function): number {
  return fn(x);
}
  

It's a piece of data, but with the type of function. And then it gets invoked, just like any other function. If you weren't aware that this was a legitimate style of programming, I encourage you to open the JavaScript console by right clicking your screen and selecting Inspect, then go to the Console section and start typing in some JavaScript. This is an important concept to understand. Without it, callbacks and event-based programming will never make sense.

So, a callback is a function reference, that we pass to the event loop to be triggered once an event signals that the task is complete. That's how event-based programming works in a nutshell, but this might still all seem a bit vague and hard to understand. Let's explore an API that uses callbacks to get a better idea of what they look like.

Callbacks

For this section, we're going to turn to the Node library for reading files. This is a very common context where asynchronous programming comes up, because reading a file takes time. We need to be able to ask the runtime to delegate reading a file for us, and let us know once it's been read so we can do something with it. Here's an example directly from Node's documentation:

import fs from 'node:fs';

fs.readFile('/Users/jill/test.txt', 'utf8', (err, data) => {
  if (err) {
    console.error(err);
    return;
  }
  console.log(data);
});
  

What's going on here? First, we import the file system library (that's what fs stands for) from Node. Then we ask it to read a file. The readFile method takes 3 parameters. The first one is the name of the file. The second one is the character encoding. These are both straightforward. The third parameter is the one that needs our attention.

The third parameter is a function definition. That's what this syntax, () => {}, means in JavaScript. If this is unfamiliar, look up arrow functions. They are important to understand. Briefly, the names between the parentheses are parameters. Those will be filled in at a future point when the function gets invoked. Everything between the curly brackets is code that will be executed at that point in time. The parameters will be defined when that code gets executed.

We can actually define this code as a function all on its own, which will resemble the sorts of functions you might be more familiar with:

import fs from 'node:fs';

function logFileContents(err, data) {
  if (err) {
    console.error(err);
    return;
  }
  console.log(data);
}

fs.readFile('/Users/jill/test.txt', 'utf8', logFileContents);
  

See? Now we are defining the function in the way you are used to, then passing in a function reference to the asynchronous function. Scroll up and have a look at the version with the arrow function again. You should be able to see how it is just defining the exact same behaviour and parameters (I copied and pasted, and all I had to do was add a name and remove the arrow). The arrow function is a handy way of giving us an unnamed function, in a case where we don't expect to repeat that behaviour elsewhere, so we don't need to name it. But it is fundamentally the same thing, so don't let the syntax distract you.

When we execute fs.readFile, the runtime will delegate the process of reading a file, then it will return to us, allowing the user to continue using the UI, or the server to keep serving requests, whatever it is that needs to be done in the meantime. It will keep on checking it's event queue. Eventually, an event will show up that tells the event loop that the file has been read. It will check what functionality it needs to perform on that event, and it will find our function. At that point, it will execute the function, and our data will be logged to the console. That's how callbacks work.

Promises vs Callbacks

Now we understand callbacks, but we still haven't explained promises. To do that, we need to start by understanding a limitation of callbacks. The big problem with callbacks is that we don't have any control over when they execute. This means that the code we write in response to each callback has to be nested inside it. This can lead us to an unpleasant situation known as callback hell, where callbacks get so deeply nested that it becomes very difficult to trace them.

The top answer to this Stackoverflow question has a good illustration of what this looks like, but doing an internet search for "callback hell" will show plenty of other examples of how nested this code can get.

You want to display some updated data to a user, but you can only do that once you have saved the updated data, and you can only do that once you've updated the data, and you can only do that once you've fetched the data and ... you get the idea. It quickly creates a mess.

That's where promises come in. They offer a much more intuitive way of writing asynchronous code, which is easier to follow, and therefore easier to debug. But importantly, they also return the data to you, allowing you to handle it when you are ready, rather than being chained to the timing of the event loop, and needing to put all your handling code inside a callback.

As usual, we need to follow this up with an example. Here's the promise-based API for the FS library.

import fs from 'node:fs/promises';

try {
  const data = await fs.readFile('/Users/jill/test.txt', { encoding: 'utf8' });
  console.log(data);
} catch (err) {
  console.error(err);
}
  

(Observe that the import statement at the top now imports from the promises library. The method has the same name, but it's a different method.)

Notice the difference? We have a returned value. We ran the promise, we waited for it to execute, then we received some data, and we got to do what we liked with it afterwards. We didn't have to put all the handling code inside a callback and pass it to the readFile function.

This is also where we get to understand the term non-blocking. The readFile method has executed in the background. The async/await makes it look like they are blocking, since the rest of the statements occur after that line, but the JavaScript runtime has still been able to execute other tasks like listening to the UI or handling requests in the background.

There are a few interesting quirks here to understand, especially if you come from programming in a synchronous language, so let's get into those next.

Promises

This guide isn't going to get into the details of how to implement promises from scratch. There are some great guides that do that already, notably this one.

I am going to explain what promises do at a high level, and I'm going to do it very simply. Promises resolve values. That's it. That's all they do. You make a request to read a file, or fetch data from a remote endpoint, and the promise tells you, it promises to you, that it will do that. Then when it finishes, it returns the value to you, to do what you want with it.

That's why you see a return value coming from the promise API, where the callback API required you to pass in the behaviour you wanted and then allow it to be called when the runtime was ready to.

There are still a couple of things to be aware of here. Firstly, a promise is still asynchronous. You can't just call

const data = fs.readFile'/Users/jill/test.txt', { encoding: 'utf8' });
  

If you tried to use that data straightaway, it would be null. You have to wait for the promise to be resolved, which is what the await statement is for. That's why it has to be used in an async function or at the top level. That way the runtime knows that we are going to wait for some results inside, rather than just executing a statement and then moving on without checking the result.

There's another way of using promises too. It looks like this:

import fs from 'node:fs/promises';

fs.readFile('/Users/jill/test.txt', { encoding: 'utf8' }).then((data) => {
  console.log(data);
}).catch((err) => {
  console.error(err);
});
  

This reads very similarly to the callback code from earlier. We are in fact passing a callback to it. The difference is that now we can chain them. The then method of a promise returns a new promise object, so you can call then on that object with a new callback. Look up promise chaining for more information. The main point is, you thereby escape callback hell. The code remains nested at one depth. And the callback you passed in still waits until the data is written before executing, so you don't get any null references like you would if you used the return value of a promise without awaiting.

As long as you can understand the then/catch pattern and the async/await pattern for interacting with promises, you will be able to use the APIs properly.

The key insight is that a promise is just returning a value. The callback you pass it, or the action you do afterwards with the return value, is unrelated to the promise itself. (Yes, of course, a promise can trigger other side effects, such as when you fetch from a POST endpoint and cause an update to a remote database, which has to be completed before a response is returned for the promise to resolve, but the promise itself is just a promise of a return value.)

You should now understand promises well enough to know when to use the methods on them, what to pass in to them, when to await their results, and how they differ from callbacks. There's one other area that might still baffle you, as it did me, until recently. And that's how to create your own promises. This isn't the same as implementing a promise library. This is just creating new promises. Especially if you have a callback that you want to wrap in a promise so that you don't have to deal with callback hell. We'll look at that next.

How to Create a Promise-based API

Let's take the FS library again as an example. We have a callback-based version, where the only way to use the return value is to pass in a callback and rely on the runtime to execute it for us when the value is ready. And we have a promise-based API where we can await the return value and use it when we are ready, allowing our code to read just like synchronous code.

So how would we create a promise that imitates the FS library's version of a promise? Let's go through it step by step.

As a reminder, this is what the callback-based version looks like:

import fs from 'node:fs';

fs.readFile('/Users/jill/test.txt', 'utf8', (err, data) => {
  if (err) {
    console.error(err);
    return;
  }
  console.log(data);
});
  

We pass our callback into the method call and allow it to be executed later.

Here's an example of a promise which wraps the callback version:

import fs from 'node:fs';

function myReadFile(path: string): Promise<string> {
  return new Promise((resolve, reject) => {
    fs.readFile(path, 'utf-8', (err, data) => {
      if (err) {
        reject(err);
      }
      resolve(data);
    });
  });
}
  

Have a look closely at what happens inside the new promise. The call to the readFile method is identical to the example above it. We pass in the path, the encoding, and a callback. In the version above, the data represents the value returned from the method, once it has finished reading the file. The file contents will be passed as an argument to our callback when it is ready. And that's what's happening here too. But now we take that returned value and we pass it to resolve, which is also a function reference.

That function reference is itself the parameter of a callback, which we pass in as an argument to the Promise. If you want to really understand how that callback gets stored, what gets passed in as an value to the resolve parameter, and how the Promise internally tracks all of it, go and look at one of those guides about implementing a Promise.

But for the purposes of understanding how to create a Promise, you just need to know this: when the callback to fs.readFile is triggered, the value is resolved. It gets set as the return value when you await the result of the promise, as when you call that function, myReadFile, or it gets passed to your callback when you call the then method.

Here's an example of calling it:

try {
  const data = await myReadFile('/Users/jill/test.txt');
  console.log(data);
} catch (err) {
  console.error(err);
}
  

It's just like the promise-based API to fs.readFile.

Hopefully you now understand how to create a Promise. It's time to take one more example, and show how we can create a promise-based API for a callback-based one that doesn't yet have promises.

Using Promises to Create a Basic Server

Node also provides a library for working with sockets. Sockets are another area that involve a lot of asynchronous demands for programming, as you will usually need to wait an unknown amount of time to receive a request from a client, or a response from a server. But the Node net.Socket API is strictly callback-based. So we will implement a basic promise-based version. It won't be robust enough for production needs, but it will illustrate the pattern.

Here's the callback-based version:

import * as net from "net";

function newConn(socket: net.Socket): void {
  console.log('new connection', socket.remoteAddress, socket.remotePort);

  socket.on('end', () => {
    console.log('EOF');
  });

  socket.on('data', (data: Buffer) => {
    console.log('data:', data);
    socket.write(data);

    if (data.includes('q')) {
      console.log('closing');
      socket.end();
    }
  });
}

let server = net.createServer();
server.on('error', (err: Error) => { throw err; });
server.on('connection', newConn);
server.listen({ host: "127.0.0.1", port: 1234 });
  

This code simply listens for a response from the client, then writes the same response back again. (As a sidenote, I'm endebted for some of my understanding of this code to this book, but unfortunately the section on converting to promises makes some strange decisions which significantly reduce clarity without explanation, so I had to abandon that book and just work out the conversion process myself.)

Each time we call socket.on we are passing in two parameters. One is the name of the event to listen for, the other is the callback that should be executed when that event occurs. The same goes for the server.on calls. Let's create some new promises.

function handleConnection(socket: net.Socket): Promise<Buffer> {
  return new Promise((resolve, reject) => {
    socket.on('error', (err: Error) => { reject(err) });
    socket.on('data', (data: Buffer) => {
      resolve(data);
    });
  });
}

function getConnection(server: net.Server): Promise<net.Socket> {
  return new Promise((resolve, reject) => {
    server.on('error', (err: Error) => { reject(err) });
    server.on('connection', (socket: net.Socket) => {
      resolve(socket);
    });
  });
}
  

Again, observe the basic pattern here. We make the same call and pass in a callback, but instead of executable code, we just resolve the return value, whether it's a socket returned from a server, or data returned from a socket. Then we can use the return values as we like:

try {
  while (true) {
    const socket = await getConnection(server);
    console.log('new connection', socket.remoteAddress, socket.remotePort);
    const data = await handleConnection(socket);
    socket.write(data);
    console.log(data);
    console.log('closing');
    socket.end();
  }
} catch (err) {
  console.error(err);
} finally {
  server.close();
}
  

(Don't forget to import the library and instantiate the server and listen on it, just like in the previous example.)

And with that, you have a promise-based API to the socket library. You can now build up a more sophisticated pattern of behaviour to parse an HTTP request, route to a framework, or whatever else you need to do with a server.

Conclusion

I hope these examples have shown you how straightforward it is to create and use promises once you understand the core concepts underneath them. The best way to improve that understanding is to go out and build some more promises. Use them in interesting ways, learn how to implement them, try using them in more complex flows and see how the asynchronous behaviour shows up.

Promises are not a difficult concept to understand as long as you remember that they're concealing a background process of waiting for a value. The rest of the complexity is introduced by your own callbacks. Keep those distinct in your mind and you should have a good enough working model to take you a long way.