Promises seems simple at first, and the basic use case for Promises is, in fact, straightforward and simple. But they can become surprisingly confusing for anything beyond [...]"You may need to read this section more than once" - or in my case, read the section more than once, struggle with production code, read the section some more, and then write a long-winded article about asynchronous programming in Javascript.
This can be one of the trickiest parts of Javascript to understand, and you may need to read this section more than once.
- David Flanagan, Javascript: The Definitive Guide
Did I say long-winded? I meant "comprehensive but refreshingly concise". So let's get right into it.
Part 1 of this article is about the different kinds of important asynchronous programming patterns in modern Javascript. Part 2 is about popular libraries for asynchronous code, which not only explains how these libraries work but hopefully gives a lot of insight into the different ways that people like to write async code, and they way it has evolved over time. Part 3 will be about some technical details and "gotcha"s that I have run into while writing async code, and Part 4 will be a list of examples of async code "in the wild" that I thought were instructive - all taken from a recent refactoring project I did.
Quick Links
Part 1: Asynchronous Programming Patterns
Let's Be Side-By-Side
In Python it would work like this:(Ex: 1)
1 print("Let's read a file")
2 f = open("file.txt", "r")
3 fileText = f.read()
4 print("The file has been read.")
Pretty straightforward. This is a single-threaded program, everything uses the same call stack, and everything happens in the same sequence of the lines of code. The file is read synchronously, which means it is in-sync with the rest of the program.
- We print a message to the console
- We open the file
- We read the file
- We print a message to the console.
Here's how we might do it in Node.js:(Ex: 2)
1 fs = require("fs")
2 console.log("Let's read a file")
3 fs.readFile(path, encoding, callback)
4 console.log("We did it!")
Wrong.
Like the Python code, this Javascript code is still single-threaded, uses one call stack, and everything happens in order:
- We import the required module
- We log a message to the console
- We make a request to the node.js "fs.readFile" api
- We log a message to the console.
See the difference? The readFile function doesn't actually read the file. It tells the environment (in this case it is Node, but it could be a browser or anything else) to read the file. The actual reading of the file is done "asynchronously", that is, it happens out of sync with the rest of the code. We don't actually know when the file will finish being read. So how do we work with it?
Will you call me back?
You will notice thatfs.readFile() takes a "callback" as the third argument.
This argument expects a function, and the node api will run that function once it has completed.
The structure may vary, but callbacks in Node are usually passed two arguments.
The first will be an error if an error was thrown, or null if no error was thrown.
The second will be the return data, if present.
So for example, if we wanted to print out the contents of the file to the console, our code would look like this:(Ex: 3)
1 fs = require("fs")
2 console.log("Let's read a file")
3 fs.readFile(path, encoding, (err, data) => {
4 if (err) {
5 console.error(err)
6 return
7 }
8 console.log(data)
9 })
10 console.log("We did it!")
Let's read a file
We did it!
[file text]
It prints "out-of-order" because the call stack has to clear before the node api will put the callback onto the stack, so no matter when the file finishes being read, it will always run the callback after
console.log("We did it!").
The implication of this is that when the console says "We did it!", we will not, in fact, have done it, and we may never do it if an error is thrown during the file read.
fs.readFileSync() instead of fs.readFile(), and save yourself the trouble of reading this article.
If you make me a promise...
1 fs = require("fs")
2 console.log("Let's read a file")
3 let file = new Promise((resolve, reject) => {
4 fs.readFile(path, encoding, (err, data) => {
5 if (err) return reject(err)
6 resolve(data)
7 })
8 })
9 console.log("We did it!")
Again: a promise represents the result of an asynchronous computation.
A promise represents the result of an asynchronous computation.
Okay okay but what even is the result of an asynchronous computation? After all, isn't it initially undefined because the computation hasn't been performed?
This isn't the easiest question to answer, but it isn't the hardest either. There are fives states a promise can be in:
-
Settled
- Fulfilled
- Rejected
- Resolved
- Pending
So in Example 4 above, the asynchronous computation is
fs.readFile() (and the corresponding callback).
Initially, the Promise is Pending. At some point, Node will finish with readFile() and run the callback.
If the callback receives an error, it rejects the promise by calling reject(), and otherwise it calls resolve().
Immediately after the callback rejects or resolves, the Promise will be either Rejected or Resolved.
Not so bad, right?
Hey, you just said a Settled Promise would be Rejected or Fulfilled, not Rejected or Resolved, so what the heck is going on here? Why doesn't the callback call a function named Fulfil()"Okay fine it's not quite that simple. Remember what we just said about Settled Promises? For a value to be "Settled", it should be in a state that won't change anymore. However, the argument we pass in to
- You, just now
resolve() could itself be an unsettled Promise.
In this case, the Promise has Resolved, but to an unsettled value. It is a subtle but meaningful distinction.
A Resolved Promise will Fulfil or Reject as soon as the value it has resolved to is settled. For a simple example like our
readFile() Promise, this will happen immediately and we will never have to worry about these subtleties.
... you better keep your promise!
There are two ways to use Promises.
The first is to use callbacks with them. Promises have methods called
.then() and .catch(), which let you define callbacks which will be run when the promise Fulfils or Rejects.
You can chain multiple .then() calls together, to pass the result of one to the next.
This is a great way of writing asynchronous code, it was the original goal of Promises when they were first introduced,
its very useful and solves a lot of the problems with purely callback-based code, and I'm not going to talk about it at all.
This whole article assumes that the reader already has some familiarity with async Javascript, and there are plenty of good descriptions already out there.
After promises were introduced, some smart people figured that it would be great if you could tell your program that you want to wait for the Promise to Settle before it moves to the next line of code, which led to the development of a second use pattern for Promises.
Consider this pseudo-example:(Ex: 5)
1 fs = require("fs")
2 console.log("Let's read a file")
3 WAIT let file = new Promise((resolve, reject) => {
4 fs.readFile(path, encoding, (err, data) => {
5 if (err) return reject(err)
6 console.log("The file was read:")
7 resolve(data)
8 })
9 })
10 console.log(file)
11 console.log("We did it!")
console.log() statements in the order that they appear in the program, and at the point when console.log(file) runs, "file" is guaranteed to be Settled.
It still wouldn't be synchronous code, strictly speaking, but we could write it and reason about it as though it is, which makes a lot of things easier.
Okay so if you're like me when I first learned this, at this point you're going saying something like this:
What?? I thought the whole point of asynchronous code was that it DOESN'T pause the program execution? Why would we go through all the trouble of writing async code and then go to more trouble to make it act like its synchronous?And you're right! You shouldn't do that. What gets missed here is the fact that this whole thing is still going to be async. Consider this example:(Ex: 6)
- David Booth, 2021
You have a program you're writing and you want to be able to call a function to find out what the current time is.
You also want to be able to pass in a string describing the format you want the time to be returned in.
I say "great, I'll write a module for you that will expose a function that does that, named whatTimeIsItRightNow(format)"
So I write the module: it parses the input, queries WhatTimeIsIt.com to find out the current time, then formats the time the way you want and returns the formatted time as a string.
Inside my module, when I query WhatTimeIsIt.com, that query will be an asynchronous operation, and I need to be able to wait for the response before I can format it and pass it back. That's where I would use the kind of construction shown in Example 5.
This means that whatTimeIsItRightNow() is going to return a promise, and the program that imports and runs it is still going to have to deal with it somehow.
Nonetheless, the ability to wait for a Promise was useful, because no matter what else we do, we still want to format the date string, and we still have to wait for the query to resolve before we do that, and being able to wait for the promise made it easy for us to do those things. And of course, waiting for the query didn't block the call stack, because the whole function is async.
I promise I'll wait for you
1 const whatTimeIsItRightNow = async function (format) {
2 // Parse format argument
3 let format = parseFormat(format)
4
5 // Get time from server
6 let time = await queryWhatTimeIsIt()
7
8 // Format time
9 let formattedTime = formatTime(time, format)
10
11 // Return
12 return formattedTime
13 }
- It will return a Promise.
- We are allowed to use the "await" keyword inside the function.
Part 2: Popular Practical Promise Paradigms and Patterns
- There is a lot of useful async functionality possible that we have not discussed: parallel computation, awaiting for multiple promises at once, promise maps, etc.
- People have been writing asynchronous Javascript for a long time - before Async/Await, and even before Promises were introduced
Here in Part 2 we will look at some of the async libraries that exist, to understand different async patterns and stuff.
Modern Promise Functionality in JS
- I have a block of code that I want to run once the promise settles, whether it resolved or rejected.
- I have multiple async operations to perform, and I want all of them to complete before I call
.then()or before moving on fromawait. - I have multiple async operations I'm going to perform, but I only need the result of the first one that finishes
1:
- Use
.finally() at the end of the Promise chain. The function you pass in does not take any arguments.2:
- If you want it to "fast-fail" (reject the whole thing if ANY of the Promises reject) use
Promise.all() to wrap an iterable of the Promises.- If you want to get an array of the results - resolved or rejected - use
Promise.allSettled() to wrap the Promises.3:
- If you want the result of the first Promise that fulfils successfully, and to only get an error if ALL fail, wrap an iterable of the Promises in
Promise.any().- If you want the result of the first Promise to settle, regardless of whether if fulfils or rejects, wrap with
Promise.race()Before Modern Features - Thunks, Promises, and Yield with co
co(), and co.wrap().Let's see what some of the examples from Part 1 would look like if written with co:
Example 5 with co:
1 fs = require("fs")
2 co( function* () {
3 console.log("Let's read a file")
4 yield let file = new Promise((resolve, reject) => {
5 fs.readFile(path, encoding, (err, data) => {
6 if (err) return reject(err)
7 console.log("The file was read:")
8 resolve(data)
9 })
10 })
11 console.log(file)
12 console.log("We did it!")
13 })
Example 7 with co:
1 const whatTimeIsItRightNow = co.wrap( function* (format) {
2 // Parse format argument
3 let format = parseFormat(format)
4
5 // Get time from server
6 let time = yield queryWhatTimeIsIt()
7
8 // Format time
9 let formattedTime = formatTime(time, format)
10
11 // Return
12 return formattedTime
13 })
co.wrap() is used to wrap a function2 into an async function,
and co() just directly runs a block of code as though it were async (like an IIFE).
Both functions return a promise and are "then-able", i.e. the following are valid:
co(function* () { [...] }).then().catch()
const asyncFunc = co.wrap(function* () { [...]})
asyncFunc().then().catch()
Pretty straightforward, right?
Hey, co was first released in 2013, but Promises weren't even part of the ECMAScript spec until 2015. Also, I just looked at the co documentation and it lists six different yieldable types, not just Promises! What aren't you telling us?"Okay okay fine. Here's the deal. Co can also handle
- You, again
yield-ing a generator or generator function (basically nesting the operation),
it can also handle arrays and objects so that you can do parallel processing, and - this is the big one - it can handle "thunks".
Before Promises took over, async code was all about functions, and passing functions around. Sure enough, this is how co worked as well. You would
yield a function which took just one argument, a callback, and co would pass in its own special callback and work its magic:
1 const asyncFunc = (callback) => {
2 try {
3 result = someOperation()
4 callback(null, result)
5 } catch (err) {
6 callback(err)
7 }
8 }
9
10 co( function* () {
11 yield asyncFunc
12 })
This is great if you have a function that doesn't take any arguments, but most of the time that isn't the case.
Here is the solution, for a simple (if contrived) example of making an XHR request:
1 const makeRequest = function (url) {
2 function (callback) {
3 xhr(url, callback)
9 }
10 }
11
12 co( function* () {
13 yield makeRequest("http://www.example.org/example.txt")
14 })
yield makeRequest(args) works just fine, because makeRequest(args) evaluates to a function which takes the callback as its only argument.
The co() block is still super readable. The only thing we need to understand is the little functional programming pattern we used to define makeRequest.
makeRequest is a function which returns a function. The specific pattern here is defined by two things:
- The inner function that gets returned is something that needs to take very specific arguments in a specific order
- The arguments of the outer function are used to define the behavior of the inner function
That's right, this pattern is called a "thunk", and that little play on words is the reason why it has that name. I swear I'm not making this up.
Thunks aren't just used for async code with co, they are a broader functional programming pattern that crop up in other places, like in redux.
Don't keep me in suspense! yielding with suspend
The suspend library is not popular (~150 weekly downloads at time of writing), but it was used by the codebase I am responsible for professionally, and it is a little instructive, so I'll talk aboutsuspend.run() which is the feature that we used.
In a simple case,
suspend.run() is basically identical to co(). Consider example 5 with suspend.run():
1 fs = require("fs")
2 suspend.run( function* () {
3 console.log("Let's read a file")
4 yield let file = new Promise((resolve, reject) => {
5 fs.readFile(path, encoding, (err, data) => {
6 if (err) return reject(err)
7 console.log("The file was read:")
8 resolve(data)
9 })
10 })
11 console.log(file)
12 console.log("We did it!")
13 })
It's exactly the same as when we used
co(). suspend.run() wraps the code and allows us to use yield to wait for the Promise to Settle.
Lets take a step back for a second to look at what's going on here.
fs.readFile() is a callback-based async function, so we wrapped it in a Promise, so that we could wait for the Promise to Settle and then use that Settled value.
If we used async/await, this would be necessary, but Suspend offers another way:
1 fs = require("fs")
2 suspend.run( function* () {
3 console.log("Let's read a file")
4 file = yield fs.readFile(path, encoding, suspend.resume())
5 console.log("The file was read:")
6 console.log(file)
7 console.log("We did it!")
8 })
Instead of wrapping
fs.readFile() in a Promise, we simply pass suspend.resume() as the callback.
When the computation that yield is operating on is complete and the callback is run, Suspend knows how to resume the function execution and return the computation result.
This is similar to what we did in co(), but suspend lets you manually tell it how to pass the callback around, so you don't have to write thunks.
async functions with ... async
Gotta write this!
Part 3: Boring Details and Spicy Gotchas
Call Stack and Event Loop
9 // getUser is a function that queries the database for the currently logged in user
10 // it returns a promise which will resolve to the user if it is successful
11
12 console.log("search for user")
13
14 const userTalk = async function () {
15 console.log("Hi there, remember me?")
16 let user = await getUser()
17 console.log("Of course I remember you, " + user.name + "!")
18 }
19
20 userTalk().catch(err => console.error(err))
21
22 process.exit()
- On line 12 it outputs "search for user", which obviously works.
- An async function is defined
- On line 20 it calls the function
- Stepping in to the function: on line 15 it outputs "Hi there, remember me?"
- On line 16, wait for the result of getUser()
- On line 17, output another message
- Now that the function is done, continue past line 20, to process.exit() on line 22
So that's all good and fine, except that its wrong. If you've ever read a tutorial or description about async/await that says something like "await makes the code act like its synchronous even though its asynchronous" this is what they are talking about. Even though it LOOKS synchronous, under the hood it still isn't. This is one of those examples where there will be unexpected behavior if you don't know the difference.
Here is the actual execution:
- line 12 outputs a message
- an async function is defined
- line 20 calls the async function
- line 15 outputs a message
- line 16 sends off getUser() to be executed
-
await makes the async function exit for now, ready to resume when the result of getUser() comes back- the call stack, which is currently just main(), continues executing
- line 22 kills the process
---- if line 22 wasn't there ----
- the stack clears, and when getUser() is done, the event loops pulls it back onto the stack
- the async function resumes with the result of getUser()
- line 17 outputs another message
One of the important things to notice here is that if there was no
process.exit() on line 22, it would behave exactly like the first execution sequence we listed,
and look very much like synchronous code.
Explanation of call stack, promise and callback queues, API land, and event loop.
IIFEs
10 const doSomethingAwesome = () => {
...
18 }
19
20 co( function ()* {
...
26 })
co() block.
Technically async/await has no exact replica of this, because co() runs the contained code immediately,
but if we want to do the same thing with async/await we can define a function and immediately invoke it.
This is called an "immediately invoked function expression" or IIFE.
10 const doSomethingAwesome = () => {
...
18 }
19
20 (async function () {
...
26 })()
.then() etc. to the end as well. So what does this code do?
The code above defines a function called doSomethingAwesome and then immediately calls it with an async function as an argument. Immediately invoked function expression!
Wait what?
No I didn't make a mistake. The function which is defined and immediately called in the above code is the function
doSomethingAwesome.
Remember that in Javascript, semicolons denote breaks between statements, not line breaks. Javascript will usually interpret a line break as a semicolon,
but not if it is able to continue the statement using the next line. David Flanagan says "in general, if a statement begins with (, [, /, +, or -, there is a chance it could be interpreted as a continuation of the statement before."
You can put a semicolon at the end of line 18, of course, but if the code gets changed later the same problem might appear.
A "defensive semicolon" can be used to resolve this:
10 const doSomethingAwesome = () => {
...
18 }
19
20 ;(async function () {
...
26 })()
Part 4: Examples
Placeholder
Gotta write this!