Why is async considered better performing than multithreading?

Question

I understand both asynchronous and multithreaded programming and I have done both and can do them with ease. However one thing still bugs me: why is the general consensus that async is better performing than multithreading? (added: I'm talking about the case where either approach is viable and you get to make a choice)

At the first glance the reason seems clear - less threads, less work for the OS scheduler, less memory wasted for stack space. But... I don't feel like these arguments hold water. Let's look at them individually:

1. Less work for the OS scheduler. True, but does it mean less work in total? There are still N tasks running in parallel, SOMEBODY has to switch between them. Seems to me like we've simply taken the work from the OS kernel and started doing it in our own userland code. But the amount of work that needs to be done hasn't changed because of it. Where then does the efficiency come from?
2. Less memory wasted for stack space. Or is it?
   • First of all, I don't know about other OS'es, but at least in Windows the stack space for a thread isn't committed all at once. There are virtual memory addresses reserved, but actual memory is committed only when it's needed.
   • And even if it was committed, it wouldn't matter much, because just allocating memory doesn't slow your program down. Not unless you run out of it, and modern computers have enough memory for thousands of stacks, especially servers.
   • And even if the stacks DO get committed and DO end up causing a memory shortage, most stacks will only be used a bit at the start (if your program is flirting with a stack overflow, you've got bigger problems to worry about). Which means that it will be possible to page most of them out anyway.
   • The real problem with large memory usage is that the CPU cache gets trashed a lot. When you've got lots of data all over the place that you need, and the CPU cache cannot keep up with it all and needs to fetch things from main RAM again and again - that's when things get slow. But async programming doesn't help in this in any way. If anything, it actively uses more memory. Instead of a lean stack frame we now have separate Task objects allocated on the heap essentially for every stack frame which contain state and local variables and callback references and everything. Plus it's fragmented all over the address space which gives even more headaches to the CPU cache, because pre-fetching will be useless.

So... which elephant in the room have I missed?

Answer 1

why is the general consensus that async is better performing than multithreading? (added: I'm talking about the case where either approach is viable and you get to make a choice)

On the server side, async lets you make maximum use of threads. Why have one thread handle a single connection when it can handle hundreds? On the server side, it's not an "async vs threads" scenario - it's an "async and threads" scenario.

On the client side - where either approach is truly viable - it doesn't matter as much. So what if you spin up an extra unnecessary thread? It's just not that big of a deal, even for mobile apps these days. While technically, async can help be more efficient especially in a memory- and battery-constrained device, at this point in history it's not that terribly important. However, even on the client side, async has a tremendous benefit in that it allows you to write serial code rather than mucking around with callbacks.

There are still N tasks running in parallel, SOMEBODY has to switch between them.

No. I/O tasks as used by async do not "run" anywhere, and do not need to be "switched" to. On Windows, I/O tasks use IOCPs underneath, and I/O tasks do not "run" - they only "complete", which happens as the result of a system interrupt. More info in my blog post "There Is No Thread".

Where then does the efficiency come from?

The word "efficiency" is tricky. For example, an asynchronous HTTP server handler will actually respond more slowly than a synchronous handler. There's overhead to setting up the whole async thing with callbacks, etc. However, that slowdown AFAICT is unmeasurably small, and asynchronous code allows that server to handle more simultaneous requests than a synchronous server ever could (in real-world tests, we're talking 10x as a conservative estimate). Furthermore, asynchronous code is not limited by the thread injection rate of the thread pool, so asynchronous server code responds faster to sudden changes in load, reducing the number of request timeouts as compared to a synchronous server in the same scenario. Again, this is due to "async and threads", not "async instead of threads".

A few years ago, Node.js was heralded as an incredibly efficient server - based on real-world measurements. At the time, most ASP.NET apps were synchronous (writing asynchronous apps was quite hard before async, and companies knew it was cheaper to just pay for more server hardware). Node.js, in fact, only has one server thread that ever runs your app. It was 100% asynchronous, and that's where it got its scalability benefits from. ASP.NET took note of this, and ASP.NET Core (among other changes) made its entire stack asynchronous.

Answer 2

(In this answer, I will talk about .NET, as it is the first technology which came out with async/await)

We use threads to parallelize CPU-bound tasks, and we use asynchronous IO to parallelize IO-bound tasks.

CPU - wise:
We all know that a thread per task is wrong. we don't want too many threads, because the context switches will freeze the entire system. we don't want too few of them, because we want the tasks to be finished as soon as possible. of-course, we're looking at some sort of a threadpool.
ThreadPool was the default way of scheduling an asynchronous task on the pre Task epoch. but the threadpool had one sore problem - it was really hard to know when the asynchronous is finished, and what is the asynchronous result or exception are.
Then, came the Task. not only does the task schedules a delegate on the thread pool, when the task is done, you can get the result or exception and continue working from them with Task.ContinueWith.

IO - wise:
We all know that thread-per connection is a bad thing. if we want our optimized server to serve millions of requests per seconds, we can't just spawn a new thread for each new connection. our system will suffocate on context switching. So we use asynchronous IO. in the pre Task epoch, we used BeginRead/EndRead and BeginWrite/EndWrite, which were error prone and just pain to work with - we had to work with terrible paradigm of event-driven programing
Then, came the Task. we can initiate an asynchronous IO action, and receive the result or exception with Task.ContinueWith. it made asynchronous IO much easier to work with.

Task is the glue which bridges Asynchronous CPU tasks with asynchronous IO tasks. with one interface, we can schedule an asynchronous function and get the result with Task.ContinueWith. No wonder programing with Tasks become so popular.

Task.ContinueWith is highly unreadable and unwritable.
basically, chaining a task to a task to a task a to task... is a headache. as Node.js developers complains (even in JS async/await will be standarized sometime in the future). async/await come to the rescue here. basically, the C# compiler does a neat voodo behind the scenes. in a nutshell, it takes everything that comes after await and package it with state-machine which is called when the everything before await is done. the compiler takes synchronous code (annotated with async/await) and does the ContinueWith for you.

So, why use async/await + Task instead of a multi-threaded code?

1. async/await is thre easiest way of getting the asynchronous result or exception. (and believe me, I wrote asynchronous code in C++, C#, Java and Javascript, async/await is a paradise in that field.)
2. async/await works both with CPU-bound tasks and IO-bound tasks. same interface for two different but similar fields.
3. If you want asynchronous IO, threads will not help you anyway.
4. Task anyway is a IThreadPoolItem and is scheduled to the .Net thread pool. async/await just removes the chaining hell away. back to step one -> multi thread code.
5. Tasks + async/await synchronizes the code flow for you. most developers are not system developers. they do not know the hidden costs of synchronization objects and techniques. in most cases, the implementation provided by the framework is faster than the average implementation that the average developer can think about. ofcourse if you really try, you can write something extremely customized for your needs hence more performant, but that doesn't apply for most developers.
6. Depending on you programing language, await can be faster than a callback. Gor Nishanov is the original (Microsoft) developer who offered standarize await in C++. In his 2015 lecture, he shows that the C++ version of await is actually more performant than callback-style asynchronous networking IO. (switch to 39:30)

For specific questions :

Less work for the OS scheduler. True

False. async/await compiles to state machine. a task continuation invokes that state machine when its done. a task run on the threadpool anyway. async/await yields the same amount of scheduling as a multi threaded code / queuing a thread pool work. it's the simplicity you get which matters.

Less memory wasted for stack space. Or is it?

False. again, async/await compiles to a state machine. when invoked on task completion, it will use the same amount of stack memory for local varaiables. the continuation will anyway run on a thread (usually a thread-pool thread), so that argument is invalid.

Why is async considered better performing than multithreading?

When your code CPU-bound, there will be a little difference between between Tasks + async/await and pure multi-threaded code. In IO bound code, multi threading is the worst throughput you can have. Tasks + async/await will blow away any IO-bound-threadpool you can write your own. threads don't scale. usually (especially on Server side) you have both. you read some data from a connection (IO), then continue processing it on the CPU (json parsing, calculations etc.) and write the result back to the connection (IO again). Tasks + async/await are faster in this case than a pure multi threaded code.

It's the simplicity which makes async/await so appealing. writing a synchronous code which is actually asynchronous. if this is not "high level programing", what is?