Introducing Concurrency

Concurrency

Concurrency is not a new concept, but it has gained popularity over the years as the number of cores and threads per core on modern CPUs has increased. And it's the only way to improve performance with limited resources.

Concurrency is important and widely used in modern computing systems, operating systems (OSs), and large distributed clusters. It helps model the real world, maximises system efficiency from users’ and developers’ perspectives, and enables developers to solve large, complex problems.

Latency vs. Throughput

Performance in computing is quantified in two ways: latency and throughput.

• Latency is the measure of how long a single task takes.
• Throughput is the number of tasks a system can handle over a period of time.

For example, if it takes 1 hour for one person to go from point A to point B on a motorcycle and then to a bus carrying 50 people, it takes 2 hours; the bike's latency is lower, but the bus's throughput is 25x that of a motorcycle.

Optimising for one may degrade another.

Concurrency can help with latency by breaking down a long-running task and executing it in parallel. Concurrency can improve throughput by enabling us to perform multiple tasks simultaneously.

Therefore, using concurrency can improve system performance in three main ways:

• It can reduce latency (i.e., make a unit of work complete faster).
• It can hide latency (that is, allow the system to accomplish something else during a high-latency operation).
• It can increase throughput (that is, make the system able to do more work).

Scalability

Concurrency can help us achieve scalability by allowing us to scale out (scale horizontally) rather than scale up (scale vertically).

• Scaling vertically means buying more RAM or a faster processor to get only marginal improvements, and vertical scaling stops once you have already bought the fastest system.
• But we can scale horizontally by distributing the work across multiple machines.

Concurrency can be used to decouple the system. If only one service needs to scale, that can be done easily in a decoupled system rather than to scale everything on a tightly coupled system.