12.2. Asynchronous Computation¶
Today’s computers are highly parallel systems, consisting of multiple CPU cores (often multiple threads per core), multiple processing elements per GPU and often multiple GPUs per device. In short, we can process many different things at the same time, often on different devices. Unfortunately Python is not a great way of writing parallel and asynchronous code, at least not with some extra help. After all, Python is single-threaded and this is unlikely to change in the future. Deep learning frameworks such as MXNet and TensorFlow utilize an asynchronous programming model to improve performance (PyTorch uses Python’s own scheduler leading to a different performance trade-off). For PyTorch, by default, GPU operations are asynchronous. When you call a function that uses the GPU, the operations are enqueued to the particular device, but not necessarily executed until later. This allows us to execute more computations in parallel, including operations on CPU or other GPUs.
Hence, understanding how asynchronous programming works helps us to develop more efficient programs, by proactively reducing computational requirements and mutual dependencies. This allows us to reduce memory overhead and increase processor utilization. We begin by importing the necessary libraries.
from d2l import mxnet as d2l
import numpy, os, subprocess
from mxnet import autograd, gluon, np, npx
from mxnet.gluon import nn
npx.set_np()
from d2l import torch as d2l
import numpy, os, subprocess
import torch
from torch import nn
import numpy
12.2.1. Asynchrony via Backend¶
For a warmup consider the following toy problem - we want to generate a random matrix and multiply it. Let us do that both in NumPy and in MXNet NP to see the difference.
with d2l.Benchmark('numpy'):
for _ in range(10):
a = numpy.random.normal(size=(1000, 1000))
b = numpy.dot(a, a)
with d2l.Benchmark('mxnet.np'):
for _ in range(10):
a = np.random.normal(size=(1000, 1000))
b = np.dot(a, a)
numpy: 0.8547 sec
mxnet.np: 0.0046 sec
This is orders of magnitude faster. At least it seems to be so. Since both are executed on the same processor something else must be going on. Forcing MXNet to finish all computation prior to returning shows what happened previously: computation is being executed by the backend while the frontend returns control to Python.
with d2l.Benchmark():
for _ in range(10):
a = np.random.normal(size=(1000, 1000))
b = np.dot(a, a)
npx.waitall()
Done: 0.8341 sec
Broadly speaking, MXNet has a frontend for direct interaction with the users, e.g., via Python, as well as a backend used by the system to perform the computation. As shown in Fig. 12.2.1, users can write MXNet programs in various frontend languages, such as Python, R, Scala, and C++. Regardless of the frontend programming language used, the execution of MXNet programs occurs primarily in the backend of C++ implementations. Operations issued by the frontend language are passed on to the backend for execution. The backend manages its own threads that continuously collect and execute queued tasks. Note that for this to work the backend must be able to keep track of the dependencies between various steps in the computational graph. Hence, it is not possible to parallelize operations that depend on each other.
For a warmup consider the following toy problem - we want to generate a
random matrix and multiply it. Let us do that both in NumPy and in
PyTorch tensor to see the difference. Note that PyTorch tensor
is
defined on a gpu.
# warmup for gpu computation
device = d2l.try_gpu()
a = torch.randn(size=(1000, 1000), device=device)
b = torch.mm(a, a)
with d2l.Benchmark('numpy'):
for _ in range(10):
a = numpy.random.normal(size=(1000, 1000))
b = numpy.dot(a, a)
with d2l.Benchmark('torch'):
for _ in range(10):
a = torch.randn(size=(1000, 1000), device=device)
b = torch.mm(a, a)
numpy: 1.0118 sec
torch: 0.0015 sec
This is orders of magnitude faster. At least it seems to be so. Numpy dot product is executed on the cpu processor while Pytorch matrix multiplication is executed on gpu and hence the latter is expected to be much faster. But the huge time difference suggests something else must be going on. By default, GPU operations are asynchronous in PyTorch. Forcing PyTorch to finish all computation prior to returning shows what happened previously: computation is being executed by the backend while the frontend returns control to Python.
with d2l.Benchmark():
for _ in range(10):
a = torch.randn(size=(1000, 1000), device=device)
b = torch.mm(a, a)
torch.cuda.synchronize(device)
Done: 0.0046 sec
Broadly speaking, PyTorch has a frontend for direct interaction with the users, e.g., via Python, as well as a backend used by the system to perform the computation. As shown in Fig. 12.2.1, users can write PyTorch programs in various frontend languages, such as Python and C++. Regardless of the frontend programming language used, the execution of PyTorch programs occurs primarily in the backend of C++ implementations. Operations issued by the frontend language are passed on to the backend for execution. The backend manages its own threads that continuously collect and execute queued tasks. Note that for this to work the backend must be able to keep track of the dependencies between various steps in the computational graph. Hence, it is not possible to parallelize operations that depend on each other.

Fig. 12.2.1 Programming Frontends.¶
Let us look at another toy example to understand the dependency graph a bit better.
x = np.ones((1, 2))
y = np.ones((1, 2))
z = x * y + 2
z
array([[3., 3.]])
x = torch.ones((1, 2), device=device)
y = torch.ones((1, 2), device=device)
z = x * y + 2
z
tensor([[3., 3.]], device='cuda:0')
Fig. 12.2.2 Dependencies.¶
The code snippet above is also illustrated in
Fig. 12.2.2. Whenever the Python frontend thread executes
one of the first three statements, it simply returns the task to the
backend queue. When the last statement’s results need to be printed, the
Python frontend thread will wait for the C++ backend thread to finish
computing result of the variable z
. One benefit of this design is
that the Python frontend thread does not need to perform actual
computations. Thus, there is little impact on the program’s overall
performance, regardless of Python’s performance.
Fig. 12.2.3 illustrates how frontend and backend interact.
Fig. 12.2.3 Frontend and Backend.¶
12.2.2. Barriers and Blockers¶
There are a number of operations that will force Python to wait for
completion: * Most obviously npx.waitall()
waits until all
computation has completed, regardless of when the compute instructions
were issued. In practice it is a bad idea to use this operator unless
absolutely necessary since it can lead to poor performance. * If we
just want to wait until a specific variable is available we can call
z.wait_to_read()
. In this case MXNet blocks return to Python until
the variable z
has been computed. Other computation may well
continue afterwards.
Let us see how this works in practice:
with d2l.Benchmark('waitall'):
b = np.dot(a, a)
npx.waitall()
with d2l.Benchmark('wait_to_read'):
b = np.dot(a, a)
b.wait_to_read()
waitall: 0.0048 sec
wait_to_read: 0.0045 sec
Both operations take approximately the same time to complete. Besides
the obvious blocking operations we recommend that the reader is aware of
implicit blockers. Printing a variable clearly requires the variable
to be available and is thus a blocker. Lastly, conversions to NumPy via
z.asnumpy()
and conversions to scalars via z.item()
are
blocking, since NumPy has no notion of asynchrony. It needs access to
the values just like the print
function. Copying small amounts of
data frequently from MXNet’s scope to NumPy and back can destroy
performance of an otherwise efficient code, since each such operation
requires the computational graph to evaluate all intermediate results
needed to get the relevant term before anything else can be done.
with d2l.Benchmark('numpy conversion'):
b = np.dot(a, a)
b.asnumpy()
with d2l.Benchmark('scalar conversion'):
b = np.dot(a, a)
b.sum().item()
numpy conversion: 0.0069 sec
scalar conversion: 0.0144 sec
12.2.3. Improving Computation¶
On a heavily multithreaded system (even regular laptops have 4 threads
or more and on multi-socket servers this number can exceed 256) the
overhead of scheduling operations can become significant. This is why it
is highly desirable to have computation and scheduling occur
asynchronously and in parallel. To illustrate the benefit of doing this
let us see what happens if we increment a variable by 1 multiple times,
both in sequence or asynchronously. We simulate synchronous execution by
inserting a wait_to_read()
barrier in between each addition.
with d2l.Benchmark('synchronous'):
for _ in range(1000):
y = x + 1
y.wait_to_read()
with d2l.Benchmark('asynchronous'):
for _ in range(1000):
y = x + 1
y.wait_to_read()
synchronous: 0.1041 sec
asynchronous: 0.0934 sec
A slightly simplified interaction between the Python frontend thread and the C++ backend thread can be summarized as follows:
The frontend orders the backend to insert the calculation task
y = x + 1
into the queue.The backend then receives the computation tasks from the queue and performs the actual computations.
The backend then returns the computation results to the frontend.
Assume that the durations of these three stages are \(t_1, t_2\) and \(t_3\), respectively. If we do not use asynchronous programming, the total time taken to perform 1000 computations is approximately \(1000 (t_1+ t_2 + t_3)\). If asynchronous programming is used, the total time taken to perform 1000 computations can be reduced to \(t_1 + 1000 t_2 + t_3\) (assuming \(1000 t_2 > 999t_1\)), since the frontend does not have to wait for the backend to return computation results for each loop.
12.2.4. Improving Memory Footprint¶
Imagine a situation where we keep on inserting operations into the
backend by executing Python code on the frontend. For instance, the
frontend might insert a large number of minibatch tasks within a very
short time. After all, if no meaningful computation happens in Python
this can be done quite quickly. If each of these tasks can be launched
quickly at the same time this may cause a spike in memory usage. Given a
finite amount of memory available on GPUs (and even on CPUs) this can
lead to resource contention or even program crashes. Some readers might
have noticed that previous training routines made use of synchronization
methods such as item
or even asnumpy
.
We recommend to use these operations carefully, e.g., for each minibatch, such as to balance computational efficiency and memory footprint. To illustrate what happens let us implement a simple training loop for a deep network and measure its memory consumption and timing. Below is the mock data generator and deep network.
def data_iter():
timer = d2l.Timer()
num_batches, batch_size = 150, 1024
for i in range(num_batches):
X = np.random.normal(size=(batch_size, 512))
y = np.ones((batch_size,))
yield X, y
if (i + 1) % 50 == 0:
print(f'batch {i + 1}, time {timer.stop():.4f} sec')
net = nn.Sequential()
net.add(nn.Dense(2048, activation='relu'),
nn.Dense(512, activation='relu'), nn.Dense(1))
net.initialize()
trainer = gluon.Trainer(net.collect_params(), 'sgd')
loss = gluon.loss.L2Loss()
Next we need a tool to measure the memory footprint of our code. We use
a relatively primitive ps
call to accomplish this (note that the
latter only works on Linux and MacOS). For a much more detailed analysis
of what is going on here use e.g., Nvidia’s
Nsight or
Intel’s vTune.
def get_mem():
res = subprocess.check_output(['ps', 'u', '-p', str(os.getpid())])
return int(str(res).split()[15]) / 1e3
Before we can begin testing we need to initialize the parameters of the network and process one batch. Otherwise it would be tricky to see what the additional memory consumption is. See Section 5.3 for further details related to initialization.
for X, y in data_iter():
break
loss(y, net(X)).wait_to_read()
To ensure that we do not overflow the task buffer on the backend we
insert a wait_to_read
call for the loss function at the end of each
loop. This forces the forward propagation to complete before a new
forward propagation is commenced. Note that a (possibly more elegant)
alternative would have been to track the loss in a scalar variable and
to force a barrier via the item
call.
mem = get_mem()
with d2l.Benchmark('time per epoch'):
for X, y in data_iter():
with autograd.record():
l = loss(y, net(X))
l.backward()
trainer.step(X.shape[0])
l.wait_to_read() # Barrier before a new batch
npx.waitall()
print(f'increased memory: {get_mem() - mem:f} MB')
batch 50, time 3.1852 sec
batch 100, time 6.2475 sec
batch 150, time 9.5391 sec
time per epoch: 9.5523 sec
increased memory: 24.284000 MB
As we see, the timing of the minibatches lines up quite nicely with the overall runtime of the optimization code. Moreover, memory footprint only increases slightly. Now let us see what happens if we drop the barrier at the end of each minibatch.
mem = get_mem()
with d2l.Benchmark('time per epoch'):
for X, y in data_iter():
with autograd.record():
l = loss(y, net(X))
l.backward()
trainer.step(X.shape[0])
npx.waitall()
print(f'increased memory: {get_mem() - mem:f} MB')
batch 50, time 0.1210 sec
batch 100, time 0.2446 sec
batch 150, time 0.3853 sec
time per epoch: 9.8210 sec
increased memory: -8.228000 MB
Even though the time to issue instructions for the backend is an order of magnitude smaller, we still need to perform computation. Consequently a large amount of intermediate results cannot be released and may pile up in memory. While this didn’t cause any issues in the toy example above, it might well have resulted in out of memory situations when left unchecked in real world scenarios.
12.2.5. Summary¶
MXNet decouples the Python frontend from an execution backend. This allows for fast asynchronous insertion of commands into the backend and associated parallelism.
Asynchrony leads to a rather responsive frontend. However, use caution not to overfill the task queue since it may lead to excessive memory consumption.
It is recommended to synchronize for each minibatch to keep frontend and backend approximately synchronized.
Be aware of the fact that conversions from MXNet’s memory management to Python will force the backend to wait until the specific variable is ready.
print
,asnumpy
anditem
all have this effect. This can be desirable but a carless use of synchronization can ruin performance.Chip vendors offer sophisticated performance analysis tools to obtain a much more fine-grained insight into the efficiency of deep learning.
12.2.6. Exercises¶
We mentioned above that using asynchronous computation can reduce the total amount of time needed to perform \(1000\) computations to \(t_1 + 1000 t_2 + t_3\). Why do we have to assume \(1000 t_2 > 999 t_1\) here?
How would you need to modify the training loop if you wanted to have an overlap of one minibatch each? I.e., if you wanted to ensure that batch \(b_t\) finishes before batch \(b_{t+2}\) commences?
What might happen if we want to execute code on CPUs and GPUs simultaneously? Should you still insist on synchronizing after every minibatch has been issued?
Measure the difference between
waitall
andwait_to_read
. Hint: perform a number of instructions and synchronize for an intermediate result.