An inconvenient part of GPU driver development is the fact that GPUs can crash and hang for various reasons, just like CPUs. Really, this can happen on any operating system to GPUs from any manufacturer. We’ve seen many such issues on the open source Linux driver stack for AMD GPUs. We’ve been tackling many bugs that lead to GPU hangs. We need to deal with the fact that hangs can happen and when they happen, we need to make sure that the system can recover and the user can keep using their Linux desktop. The purpose of this blog post is to give an overview of what the problem is and what steps we are taking to improve it.

Why does a GPU hang or freeze in the first place?

Many users are referring to this topic as “the bug”, as if it was just one problem. If only we lazy driver devs fixed just that one problem, life would be better, wouldn’t it?

In reality however, there are many different reasons why a GPU can hang, in fact there are so many possible reasons for hangs that it would be impossible to list all of them. We have fixed so many of these kinds of bugs over the years that it’s really impossible to even give a list of those bug fixes.

I’ll try to come up with some examples:

  • Issues with shader instructions, eg. incorrect opcodes, unmitigated hazards, infinite loops, etc.
  • On some GPUs, accessing unmapped memory (ie. a page fault) will also lead to a GPU hang
  • Invalid commands submitted to the GPU
  • Deadlock, ie. waiting for something that never happens (sometimes also caused by coherency issues)
  • Missed interrupt which leads to a deadlock
  • And many others…

Setting expectations

When talking about this problem space, it’s often difficult to judge what we are talking about, what is possible and what isn’t, so first let’s start with setting some expectations about what we can do.

Firstly, we need to acknowledge that every GPU has different IP blocks (parts such as graphics, compute, DMA engines, display, video decoder, etc.) and each IP block may offer a very different uAPI and programming model. Therefore, each of them can fail in different ways and need to be recovered in different ways, too. It is impossible to handle every conceiveable issue with every IP block the same way. For example:

  • In order to use graphics and compute, we have a command submission uAPI which userspace applications can use to submit jobs. Under the hood, user-mode drivers (UMD) use this uAPI to execute Vulkan, OpenGL commands.
  • The display engine offers the modesetting uAPI, which is largely shared between all GPU vendors and works on very different principles.
  • Some IP blocks like MC (memory controller) and others are not directly exposed to userspace, but are essential for correct functionality. When something goes wrong with one of these, that requires separate consideration.

This blog post focuses on GPU hangs caused by jobs submitted by userspace applications.
The other topic are out of scope for this post.
So, what can we reasonably expect about job submissions?

  • When an app (or game) submits commands that crash or hang the GPU, the graphics context of that app is considered “guilty”. We need to accept that it won’t be able to continue and will crash (unless it was specifically designed to handle GPU failure). In some cases (such as VM faults, known hazards etc.) we can make an effort to try to stop it from crashing, but not always.
  • We should do our best to make sure we don’t freeze or crash other “non-guilty” applications or the whole desktop. However, sadly, trade-offs need to be made for the sake of performance.

Why is it difficult to deal with GPU hangs?

Let’s start with a quick recap of how graphics drivers work. The way the graphics stack handles GPU jobs is the following:

  • A userspace driver generates commands for the GPU (and compiles shaders, etc.)
  • A job with those commands is submitted to the kernel driver through a uAPI (userspace API)
  • The kernel resolves job dependencies, BO (buffer object) state etc. and writes the job to a hardware ring buffer (which is shared accross many processes)
  • The GPU executes the commands from the job and when they are completed, signals a fence

The above model has barely any room for detecting and handling errors. The only thing we can detect is that a GPU job didn’t complete within a timeout (eg. 2 seconds). We don’t even know if that’s because the commands are taking too long (and would complete in a longer timeout) or the GPU is stuck somewhere and isn’t making any progress. Furthermore, job execution may overlap, so when it hangs, we can’t always know for sure which job was responsible.

Actually, it’s even worse than that. Modern GPUs have multiple job queues that can execute in parallel and all share the same resources (eg. compute units for running shaders). That means it is not even possible to be sure which queue is really responsible for a timeout: If two different jobs are executing on two queues in parallel, it can happen that the “guilty” job hogs all compute units, causing the other queue to “starve” and time out.

So, where does that leave us?

  • We can detect when a job times out
  • We don’t know which job is really responsible
  • We don’t know which queue is really responsible
  • We do know which jobs were in flight when a timeout happened
  • We can’t know why the timeout happened

How do we deal with GPU hangs, anyway?

Thanks to the excellent work of Alex Deucher and his team at AMD on the amdgpu kernel driver, various strategies were developed over the years to recover GPUs from a hung state and to make the problems less severe.

Enforcing isolation

Enforcing isolation means that we try to reduce how much an application using the GPU can affect another, effectively disabling parallel execution from different contexts. This greatly limits the scope of what jobs are potentially affected when there is a crash or hang, so makes it less likely that a “guilty” context can crash others.

  • Currently, amdgpu does not allow jobs from different contexts to overlap on the same queue. This means when only one queue is active, we can 100% identify the “guilty” context correctly.
  • There is a kernel parameter amdgpu.enforce_isolation which will additionally isolate different contexts that are running on different queues. This is currently disabled by default for performance reasons, you can enable it for better stability.

ASIC reset

ASIC reset is the simplest, and most destructive type of reset:

  • All pending or in-flight jobs are killed
  • The GPU is completely reinitialized
  • On dedicated GPUs, VRAM is erased

That means that all processes that used the GPU will lose all resources they may have had in VRAM (except on APUs). The user sees that the screen turns black briefly, then every application and the desktop just crash (unless the compositor and apps were robust). This reset strategy means that just one misbehaving application can cause all other applications and the entire desktop to crash. It is better than watching a completely frozen screen, but not by much.

Soft recovery

Soft recovery was the first attempt at making resets less destructive. What it does is it kills all currently running shaders on a specific queue and hopes that the GPU can then move on and complete the job.

This can solve quite a few issues such as infinite loops in shaders, but it is somewhat dangerous because the kernel doesn’t really have any knowledge if it worked, and can only judge by seeing whether the job now completes within a certain time or not. There is no way to know if the current job or subsequent jobs will actually work. So, soft recovery should be avoided when better recovery methods are available.

Queue reset

Queue reset (as its name suggests) attempts to reset just one specific queue without affecting others. There are mainly two ways this can be implemented for different hardware blocks:

  • Graphics and compute queues on newer GPUs have a firmware-assisted queue reset, which means it’s a feature of the CP (command processor) firmware. It basically terminates all operations (including shaders) that are executing or pending on the specific queue, and then moves on.
  • SDMA, VCN and other queues can be reset by simply resetting the whole hardware IP block. These blocks usually only have one single queue, so the reset doesn’t perturb anything else.

In my opinion, queue reset is the best way to handle GPU recovery. The only issue with queue reset was that in itself, it would still kill all currently executing or pending jobs from the given queue, even those jobs that haven’t started yet. From a user perspective, that means it could still crash other apps and the desktop (unless you enabled enforcing isolation too).

Starting from Linux 6.18, an important improvement was made to queue resets: the kernel can now re-emit pending jobs from other contexts after the queue reset is complete, so in practice it is very likely that only the “guilty” misbehaved app is killed and everything else can continue. This is not 100% guaranteed though, because it is still possible that a job from a different queue can starve other jobs. For the safest user experience, you should turn on enforcing isolation too.

IP block soft reset

I introduced IP block soft reset as a GPU recovery method very recently. This method is more blunt than the queue reset, because IP block soft reset will reset an entire IP block including every queue it has. For the graphics/compute block this means it will practically reset all graphics and compute queues at the same time. The reason why I added this is because it works on GPUs that don’t have firmware support for queue reset, or in situations where the firmware failed to do the reset. This method uses the same re-emit code that was added for queue reset so it is very likely to be able to keep your system running after a hang.

The main benefit of this reset method is that it’s markedly better than a full ASIC reset because it doesn’t erase the contents of VRAM, and it can be used on old APUs where ASIC reset is not available at all.

Starting from Linux 7.3 you can benefit from this new recovery method on GCN 1-4.

But… what about the page flip timeout and other issues?

Page flip timeouts are a different beast entirely, because they are caused by issues with the display engine which has a different programming model and userspace interacts with it using a different uAPI. So this is out of scope for the current post.

However, I need to mention that Leo Li has done some excellent work tracking down the root cause for many page flip timeout related issues.

Recommendations

As you can see, a lot of improvements have been made to GPU recovery recently, so if you experience a lot of GPU hangs, my recommendation is to upgrade your kernel if possible. If that’s not possible, consider enabling enforcing isolation.

Here is some simple advice in a nutshell:

  • For good GPU recovery
    • Queue resets with re-emit on RDNA ― use Linux 6.18 or newer
    • Queue resets with re-emit on Vega ― use Linux 7.0 or newer
    • IP block soft reset with re-emit on GCN 1-4 ― use Linux 7.3 or newer
    • For the safest experience, use amdgpu.enforce_isolation=1
  • If you need to use older kernels
    • On RDNA, use amdgpu.enforce_isolation=1 which will make queue resets work much better
    • You need to accept that Vega and older don’t have any decent recovery options on those kernels
  • If you use newer kernels but still experience problems
    • Try amdgpu.enforce_isolation=1
    • If that didn’t help, open an issue here and please don’t forget to mention your system specs, upload a dmesg log and write down the steps to reproduce

Hope this helps!