AMD is known to be Linux friendly and has had an open source driver stack for their GPUs for more than a decade. However, there was one GPU that has always been an outcast on Linux because it has never worked: the Radeon HD 7870 XT. This post is about how I fixed that so that Linux gamers can enjoy this GPU being fully functional now.

The Radeon HD 7870 XT is built on the GCN 1 architecture (also known as Southern Islands or SI, or GFX6). It has a chip called “Tahiti LE” which is a variant of the Tahiti chip that was quite high-end by 2012 standards. Tahiti and other GCN 1 chips have been supported on Linux for more than a decade, first by the radeon kernel driver and more recently by the amdgpu kernel driver.

We have every reason to assume that the 7870 XT should “just work”, so why doesn’t it?

Motivation

Why care whether a 15 years old GPU works today or not?

  • If the driver stack claims to support GCN 1, we should indeed support all GCN 1 chips without exceptions.
  • If there is a serious bug that prevents a GPU from working, there is a chance the bug also affects other GPUs. It is worth an investigation.
  • Most importantly, it’s a good challenge to see if I can figure out a problem like this.

Story time: what is a harvested (cut down) chip?

If you look at any GPU manufacturer, they have a lot of different products every generation, and those products use different variants of the same few different chips. How is that possible?

Due to economies of scale, there is a common practice among chip makers: they prefer to manufacture massive quantities of the same few chip instead of a small quantity of many different kind of chips. However, in order to have many different products, they can configure the same chip in different ways and sell those variants under different product names. Today we are focusing on AMD’s old Tahiti chip, so let’s use that as an example. How many different products did they launch that use the Tahiti chip and its different variants or refreshes? We can use Wikipedia to check that:

  • Radeon HD 7870 XT, 7950, 7970, 7990, 8950, 8970, 8990
  • Radeon R9 280, 280X
  • FirePro W8000, W9000, D500, D700, S9000, S9050, S10000

All of those use the Tahiti chip. But what are the differences between those products?

  • Different target audience, eg. workstation vs. consumer
  • Different memory configuration (memory size and bus width)
  • Different shader and memory clock speed
  • Different amount of compute units, render output units (aka. render backend or RB), etc.

How is this achieved?

At the factory, each chip is examined automatically. Due to variance in the chip manufacturing process, not all chips end up the same, even if we do everything to make them the same. In practice that means there may be defects on the chip, or maybe not all units perform up to spec. This is the so-called “silicon lottery”. The chips are then sorted according to how well they ended up performing and that’s when the manufacturer decides what to do with them and what product they can be sold as.

There really are no bad GPUs, just incorrectly priced GPUs. Those that are still usable but ended up less than ideal will still be put to use: some parts (eg. compute units) are fused off and disabled, and the GPU is overall still functional as a weaker, cheaper GPU. That’s how we end up with products like the Radeon HD 7870 XT.

Starting point for investigating the problem

Initial testing

To start with this work, open source enthusiast Leonardo Frassetto helped me to buy a used 7870 XT in good condition from an Italian used hardware site. After plugging in this GPU and booting my system, I noticed the following:

  • Firmware (BIOS) can recognize the GPU, show a logo and boot grub
  • When amdgpu loads, the picture disappears and becomes a colorful mess
  • Looking at the logs, it seems the GFX block immediately hangs and goes into a GPU reset loop, as the kernel attempts to reset the GPU to fix the hang, which is expected

Information from Wikipedia

From Wikipedia, we got a list of AMD GPUs and an article about GCN to give us some basic info.

We can see that Tahiti LE is a harvested (cut down) version of the full Tahiti chip as the 7870 XT has lower specs than the 7970 (or R9 280X). There are plenty of GPUs with disabled CUs supported already, so I went with the assumption that the disabled CUs aren’t the issue.

Tahiti (top spec) Tahiti LE (cut down)
32 CU (compute units) 24 CU
32 RB (render output units) 32 RB
384-bit memory bus 256-bit memory bus

Information from Freedesktop Bugzilla

Someone opened a bug report on the old Freedesktop Bugzilla in 2013 complaining that the 7870 XT didn’t work. Although the issue has never been solved, we can still glean some interesting information from that bug report:

  • The display (ie. modesetting) should work, and the issue is “only” with 3D acceleration.
  • One commenter was able to get the 7870 XT to work with basic compute shaders but not much else (definitely not a full desktop).
  • The register dumps attached to the bug only contain the DCE (display engine) registers, so are not really conclusive.
  • There were suggestions to change the CGTS_TCC_DISABLE register in the kernel driver, which didn’t help.
  • The developers already corrected the register programming for harvested (cut down) RBs, and the 7870 XT doesn’t have those anyway, so that isn’t the issue.

Booting a working system

How do we even begin to diagnose what the problem is if the GPU hangs immediately at boot?

Booting in runlevel 3

I started by booting to runlevel 3 (basically just a terminal and nothing else). In this mode, the amdgpu kernel driver can initialize all blocks in the GPU correctly and the display works. We are in a terminal only environment so there is nothing submitting jobs to the GPU.

Booting a desktop with software rendering

I configured the system to use software rendering for both OpenGL and Vulkan by setting the following environment variables temporarily:

# Ask OpenGL loader for software rendering
LIBGL_ALWAYS_SOFTWARE=1
# Force using lavapipe as a Vulkan driver
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.x86_64.json
# Force using swrast as a Gallium driver (for OpenGL)
MESA_LOADER_DRIVER_OVERRIDE=swrast

With that, the system can indeed boot with the 7870 XT, although the graphical user interface is slow because we are not using any hardware acceleration (obviously). I would not recommend anyone to regularly use their system this way.

Testing simple shaders

Even though all the system uses software rendering, we can still set different environment variables for specific apps and have just one test application run with “real” drivers for the hardware. I decided to use the vkrunner suite and wrote a very simple test case with a very simple compute shader. The test case allocates two buffers (SSBO). The shader has only one invocation that reads a small piece of data from the values_in buffer and writes it to the values_out buffer.

[compute shader]
#version 450

layout(local_size_x = 1, local_size_y = 1, local_size_z = 1) in;
layout(binding = 1) buffer block_out { uint values_out[]; };
layout(binding = 2) buffer block_in { uint values_in[]; };

void main()
{
  values_out[gl_LocalInvocationIndex] = gl_WorkGroupID.x;
}

[test]
ssbo 1 1048576
ssbo 2 1048576

compute 1 1 1

That indeed works correctly, confirming what we found in the Freedesktop bugzilla, that simple compute shaders indeed work. What happens if we slightly complicate it though? The first thing we can do is to increase the number of invocations so that more than one invocation reads and writes memory.

layout(local_size_x = 96, local_size_y = 1, local_size_z = 1) in;

Whoops! This test case now hangs the GPU. I guess the way we changed the memory access must have caused the GPU to hang. With some trial and error, we can find that local_size_x = 64 still works but local_size_x = 65 and higher hangs. After some further trial and error I noticed that I can still use a larger workgroup if I skip memory accesses to certain address ranges. This definitely confirms that the issue is with memory access somehow.

Let’s look at registers with umr

One of the developers who commented on the old bugzilla thread were mentioning the CGTS_TCC_DISABLE register. Although the suggested patch didn’t help, I thought okay, let’s see what is actually the value of this register using umr:

$ sudo umr -O bits -r tahiti.gfx600.mmCGTS_TCC_DISABLE
gfx600.mmCGTS_TCC_DISABLE => 0x09240001
        .TCC_DISABLE[16:31] == 2340 (0x00000924)

Each bit represents a disabled TCC. We can see that some TCC units are disabled by default: 2, 5, 8, 11. Why are they disabled and what does that mean exactly? We saw from Wikipedia that the 7870 XT only has a 256-bit memory bus, although other Tahiti based GPUs have a 384-bit bus. This means that only two-thirds of the memory channels on this GPU are enabled and the rest are disabled.

I started to think, is it possible the GPU is trying to read from those disabled memory channels or use some disabled TCC units, or something like that? Technically, the disabled parts should be fused off and not accessible, but maybe it still tries to interact with them?

I went to take a look at the registers and register fields of GFX 6 to see if there is anything that stands out. I basically searched for “TCC” and looked at the results hoping for a clue. I found two interesting things:

  • The TCP_ADDR_CONFIG register has a field called NUM_TCC_BANKS
  • There are TCP_CHAN_STEER_LO and TCP_CHAN_STEER_HI registers which have fields for channels

Let’s see the value of these registers:

$ sudo umr -O bits -r tahiti.gfx600.mmTCP_ADDR_CONFIG
gfx600.mmTCP_ADDR_CONFIG => 0x000002fb
        .COLHI_WIDTH[6:8]    ==  3 (0x00000003)
        .NUM_BANKS[4:5]      ==  3 (0x00000003)
        .NUM_TCC_BANKS[0:3]  == 11 (0x0000000b)
        .RB_SPLIT_COLHI[9:9] ==  1 (0x00000001)

$ sudo umr -O bits -r tahiti.gfx600.mmTCP_CHAN_STEER_LO
gfx600.mmTCP_CHAN_STEER_LO => 0xa9210876
        .CHAN0[0:3]   ==  6 (0x00000006)
        .CHAN1[4:7]   ==  7 (0x00000007)
        .CHAN2[8:11]  ==  8 (0x00000008)
        .CHAN3[12:15] ==  0 (0x00000000)
        .CHAN4[16:19] ==  1 (0x00000001)
        .CHAN5[20:23] ==  2 (0x00000002)
        .CHAN6[24:27] ==  9 (0x00000009)
        .CHAN7[28:31] == 10 (0x0000000a)

$ sudo umr -O bits -r tahiti.gfx600.mmTCP_CHAN_STEER_HI
gfx600.mmTCP_CHAN_STEER_HI => 0x0000543b
        .CHAN8[0:3]   == 11 (0x0000000b)
        .CHAN9[4:7]   ==  3 (0x00000003)
        .CHANA[8:11]  ==  4 (0x00000004)
        .CHANB[12:15] ==  5 (0x00000005)
        .CHANC[16:19] ==  0 (0x00000000)
        .CHAND[20:23] ==  0 (0x00000000)
        .CHANE[24:27] ==  0 (0x00000000)
        .CHANF[28:31] ==  0 (0x00000000)

Whoah!!! It seems like the registers indeed refer to the disabled memory channels:

  • TCP_ADDR_CONFIG.NUM_TCC_BANKS indicates 12 channels (value of 11) even though the GPU only has 8 active memory channels.
  • TCP_CHAN_STEER_LO/HI refer to channels 2, 5, 8, 11 even though those memory channels are disabled.

Let’s try our luck and see what happens if we remove those. We simply need to edit the bit pattern for these registers, I typically use Gnome Calculator in programming mode to toggle the bits by hand.

# Change NUM_TCC_BANKS to 7
$ sudo umr -w tahiti.gfx600.mmTCP_ADDR_CONFIG 0x000002f7
# Use channels 0, 1, 3, 4, 6, 7, 9, 10
$ sudo umr -w tahiti.gfx600.mmTCP_CHAN_STEER_LO 0x43a91076
# We don't have any more channels so clear this to zeroes
$ sudo umr -w tahiti.gfx600.mmTCP_CHAN_STEER_HI 0x00000000

And… Yesss!!! with that, the test doesn’t hang anymore, and the GPU starts working well.

What’s actually happening on this GPU?

I had a chat with the lead developer of amdgpu Alex Deucher and shared my findings. He helped me understand what’s what:

  • TCC (texture cache per channel) is the L2 cache that is attached to each memory channel. For disabled memory channels, the corresponding TCC should be also disabled and should not be used (obviously).
  • TCP (texture cache per pipe) is the L1 cache that is part of each CU.
  • The TCP_ADDR_CONFIG and TCP_CHAN_STEER registers can tell the TCPs which TCCs they are allowed to use. These registers were programmed from the “golden” registers without regard to the fact that some channels may be disabled.

We can now see that the GPU was trying to use the L2 cache that is attached to the disabled memory channels, which (obviously) didn’t work and caused the GPU to hang. With that understanding, I wrote a patch. With that patch, the system can boot normally (without a need to force software rendering) and the 7870 XT works fine now.

The patch has been backported to stable kernels too. If you are using a 7870 XT, you can now enjoy it fully working on Linux!

Perf testing

I also did some perf testing to see if it makes any difference how it is configured. I used Rise of the Tomb Raider at 1080p with lowest settings for these tests. This shows that the values mentioned above are indeed optimal for this GPU, so we only need to skip the disabled TCCs but otherwise can keep the order they are in.

TCP_CHAN_STEER_HI TCP_CHAN_STEER_LO NUM_TCC_BANKS frame rate
0 0x43a91076 8 78 fps
0 0 12 27 fps
0xa147 0x43a91076 12 69.96 fps
0x4310 0x43a91076 12 61 fps
0 0xa9764310 8 66 fps

What have we learned from this?

It turns out this is one of those rare bugs which only affect a specific variant of a chip and nothing else. The issue with harvested TCCs is a problem unique to Tahiti LE, which was only ever used in the 7870 XT (and the FirePro D500 according to Wikipedia), so the fix doesn’t affect other GPUs. There have been no more GCN GPUs with this kind of harvesting (although RDNA1 has variants with harvested TCC as well).

On the bright side, it wasn’t a useless exercise for me.

  • I learned a lot about how the cache hierarchy works
  • It was interesting to see how the cache configuration affects performance
  • I feel proud that I was able to solve the mystery