My kids recently convinced me to get them Minecraft. This in turn has caused lots of discussions about Minecraft. In an effort to be able to have anything close to resembling a coherent conversation with them, I've been playing the game myself a little bit. If you've been living under a rock like me, you might not know anything about it. I'll spare you all the never-ending details, but there is one part that recently got me thinking about Fedora and 32-bit and kernels.
See, in the game there are zombies. They're not particularly dangerous by themselves. They're slow, and they kind of moan and come after you but you can usually deal with them without really any effort. They only come out at night, and if you catch them outside at sunrise they burst into flames. Unless there's a large group of them, you basically ignore them.
In Fedora, there are x86 machines running 32-bit kernels. They're not particularly dangerous by themselves. They're slow, and they kind of stumble around a lot. If you shine a light on the dim corners of the kernel dealing with that code, it usually bursts into flames. Most upstream developers ignore them. Clearly they're a lot like Minecraft zombies, except there are always lots of them and they are never, ever the same.
This makes dealing with 32-bit kernels in Fedora actually fairly difficult. With upstream focusing almost entirely on x86_64, there isn't a massive amount of interest in solving 32-bit x86 kernel problems. That isn't to say that huge issues won't be fixed, but they are clearly not a priority (fortunately, they are also rare). There are other cases where the standard advice is to not use 32-bit kernels for things. Have more than 2GiB of DRAM? Don't use 32-bit kernels because PAE is a hack. Want to run VMs? Don't use 32-bit kernels. Transparent hugepages (or hugepages in general)? 32-bit is not your friend.
Then there is the variety of workloads people are using 32-bit kernels for. Some of them are old laptops that have crusty ACPI implementations. Some of the are 32-bit servers that are running constantly and stressing various things. Crazily enough, some people even run 32-bit kernels on 64-bit CPUs. That last one is a pet peeve of mine, but I won't dive into that here. The ISA variety is a headache as well, with some CPUs not supporting PAE so that we have to build different kernels for i686 and PAE-capable i686 machines.
When you take the above, add in the bug backlog we get from the just as widely varied x86_64 machines, the fact that our 32-bit hardware is rather limited for testing, the 32-bit x86 kernels in Fedora are simply pretty low on the priority list. We build them, we make sure we grab any fixes we see or are pointed to for them, but in the larger picture the time we spend on 32-bit specific issues isn't benefiting the majority of Fedora users. So the kernels linger on.
Not surprisingly, I'm not the first person to notice this. Just today I've had 2 discussions on what to do about i686 in Fedora, and Smooge posted his idea for a way forward. Others have had similar ideas. RHEL 7 does not include 32-bit kernels at all. I'm not going to comment on those proposals yet, but it does seem to at least confirm a bit of what we see on the kernel side of things.
So what can be done here? Should we kill the 32-bit x86 kernels? Should we kill one of them? Do we spend time on a solution I previously thought about a long time ago? I don't have answers for all of that at the moment. However, in listening to a very detailed dissertation on Minecraft zombie solutions from my son, his last solution was "or you could just cure the ones that are villagers". Apparently some of the zombies in the game can be cured. Some of them, the ones that are legitimately useful otherwise, can be saved.
Can we accomplish that in Fedora for 32-bit x86 kernels? There are most certainly sane uses of 32-bit, albeit on a reduced scale overall. So in the face of all the other challenges we have in dealing with this, I'm asking for help. We're asking for help. The kernel team has asked for help before, but it is understandably daunting for us to say "Hey! The whole kernel could use help! Should be fun!" This is a call to action on a much more limited scale. So if you use x86 32-bit kernels, and you want to see them better supported then speak up. Send us an email on the Fedora kernel list, dig through bugzilla for 32-bit kernel issues, find us on IRC.
Who knows, with a little community help and elbow grease, we could get some of these issues resolved. We could cure some of the kernel zombies. The alternative is the status quo, where we're waiting for the proverbial sun to rise.