You are viewing jwboyer

pointless pontifications of a back seat driver - Fedora and Ubuntu Kernel Config Comparison [entries|archive|friends|userinfo]
jwboyer

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

Fedora and Ubuntu Kernel Config Comparison [May. 9th, 2013|02:29 pm]
Previous Entry Add to Memories Share Next Entry
[Tags|]

Every once in a while, I crawl out from under the rock that is bugzilla and I try and look around at what others are doing in the distro kernel space. Today I was curious how Fedora and Ubuntu compare in how they configure the kernel. I've long thought that for all the focus the kernel gets, it should be the most boring package in an entire distro. It should work, work well, and that is about it. It isn't there to differentiate your distro. It's there to let your distro run. So, will my personal belief stand up, or would I find something in the configs that proves one "distro" is better than another? Let's dive in.

I won't give a line by line comparison because 1) that would be very long and 2) that would be utterly boring. You don't need to know that e.g. Ubuntu enables one scsi driver that Fedora doesn't or vice versa. Instead, I'll try to focus on various functional areas and give an overview.

The kernels


For comparison purposes, I used the config from linux-image-3.8.0-19-generic Ubuntu 13.04 package and the 3.8.11-200.fc18 kernel from Fedora. These are the same base upstream kernel version, and both of them are from the latest stable release of their respective distros. I used the x86_64 (amd64 in Ubuntu speak) build, as I found that to be the most relevant in this day and age. I could have used the kernels under development, however those are more apt to change and I thought it would be better to compare things out there in wider use.

Overview


The vast majority of the config is pretty much identical. It's what you would expect from a distribution config. There is a wide range of support for all kinds of devices and buses, with much of it being provided by modules instead of built in. This should be expected, given that they're coming from the same upstream code base and really there is only one sane option for most config options.

Low-level options


A few of the low-level options did differ though. One of the more notable ones is that Ubuntu has CONFIG_NR_CPUS set to 256, whereas Fedora has it set to 128. On the face of it, one might think both values are overkill as people aren't booting on machines with 128 cores very often. However, Ubuntu has transitioned to having a single kernel between it's desktop and server variants (starting with 12.04 I think), so it does make some sense for them to have it set higher. Fedora can revisit our choice if we get a sudden clamoring for massive machine support. I await my mini-supercomputer test system.

Another area where we differed was in the NUMA options. Ubuntu enables CONFIG_X86_NUMACHIP, CONFIG_MOVABLE_NODE, CONFIG_MEMORY_HOTPLUG, and CONFIG_MEMORY_HOT_REMOVE. The CONFIG_NODES_SHIFT value was higher on Ubuntu as well, allowing for more NUMA nodes. Again, given their server variant that probably makes sense for them. Fedora just doesn't deal with that class of machine on a regular basis.

The HZ area also had some differences. Fedora defaults to HZ_1000 compared to Ubuntu's HZ_250. That means the timer tick is 1000 times per second on Fedora, versus 250 on Ubuntu. Typically a higher HZ value is used for better interactive response. On a server system where there isn't typically a desktop or user waiting for interaction, a smaller HZ setting allows more work to get done as there are few timer interrupts to process. Of course, both distros set CONFIG_NO_HZ, which means the timer interrupts are only triggered on an as-needed basis when the CPU is idle. I would expect that both will eventually also go with the new completely tickless options that were introduced in the 3.10 kernel merge window after they have matured a bit.

Ubuntu also enabled CONFIG_FAST_NO_HZ whereas Fedora doesn't. That option lets CPUs with pending RCU callbacks enter the dynamic tick state (idle) by extending the grace period. That slows down synchronize_rcu calls, but by letting the CPUs enter idle state more it improves energy efficiency. Sounds like a great thing, and it really probably is. Fedora disabled this after having it enabled when it was first introduced and hitting numerous issues with it. The option has been upstream for a few kernel releases now, so it's probably something we should revisit.

Default choices


For the most part, the default choices were the same. A couple of the areas where they diverged were in the default I/O scheduler, and the default CPU frequency governor. Ubuntu uses the deadline scheduler by default compared to Fedora's CFQ (Completely Fair Queueing). Ubuntu uses the performance governor instead of the ondemand governor that Fedora uses. To be clear, all of the options are provided by both distros. They just differ in the default setting.

LSM


The security stack is probably one of the larger areas of difference between Ubuntu and Fedora. Fedora is solidly an SELinux camp. It is the only LSM we enable and is therefore naturally the default. Ubuntu on the other hand enables all of them, including SELinux. I was kind of surprised by this, as I expected they would only enable their default LSM which is AppArmor. I wouldn't argue that this makes Ubuntu more secure than Fedora by any means, but it does give users a greater choice in what they use for MAC/DAC security.

Module options


Fedora and Ubuntu both produce signed modules. I was surprised Ubuntu was using this given their statement of it not being needed for Secure Boot purposes. So perhaps they're using it for some other purpose. Looking at their choice of using CONFIG_MODULE_SIG_SHA512 that might well be the case. Fedora uses CONFIG_MODULE_SIG_SHA256 because that is the common hash implementation used in UEFI.

Ubuntu also enables CONFIG_MODVERSIONS. This generates a CRC value for each exported symbol prototype. When a module is loaded, it compares the CRC with that in the kernel and fails the load if they do not match. It is a simple form of ABI check for modules. They also enable MODULE_SRCVERSION_ALL, which embeds a "srcversion" field. Arguably, this lets people build modules against different kernels and then load them while knowing where the module came from.

The old, the obscure, and the crappy


Ubuntu seems to have a tendency to enable a lot more old stuff than Fedora. They have support for acorn, atari, ultrix, and sysv68 partitions. They build DECNET and Parallel IDE support. ARCNET. Things one would not normally expect to find, much less want to use in this day and age. Of course, Fedora has some weird stuff too. We still enable the OSS sound stack and Ubuntu appears to have ditched this entirely.

They also seem to have a lot of "embedded" focused options enabled. Things like full MTD support, all of the random keyboard drivers, SPI, MFD, Regulator, and GPIOLIB. They have JFFS2, F2FS, and VXFS filesystem support. They enable various SoC options for things like sound and media drivers. This seems a bit at odds with their desktop and server focus, but if one wanted to boot an Ubuntu kernel on some random x86 based SoC thing they might be able to.

Fedora also tends to only enable the main DRM drivers that support KMS and very few framebuffer drivers. Ubuntu enables everything almost. Personally, I'm fine with our KMS or go away approach. Oddly, Ubuntu doesn't enable CONFIG_LOGO. WHY DO YOU HATE THE PENGUINS?

Lastly, Ubuntu enables way more staging drivers than Fedora. If you aren't aware, staging is where drivers come in to get cleaned up before being officially mainlined. They literally taint the kernel as TAINT_CRAP, and we tend to stay away from them. Ubuntu is also carrying AUFS, the union fileystem and ALX, an ethernet driver for new Atheros chips that has had a pretty horrible review history upstream.

Summary


So did my personal theory of "kernels should work and that's it" hold? I think it did mostly. There are no shocking differences between the two distribution kernels. The few areas where we differ that might matter are worth reviewing on both sides, but at the end of the day none of those options are going to make any kind of massive improvement. I consider this to be a worthwhile exercise, but one I won't repeat for a while. I can only stare at kernel configs for so long before it makes even bugzilla start looking interesting.

If you've stuck with me thus far, thanks!
linkReply

Comments:
From: (Anonymous)
2013-05-09 08:54 pm (UTC)

(Link)

Are you sure Ubuntu is using the performance governor by default ? It seems strange.
From: jwboyer
2013-05-09 09:52 pm (UTC)

(Link)

[jwboyer@zod kernel]$ grep CPU_FREQ_DEFAULT_GOV_PERFORMANCE ~/config-3.8.0-19-generic
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
[jwboyer@zod kernel]$

Yep. I thought it was strange as well.
From: (Anonymous)
2013-05-10 01:38 am (UTC)

(Link)

It gets changed to ondemand later in the boot process
From: (Anonymous)
2013-05-10 08:59 am (UTC)

(Link)

This is indeed odd. Are they really running like that or do they change it using some startup script or whatever?
From: (Anonymous)
2013-05-10 10:31 am (UTC)

(Link)

IIRC, Ubuntu uses performance governor by default during boot, but switches to ondemand governor in /etc/init.d/ondemand. I think the logic behind this was to speed up the booting process, but I could be mistaken.
From: (Anonymous)
2013-05-09 09:58 pm (UTC)

(Link)

The last time I checked, one of the differences that that Fedora optimizes the Kernel for size while Ubuntu uses O2. I was hoping you would talk about it and give us insightful comments :)
From: jwboyer
2013-05-10 12:57 am (UTC)

(Link)

Neither distribution enables CONFIG_CC_OPTIMIZE_FOR_SIZE. Fedora hasn't done so since late October of 2011. I'm not sure if Ubuntu ever did.
From: (Anonymous)
2013-05-09 11:43 pm (UTC)

(Link)

Arch Linux Kernel disable CONFIG_LOGO, they argument that this logo is buggy in KMS and Nvidia videos without patching
probably Ubuntu catch same bugs as Arch and descide make the same elction
From: jwboyer
2013-05-10 12:57 am (UTC)

(Link)

Ah, that may be the case. Thanks for the insight.
[User Picture]From: Jerez Spa
2013-05-10 12:18 am (UTC)

(Link)

Well it would be great if there is a comparision of fedora with Arch/Debian kernel config.
From: jwboyer
2013-05-10 12:58 am (UTC)

(Link)

Yes, and possibly OpenSuse. Perhaps in the future I'll get to those.
From: (Anonymous)
2013-05-13 03:42 am (UTC)

(Link)

I don't honestly think Ubuntu has as much capability to tweak kernels as much as Debian. Debian has dedicated guys that look over kernel changes. So a Fedora to Debian might be better to look at. I would guess that Debian has the older kernel.
From: (Anonymous)
2013-05-16 08:21 am (UTC)

(Link)

That doesn't make sense as Ubuntu has a dedicated kernel team which works on the distro only, and has customer focused kernel developers who work with specific hardware vendors. The most likely difference would be that Debian enables for a much wider variety of hardware and architectures.
From: (Anonymous)
2013-05-10 02:16 am (UTC)

(Link)

Well of late I have seen a few benchmarks that in general show Ubuntu perform
a little better than Fedora on same hardware on out of the box stock kernels ...
Fedora in general has a newer updated kernel and should bring better
performance as such .. so what is the reason for this .. and no these are not some odd regressions
From: jwboyer
2013-05-10 11:08 am (UTC)

(Link)

I can't really give reasons for something I've never seen. If you have links to these benchmark comparisons, that might be something we can eventually look at.
From: (Anonymous)
2013-05-10 05:33 pm (UTC)

(Link)

in my personal experience, I too noticed fedora runs slow on same machine.
From: jwboyer
2013-05-10 05:38 pm (UTC)

(Link)

I would really encourage anyone seeing performance differences between Fedora and another distro to document the details and steps to recreate. If given the same machine, the same workload, and the same base kernel version, there is a gap between one distro or another it's at least worth looking into. We can't begin to guess at where those issues might be though.

So write a blog post and send a link, or provide details directly on the Fedora kernel list. I can't promise we'll fix it immediately, but we can try to look at it as time allows.
From: (Anonymous)
2013-05-10 03:28 am (UTC)

(Link)

Who cares about the kernel config, that's just one variable. You should compare patches because Fedora tends to have hundreds of them (though to be fair most of them do go upstream).
From: jwboyer
2013-05-10 11:13 am (UTC)

(Link)

Fedora doesn't have hundreds of patches.

Rawhide currently has ~35 patches on top of the 3.10 merge window kernel. A few of those are already headed upstream.

The F19 kernel has 46 patches on top of 3.9.1, some of which are backported from 3.10 or linux-next. The rest are simple changes that we just carry for things like tainting on vbox module load, or silencing a few known irritating kernel warnings.

The one remaining large patch Fedora carries is the Secure Boot patchset. Outside of that, any patch we add is usually already in an upstream kernel maintainer's tree somewhere.
From: (Anonymous)
2013-05-10 12:45 pm (UTC)

(Link)

It would be interesting to compare the Ubuntu kernel config to the Debian.

Debian has disabled the Boot Penguin for so long that I thought it was a Knoppix-specific thing the first time I saw it.
From: (Anonymous)
2013-05-10 01:08 pm (UTC)

(Link)

Lol, I thought it was a Knoppix-specific thing until I read your comment!
[User Picture]From: infinity
2013-05-20 08:10 am (UTC)

(Link)

As to Ubuntu building a ton of crap/old drivers, the general theory when Ubuntu config review rolls around is "if it can build, and isn't known to blow up hardware, and enabling it doesn't magically disable something else in the tree, let it build 'cause someone, somewhere might be using it."

Another point to mention is that we homogenize configs between arches as much as we can so, while MTD support and jffs2 (for instance) may seem like uncommon options for x86 kernels, they're significantly more common on ARM kernels, so all our configs pick that up in the non-platform-specific options. Not that it hurts to have such support on x86 (and it has its uses), but it's certainly more commonly used on ARM devices.

Rules are, of course, more strict for evaluating what gets built in statically (and this does differ between architectures as well), but for modules, meh, it's just buildd time and a slightly larger package to include a .ko that some user somewhere may want, even if we think it's pointless.

The ondemand/performance thing was discussed above, but I wanted to confirm that, yes, we only default to performance specifically to avoid ondemand ramp-up during boot (which is actually measurable), and then ~60s into userspace, we switch to ondemand (or the Android-specific 'interactive' governor, if the running kernel supports that instead).

In general, though, it's nice to see that, really, we're not all that much different and I honestly didn't expect us to be, but who has the patience to do the comparison? Well, apparently you. You must have been seriously bored. ;)