?

Log in

pointless pontifications of a back seat driver [entries|archive|friends|userinfo]
jwboyer

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

4.4.y kernel and XFS. Sometimes being Fast and First is hard [Mar. 4th, 2016|02:51 pm]
jwboyer
[Tags|]

TLDR: Update xfsprogs to xfsprogs-4.3.0-1 before you update the kernel to 4.4.4 on Fedora 22/23.

Every once in a while we get a bug report that sounds really really bad. So bad that it makes us do a double take and start thinking about it in detail even before we complete the normal daily bug review. Today was one of those days.

This morning I read a bug report that suggested the 4.4 kernel will not boot with an xfsprogs in userspace less than version 4.3.0. That sounds pretty bad. Fedora 23 has already been rebased to the 4.4.y stable series of kernels, and it is in update-testing for Fedora 22. Given that the default filesystem in Server edition is XFS, that definitely caused some concern.

However, thinking about it even for a moment it seemed somewhat odd. First, typically the only userspace component involved in mounting an XFS filesystem is mount(8). That isn't provided by the xfsprogs package. Even if that is what was broken, one would think that there would have been bug reports long before now given that Fedora 23 was rebased a while ago already. Perhaps fsck might come into play too, and that's a possibility but it isn't very likely. So it was time to dig in further. The report linked to an upstream discussion. Here is where things started to get clearer. I'll try and explain a bit.

Around the 3.15 kernel release, XFS added support for maintaining CRC checksums for on-disk metadata objects. According to the mkfs.xfs manpage:


CRCs enable enhanced error detection due to hardware issues, whilst the format changes also improves crash recovery algorithms and the ability of various tools to validate and repair metadata corruptions when they are found. The CRC algorithm used is CRC32c, so the overhead is dependent on CPU archi‐ tecture as some CPUs have hardware acceleration of this algorithm. Typically the overhead of calculat‐ ing and checking the CRCs is not noticable in normal operation.


Yay! More robustness and better error recovery. Fantastic. However, features like this are rarely enabled by default when they are first released. In fact, the addition of this necessitated bumping the version number of the XFS on-disk format to version 5. For further caution, xfsprogs didn't start supporting this until xfsprogs-3.2.0 and didn't make it the default for mkfs.xfs until xfsprogs-3.2.3. Pretty cautious on the part of the XFS developers. But that still leaves us trying to figure out why the report was submitted and what it means for Fedora.

First, the actual problem. In the 4.4 kernel release the XFS developers started validating some additional metadata of V5 filesystems against the log. This is fine for healthy filesystems. However, if you had a crash and ran xfs_repair from an older xfsprogs package, it would unconditionally zero out the log and the kernel would be very confused and report corruption when you tried to mount it. The xfsprogs-4.3 release fixed this in the xfs_repair utility. OK, so there's validity to the 4.4 kernel needing xfsprogs-4.3, but only in certain circumstances. Namely, you have to have a V5 filesystem on disk, and you have to run xfs_repair from an older xfsprogs against it.

Fortunately, that isn't a super common situation. For Fedora 22, we released with xfsprogs-3.2.2. That means any new XFS filesystems created from installation media should still be V4 and not hit this issue (prior EOL Fedora releases are in the same boat). It's possible someone manually specified crc=1 when they created their F22 XFS partition, but that is a rare case. Fedora 23, on the other hand, shipped with xfsprogs-3.2.4 and should be creating V5 XFS filesystems by default. Those users would still need to run xfs_repair, which isn't a massively common thing to do. Unfortunately, it is common enough we need to do something about it.

The first thing to do was get xfsprogs updated in Fedora 22 and 23 (24 and rawhide are fine already). Eric Sandeen, being the massively awesome human being he is, had that done before I had even fully understood the problem. He was, in fact, so fast that he had the bodhi updates file before we even talked about what to do in the kernel package. That isn't ideal, but it's workable.

The second thing to do was to make sure the kernel package took this into account. Originally we discussed adding a Requires on the appropriate xfsprogs version. That actually isn't a great idea though. The kernel package doesn't actually Require any of the filesystem creation tools packages and it shouldn't. Users don't want to be forced to drag around xfsprogs if they aren't even using XFS filesystems, or btrfs-progs if they aren't using btrfs, etc. So we quickly realized that we needed to use Conflicts.

For the 4.4.4 kernel updates in Fedora 22 and Fedora 23, we added the Conflicts for xfsprogs < 4.3.0-1. You may see output that looks similar to this when you dnf update:

Error: package kernel-core-4.4.4-301.fc23.x86_64 conflicts with xfsprogs < 4.3.0-1 provided by xfsprogs-3.2.4-1.fc23.x86_64.

The solution here is to make sure you update xfsprogs first. Then your kernel update should work fine.

We'd like to thank the original bug reporter and Eric Sandeen for their help.

[Edit: Added TLDR for people that don't like to read and/or hate my writing]
linkpost comment

Understanding the kernel release cycle [Dec. 2nd, 2015|09:14 pm]
jwboyer
[Tags|]

A few days ago, a Fedora community member was asking if there was a 4.3 kernel for F23 yet (there isn't). When pressed for why, it turns out they were asking for someone that wanted newer support but thought a 4.4-rc3 kernel was too new. This was surprising to me. The assumption was being made that 4.4-rc3 was unstable or dangerous, but that 4.3 was just fine even though Fedora hasn't updated the release branches with it yet. This led me to ponder the upstream kernel release cycle a bit, and I thought I would offer some insights as to why the version number might not represent what most people think it does.

First, I will start by saying that the upstream kernel development process is amazing. The rate of change for the 4.3 kernel was around 8 patches per hour, by 1600 developers, for a total of 12,131 changes over 63 days total[1]. And that is considered a fairly calm release by kernel standards. The fact that the community continues to churn out kernels of such quality with that rate of change is very very impressive. There is actually quite a bit of background coordination that goes on between the subsystem maintainers, but I'm going to focus on how Linus' releases are formed for the sake of simplicity for now.

A kernel release is broken into a set of discrete, roughly time-based chunks. The first chunk is the 2 week merge window. This is the timeframe where the subsystem maintainers send the majority of the new changes for that release to Linus. He takes them in via git pull requests, grumbles about a fair number of them, refuses a few others. Most of the pull requests are dealt with in the first week, but there are always a few late ones so Linus waits the two weeks and then "closes" the window. This culminates in the first RC release being cut.

From that point on, the focus for the release is fixes. New code being taken at this point is fairly rare, but does happen in the early -rc releases. These are cut roughly every Sunday evening, making for a one week timeframe per -rc. Each -rc release tends to be smaller in new changesets than the previous, as the community becomes much more picky on what is acceptable the longer the release has gone on. Typically it gets to -rc7, but occasionally it will go to -rc8. One week after -rc7 is released, the "final" release is cut, which maps nicely with the 63 day timeframe quoted above.

Now, here is where people start getting confused. They see a "final" release and immediately assume it's stable. It's not. There are bugs. Lots and lots of bugs. So why would Linus release a kernel with a lot of bugs? Because finding them all is an economy of scale. Let's step back a second into the development and see why.

During the development cycle, people are constantly testing things. However, not everyone is testing the same thing. Each subsystem maintainer is often testing their own git tree for their specific subsystem. At the same time, they've opened their subsystem trees for changes for the next version of the kernel, not the one still in -rcX state. So they have new code coming in before the current code is even released. This is how they sustain that massive rate of change.

Aside from subsystem trees, there is the linux-next tree. This is daily merge of all the subsystem maintainer trees that have already opened up to new code on top of whatever Linus has in his tree. A number of people are continually testing linux-next, mostly through automated bots but also in VMs and running fuzzers and such. In theory and in practice, this catches bugs before they get to Linus the next round. But it is complicated because the rate of change means that if an issue is hit, it's hard to see if it's in the new new code only found in linux-next, or if it's actually in Linus' tree. Determining that usually winds up being a manual process via git-bisect, but sometimes the testing bots can determine the offending commit in an automated fashion.

If a bug is found, the subsystem maintainer or patch author or whomever must track which tree the bug is, whether it's a small enough fix to go into whatever -rcX state Linus' tree is in, and how to get it there. This is very much a manual process, and often involves multiple humans. Given that humans are terrible at multitasking in general, and grow ever more cautious the later the -rcX state is, sometimes fixes are missed or simply queued for the next merge window. That's not to say important bugs are not fixed, because clearly there are several weeks of -rcX releases where fixing is the primary concern. However, with all the moving parts, you're never going to find all the bugs in time.

In addition to the rate of change/forest of trees issue, there's also the diversity and size of the tester pool. Most of the bots test via VMs. VMs are wonderful tools, but they don't test the majority of the drivers in the kernel. The kernel developers themselves tend to have higher end laptops. Traditionally this was of the Thinkpad variety and a fair number of those are still seen, but there is some variance here now which is good. But it isn't good enough to cover all possible firmware, device, memory, and workload combinations. There are other testers to be sure, but they only cover a tiny fraction of the end user machines.

It isn't hard to see how bugs slip through, particularly in drivers or on previous generation hardware. I wouldn't even call it a problem really. No software project is going to cut a release with 0 bugs in it. It simply doesn't happen. The kernel is actually fairly high quality at release time in spite of this. However, as I said earlier, people tend to make assumptions and think it's good enough for daily use on whatever hardware they have. Then they're surprised when it might not be.

To combat this problem, we have the upstream stable trees. These trees backport fixes from the current development kernels that also apply to the already released kernels. Hence, 4.3.1 is Linus' 4.3 release, plus a number of fixes that were found "too late". This, in my opinion, is where the bulk of the work on making a kernel usable happens. It is also somewhat surprising when you look at it.

The first stable release of a kernel, a .1 release, is actually very large. It is often comprised of 100-200 individual changes that are being backported from the current development kernel. That means there are 100-200 bugs immediately being fixed there. Whew, that's a lot but OK maybe expected with everything above taken into account. Except the .2 release is often also 100-200 patches. And .3. And often .4. It isn't until you start getting into .5, .6, .7, etc that the patch count starts getting smaller. By the .9 release, it's usually time to retire the whole tree (unless it's a long-term stable) and start the fun all over again.

In dealing with the Fedora kernel, the maintainers take all of this into account. This is why it is very rare to see us push a 4.x.0 kernel to a stable release, and often it isn't until .2 that you see a build. For those thinking that this article is somehow deriding the upstream kernel development process, I hope you now realize the opposite is true. We rely heavily on upstream following through and tagging and fixing the issues it finds, either while under development or via the stable process. We help the stable process as well by reporting back fixes if they aren't already found.

So hopefully next time you're itching for a new kernel just because it's been released upstream, you'll pause and think about this. And if you really want to help, you'll grab a rawhide kernel and pitch in by reporting any issues when you find them. The only way to get the stable kernel releases smaller, and reduce the number of bugs still found in freshly released kernels, is to broaden the tester pool and let the upstream developers know as soon as possible. In this way, we're all part of the upstream kernel community and we can all keep making it awesome and impressive.

(4.3.y will likely be coming to F23 the first week of January. Greg-KH seems to have gone on some kind of walkabout the past few weeks so 4.3.1 hasn't been released yet. To be honest, it's a break well deserved. Or maybe he found 4.3 to be even more buggy as usual. Who knows!)

[1] https://lwn.net/Articles/661978/ (Of course I was going to link to lwn.net. If you aren't already subscribed to it, you really should be. They have amazing articles and technical content. They make my stuff look like junk even more than it already is. I'm kinda jealous at the energy and expertise they show in their writing.)
linkpost comment

Fedora kernel exploded tree part deux: Snakes and assumptions [Oct. 6th, 2015|01:13 pm]
jwboyer
[Tags|]

A while back I wrote about some efforts to move to using an exploded source tree for the Fedora kernel. As that post details, it wasn't the greatest experience. However, I still think an exploded tree has good utility and I didn't want to give up on the idea of it existing. So after scraping our "switch" I decided to (slowly) work on a tool that would create such a tree automatically. In the spirit of release-early and pray nobody dies from reading your terrible code, we now have fedkernel. The readme file in the repo contains a high level overview of how the tool works, so I won't duplicate that here. Instead I thought I would talk about some of the process changes and decisions we made to make this possible.

Git all the things


One of the positive fallouts of the previous efforts was that all of the patches we carried in Fedora were nicely formatted with changelogs and authorship information. Being literally the output of git-format-patch instantly improved the patch quality. When it came time to figure out how to generate the patches from pkg-git to apply to the exploded tree, I really wanted to keep that quality. So I thought about how to accomplish this and then I realized there was no need to reinvent the wheel. The git-am and git-format-patch tools existed and were exactly what I wanted.

After discussing things with the rest of the team, we switched to using git-am to apply patches in the Fedora kernel spec. The mechanics of this are pretty simple: the spec unpacks the tarball (plus any -rcX patches) and uses this as the "base" commit. Stable update patches are applied as a separate commit on top of the base if it is a stable kernel. Then it walks through every patch and applies it with git-am. This essentially enforces our patch format guidelines for us. It does have the somewhat negative side effect of slowing down the %prep section quite a bit, but in practice it hasn't been slow enough to be a pain. (Doing a git add and git commit on the full kernel sources isn't exactly speedy, even on an SSD.)

So after %prep is done, the user is left with a git tree in the working directory that has all the Fedora patches as separate commits. "But wait, isn't the job done then?", you might ask. Well, no. We could call it good enough, but that isn't really what I or other users of an exploded tree were after. I wanted a tree with the full upstream commit history plus our patches. What this produces is just a blob, plus our patches. Not quite there yet but getting closer.

Snakes


This is where fedkernel comes in. I needed tooling that could take the patches from this franken-tree and apply them to a real exploded git tree. My previous scripts were written in bash, and I could have done this in bash again but I wanted to make it automated and I wanted it to talk to the Fedora infrastructure. This means it was time to learn python again. Fortunately, the upstream python community has great documentation and there are modules for pretty much anything I needed. This makes my "bash keyboard in interactive python session" approach to the language pretty manageable and I was able to make decent progress.

To get the patches out of the prepped sources, I needed to know mainly one thing. What was the actual upstream base for this build? That is easy enough to figure out if you can parse certain macros in kernel.spec. The one part that proved to be somewhat difficult was for git snapshot kernels. We name these -gitY kernels, where X increases until the next -rcX release. E.g. kernel-4.3.0-0.rc3.git1.1, kernel-4.3.0-0.rc3.git2.1, etc. That's great for RPMs, but the only place we actually documented what upstream commit we generated the snapshot from was in an RPM %changelog comment.

Parsing it out of there is possible, but it's a bit cumbersome and it is somewhat error prone. The sha1sum is always recorded, but it isn't guaranteed to be the newest changelog. Other patches and changelogs can be added before the kernel is actually built. Fortunately, we use a script to generate these snapshots. To make it trivial to figure out the sha1sum, I modified the script to record the full commit has to a file called gitrev in pkg-git. Now fedkernel can easily read that file and use it as the base upstream revision to apply patches on top of. Yay for cheating and/or being lazy.

The rest of the code deals with prepping the tree, using the git python module to do manipulations and generate patches, and applying them to the other tree. Python actually made much of this very easy to do and again I'm really glad I used that instead of bash.

Assumptions



So now that we modified a few things in pkg-git to make this easier, the assumptions basically fall out to be:

- Patches in the prepped source can be retrieved via 'git format-patch'
- The upstream base revision is determinable from kernel.spec and the gitrev file.

Pretty simple, right? Yes. Except the code isn't complete by any means and it requires a bit more manual setup. Like having existing pkg-git and linux.git trees that it can modify which contain all the branches and proper remotes set up already. That isn't really a huge issue, but it does mean when f24 is branched and rawhide becomes f25, we'll need to do some updates. Perhaps before then we'll have fixed the code (or some nice person will submit a pull request that does so.)

I've been using the tool to generate exploded trees for the past week or so. It seems to be working well, and I've published them at https://git.kernel.org/cgit/linux/kernel/git/jwboyer/fedora.git/ once again. There is a history gap there as the tree fell into disrepair for a while, but it should be kept current going forward.

Even if the code is terrible and hacky, writing it was a good learning experience. I hope to keep refining it and improving things in the TODO over time. If you want to pitch in, patches are always welcome. You can email them to me, submit a pagure.io pull request, or mail the Fedora kernel list as usual.
linkpost comment

A word on Fedora meetings [Sep. 9th, 2015|07:01 pm]
jwboyer
[Tags|]

A few people have noted that when I chair a Fedora meeting, I seem to move quickly and remain strictly focused on the current set topic. I discourage broader discussion on semi-related topics and tend to finish meetings faster than most. There is a reason for this. It is because THAT IS HOW MEETINGS ARE SUPPOSED TO WORK.

There are many many articles about productivity and meetings. I am not an expert on this at all, but I do strictly follow my own set of rules for meetings derived partly from said articles but mostly from experience. They can be summed up as so:

1) The meeting better have an agenda. If it doesn't, I'm likely to not pay attention. The agenda should lend itself to having items decided upon and completed during said meeting. If there are hairy topics, the should be last. The vast majority of discussion of agenda items should have already taken place elsewhere, either on lists or in tickets.

2) The meeting should actually STICK to the agenda. A meeting is really not a place to bring up random or tangential topics for discussion. At most such things should be noted for further discussion and decision at future meetings, preferably during open floor.

3) The chair of the meeting is responsible for keeping people on topic and completing the meeting in a timely manner. Be polite, but firm.

4) If it is clear that a decision is not going to be reached on an item during the meeting, defer it for further discussion elsewhere.


That's it.

Meetings should be focused with clear goals. Fedora meetings taking place on IRC necessitates some amount of leeway because of the medium, but that does not mean meetings exist for the sake of meetings or for chat time. IRC is also a terrible place to have in-depth discussions on things, as people feel time pressured and the format of the dialogue can be difficult to follow.

So yes, my FESCo meetings (or any other that I chair) tend to be shorter than most. This is simply because I want the meeting to be productive and I don't want to waste people's time. And if you feel that I am wrong on several points here, that is totally fine with me. Just volunteer to chair the meeting next time ;).
linkpost comment

LPC 2015 Day 2 [Aug. 24th, 2015|03:10 pm]
jwboyer
I started day 2 of LPC with the Graphics, mode setting, and Wayland microconference. This was an extremely dense microconference with lots of information on what is missing in various areas and what has been in the works for some time. To be honest, I felt out of my depth several times. However, it was very clear that the participants in the session were making good progress. One side-note: nobody likes to use mics :).

The afternoon session was heavy in hallway track for me. I had some conversations with Josef Bacik around btrfs and Fedora (summary: not yet). I also spoke with him and another engineer from Facebook around how they handle the kernel across their various machines. It was an interesting high level look at what a company is doing at that scale.

I also touched base with Matthew Garrett on the secure boot patchset. This has been something that we've carried in Fedora since around Fedora 18. It hasn't really changed much at all, but the patches have failed to get upstreamed for a number of trivial reasons. There are a few follow-ups that need to be looked into as well though. Matthew plans on submitting them upstream again and I'm going to see what I can do to help them get merged. Hopefully the third (fourth?) time is a charm.

The remainder of the afternoon was spent catching up with some old colleagues and meeting a few new people from various companies. Conversation was wide ranging. We touched on technical topics as well as completely irrelevant things. However, I think the time was very well spent as one of the major purposes of conferences is to meet up with people face to face. Without that interaction, it is too easy to resolve a person to nothing more than an email address and that never works out well.

The evening event was held at Rock Bottom Brewery and it was delicious. We had some impromptu Fedora kernel team time over dinner and shared dessert. Once all the ice cream and brownie was gone, I turned in shockingly early. Apparently the jet lag and two late nights in a row were starting to wear on me.
linkpost comment

LPC 2015 Day 1 [Aug. 21st, 2015|12:00 pm]
jwboyer
I have the privilege of attending the Linux Plumbers Conference in Seattle this week. This is by far my favorite conference. The talks and microconferences are all of high quality and the events are very well organized.

The first day is shared with LinuxCon, which lends itself to some talks that span both audiences. The first talk I attended with "Everything is a file descriptor" by Josh Triplett. Josh gave a great overview of why file descriptors are a great mechanism and some of the more recent system calls that have been added to give userspace the ability to write less crappy code. Things like timerfd, and signalfd allow userspace to avoid some of the awkwardness that comes with using the more traditional interfaces that UNIX has provided. He also described work on clonefd, which is a new system call to allow userspace to get a file descriptor tied to a task in the kernel. This will have some interested benefits, such as being able to pass the fd to a process that is not in your process hierarchy and being able to poll for child/task exit without hanging in waitpid. At the end Josh threw out some possible future additions that haven't been implemented. Overall a very well done talk.

Following that, I sat in on Steven Rostedt's talk on the RT patchset. I've seen this talk or a version of it around three times and I'm always highly entertained. Steven likes to pack a ton of information in his talks. The overall success of the RT patchset has been very good, and they are down to some of the final really hard bits. Some of them they actually need help with to figure out the best solution in the subsystem in question (like the VFS). One of the questions from the audience was about lessons learned working on the patchset, the issues they've fixed in mainline because of it, etc. Steven said that is a great idea for another talk, so I hope he writes that up. I would love to hear it.

THe afternoon session started off with an overview of ACPI and where some of the ACPI 6 features are coming into play. Things like low power idle and other PM related features look to be headed towards us in hardware, and ACPI is of course being adapted to work with this. Overall a good overview.

Following that I did a bit of hallway track and then went to Daniel Vetter's talk on screwing up (or how to not) ioctls. Working in the DRM layer has provided him with lots of experience over the past few years on how to properly design your code to make it easier to use. One of the main points that was stressed several times was having testcases for everything. This is fairly obvious, but he pointed out that the testcases are better written when you aren't looking at the code. If you just look at the data structures and create boundary cases based on that, with all kinds of crazy input, you will often catch cases that the code itself doesn't cover. He had a good amount to say and it was entertaining.

The traditional kernel panel was the wrap up for the day, but I skipped that to catch up on some work. Then it was off to the evening event at the Experience Music Project museum. This venue was amazing and had a variety of exhibits ranging from a guitar collection to Star Wars costumes. It was very cool and the food was excellent. After spending perhaps a bit more than I expected in the gift shop, it was back to the hotel to try and sleep off some of the weariness that comes from travel delays and cramming information into your head all day.
linkpost comment

Failing Fast [Jul. 20th, 2015|10:48 am]
jwboyer
[Tags|]

Users of the Fedora exploded kernel git tree might have noticed that it has been stale for a couple of weeks now. What they might not know is why that is, and when it is going to be fixed. The answer is somewhat complicated and I'll try and summarize here.

The Fedora kernel team recently tried shifting to using the exploded tree as the canonical location for Fedora kernel work. The benefits and ideas were written here, and most of those still stand. So I went to work on some scripts that would make this easier to do. The results weren't terrible. Things worked, kernels were still built, and the exploded tree was spit out (albeit at a different location). By some measures, this was a success.

Except it really wasn't. Some of the motivation behind the change was to increase participation and transparency around the Fedora kernel. However, while we worked through the new process we quickly realized that it was much more cumbersome to actually produce kernel packages. Since the primary output of our team is packages, that seemed like a pretty bad side effect. Similarly, transparency wasn't decreased but the change made things much more confusing. Changes in the exploded tree weren't synced 1:1 back to the Fedora package repo, so it appeared like big code drops there instead of discrete commits. Telling people to look at the exploded tree was an option, but in Fedora's package-centric world it was a bit out of place.

So what do you do when you've spent time making a change and it isn't working out? Well, you scrap it. There's little point in pushing on with something that even the primary author finds to be awkward. The payoffs weren't likely to materialize in any sort of timeframe that would make it worth the continued effort there. Was the time wasted? In my opinion, no. It answered some questions we've had for a while, and showed us that both the tooling and processes Fedora uses aren't really amenable to working with exploded sources.

Unfortunately, the exploded tree has been stagnant since we changed back. However, I don't think it will remain that way forever. We tweaked some of the things we do to build the kernel package such that taking that content and creating an exploded tree will be easier. With a bit more scripting, it should be possible to almost automate the creation on every successful build. I'll be working on that off and on for the next few weeks. Hopefully by Flock I have something cobbled together to get things back on track again.

It's been a while since I've had what I would consider a pretty big failure. I make mistakes all the time like everyone else, but those are generally small and on short-term things. In a discussion with someone, they asked me if I was bothered by this being a big bigger. I'm not. Personally, I don't care if I fail spectacularly or not as long as I learn something from it. I think I did, so the exercise was worthwhile to me. Failing fast is a really good way to work through some complicated scenarios, and is far better than staying stagnant out of fear of failure. Time to move on to the next idea!

(A couple of notes:)

It was pointed out on the list that RHEL tooling can build from an exploded tree, but it seems it does this with some odd hacks that we'd likely try and avoid in Fedora. The workflow between RHEL and Fedora kernels are also massively different. I would love to see some more commonality between the two, but not at the expense of forcing ill fitting process on either one.

Pagure, is where the exploded tree was hosted in the interim. We tried it with the idea that making multiple committers there would be easier than kernel.org. It is a neat service, but I'm not sure we'd really use many of the features it was designed around. The only complaint I heard was that browsing via the web interface was slow.
linkpost comment

Fedora 22 and Kernel 4.0 [Apr. 16th, 2015|10:36 am]
jwboyer
[Tags|]

As Ryan noted yesterday, Fedora 22 is on track to ship with the 4.0 kernel release. So what does that mean in the grand scheme of things? In short, not much.

The major version change wasn't done because of any major feature or change in process or really anything exciting at all. Linus Torvalds changed it because he felt the minor version number was getting a bit large and he liked 4.0 better. It was really a whim more than any thing contained within the kernel itself. The initial merge window builds of this kernel in Fedora were even called 3.20.0-rc0.gitX until the 4.0-rc1 release came out.

In fact, this kernel release is one of the more "boring" releases in a while. It has all the normal fixes and improvements and new hardware support one would normally expect, but overall it was a fairly quiet release. So version number aside, this is really more of the same from our beloved kernel development community.

However, there is one feature that people (and various media sites) seem to have keyed in on and that is the live patching functionality. This holds the promise of being able to patch your running kernel without rebooting. Indeed, this could be very useful, but it isn't quite there yet. And it also doesn't make a whole lot of sense for Fedora at this time. The Fedora kernels have this functionality disabled, both in Fedora 22 and Rawhide.

What was merged for 4.0 is the core functionality that is shared between two live patching projects, kPatch and kGraft. kPatch is being led by a development team in Red Hat whereas kGraft is being developed by a team from SuSE. They both accomplish the same end result, but they do so via a different approach internally. The two teams met at the Linux Plumbers conference last year and worked on some common ground to make it easier to merge into mainline rather than compete with each other. This is absolutely awesome and an example of how new features should be developed upstream. Kudos to all those involved on that front.

The in-kernel code can accept patches from both methods, but the process and tools to create those patches are still being worked on in their upstream communities. Neither set are in Fedora itself, and likely won't be for some time as it is still fairly early in the life of these projects. After discussing this a bit with the live patching maintainer, we decided to keep this disabled in the Fedora kernels for now. The kernel-playground COPR does have it enabled for those that want to dig in and generate their own patches and are willing to break things and support themselves.

In reality, we might not ever really leverage the live patching functionality in Fedora itself. It is understandable that people want to patch their kernel without rebooting, but the mechanism is mostly targeted at small bugfixes and security patches. You cannot, for example, live patch from version 4.0 to 4.1. Given that the Fedora kernel rebases both from stable kernel (e.g. 3.19.2 to 3.19.3) and major release kernels over the lifetime of a Fedora release, we don't have much opportunity to build the live patches. Our update shipping infrastructure also isn't really geared towards quick, small fixes. Really, the only viable target for this functionality in Fedora is likely on the oldest Fedora release towards the end of its lifecycle and even then it's questionable whether it would be worth the effort. So I won't say that Fedora will never have a live patch enabled kernel, but there is a lot of work and process stuff to be figured out before that ever really becomes an option.

So that's the story around the 4.0 kernel. On the one hand, it sounds pretty boring and is likely to disappoint those hoping for some amazing new thing. On the other hand, it's a great example of the upstream kernel process just chugging along and delivering pretty stable quality kernels. As a kernel maintainer, I like this quite a bit. If you have any questions about the 4.0 kernel, or really any Fedora kernel topics, feel free to email us at the Fedora kernel list. We're always happy to discuss things there.
link1 comment|post comment

Fedora kernel position [Jan. 27th, 2015|10:22 am]
jwboyer
[Tags|]

As you might have seen Paul blog about, Red Hat has an immediate opening for a Fedora kernel maintainer position on my team. This is actually a fairly rare thing, as we don't have a lot of churn in our department and most of the engineering positions we hire for are primarily RHEL roles. If you have kernel experience and love working on fast-paced and frequently updated kernels, then this might be a good role for you.

The job writeup is accurate in terms of what we expect, but it is also kind of broad. That is primarily because the role is too. Yesterday davej wrote a bit about how working on a Fedora kernel is like getting a 10,000ft view of everything. It's actually a really good analogy, and Dave would know as he did it longer than anyone. We deal with a lot of varied issues, on an even more varied set of hardware. This isn't a traditional development job. Being curious and willing to learn is key to enjoying a distro kernel maintainer role.

That being said, we're also looking at ways to make a bigger impact both upstream and in Fedora itself. Filling this position is a key part of that and I'm excited to see how it plays out. If you're interested in it, please don't hesitate to send me questions via email or on IRC. Also be sure to apply via the online job posting here:

http://jobs.redhat.com/jobs/descriptions/fedora-kernel-engineer-westford-massachusetts-job-1-5076703
linkpost comment

32-bit is the zombie of kernels. It needs your help. [Jan. 20th, 2015|09:49 am]
jwboyer
[Tags|]

TLDR: Use 32-bit x86 kernels? Want to keep using them? Want to make sure they continue to work? Please help!

My kids recently convinced me to get them Minecraft. This in turn has caused lots of discussions about Minecraft. In an effort to be able to have anything close to resembling a coherent conversation with them, I've been playing the game myself a little bit. If you've been living under a rock like me, you might not know anything about it. I'll spare you all the never-ending details, but there is one part that recently got me thinking about Fedora and 32-bit and kernels.

See, in the game there are zombies. They're not particularly dangerous by themselves. They're slow, and they kind of moan and come after you but you can usually deal with them without really any effort. They only come out at night, and if you catch them outside at sunrise they burst into flames. Unless there's a large group of them, you basically ignore them.

In Fedora, there are x86 machines running 32-bit kernels. They're not particularly dangerous by themselves. They're slow, and they kind of stumble around a lot. If you shine a light on the dim corners of the kernel dealing with that code, it usually bursts into flames. Most upstream developers ignore them. Clearly they're a lot like Minecraft zombies, except there are always lots of them and they are never, ever the same.

This makes dealing with 32-bit kernels in Fedora actually fairly difficult. With upstream focusing almost entirely on x86_64, there isn't a massive amount of interest in solving 32-bit x86 kernel problems. That isn't to say that huge issues won't be fixed, but they are clearly not a priority (fortunately, they are also rare). There are other cases where the standard advice is to not use 32-bit kernels for things. Have more than 2GiB of DRAM? Don't use 32-bit kernels because PAE is a hack. Want to run VMs? Don't use 32-bit kernels. Transparent hugepages (or hugepages in general)? 32-bit is not your friend.

Then there is the variety of workloads people are using 32-bit kernels for. Some of them are old laptops that have crusty ACPI implementations. Some of the are 32-bit servers that are running constantly and stressing various things. Crazily enough, some people even run 32-bit kernels on 64-bit CPUs. That last one is a pet peeve of mine, but I won't dive into that here. The ISA variety is a headache as well, with some CPUs not supporting PAE so that we have to build different kernels for i686 and PAE-capable i686 machines.

When you take the above, add in the bug backlog we get from the just as widely varied x86_64 machines, the fact that our 32-bit hardware is rather limited for testing, the 32-bit x86 kernels in Fedora are simply pretty low on the priority list. We build them, we make sure we grab any fixes we see or are pointed to for them, but in the larger picture the time we spend on 32-bit specific issues isn't benefiting the majority of Fedora users. So the kernels linger on.

Not surprisingly, I'm not the first person to notice this. Just today I've had 2 discussions on what to do about i686 in Fedora, and Smooge posted his idea for a way forward. Others have had similar ideas. RHEL 7 does not include 32-bit kernels at all. I'm not going to comment on those proposals yet, but it does seem to at least confirm a bit of what we see on the kernel side of things.

So what can be done here? Should we kill the 32-bit x86 kernels? Should we kill one of them? Do we spend time on a solution I previously thought about a long time ago? I don't have answers for all of that at the moment. However, in listening to a very detailed dissertation on Minecraft zombie solutions from my son, his last solution was "or you could just cure the ones that are villagers". Apparently some of the zombies in the game can be cured. Some of them, the ones that are legitimately useful otherwise, can be saved.

Can we accomplish that in Fedora for 32-bit x86 kernels? There are most certainly sane uses of 32-bit, albeit on a reduced scale overall. So in the face of all the other challenges we have in dealing with this, I'm asking for help. We're asking for help. The kernel team has asked for help before, but it is understandably daunting for us to say "Hey! The whole kernel could use help! Should be fun!" This is a call to action on a much more limited scale. So if you use x86 32-bit kernels, and you want to see them better supported then speak up. Send us an email on the Fedora kernel list, dig through bugzilla for 32-bit kernel issues, find us on IRC.

Who knows, with a little community help and elbow grease, we could get some of these issues resolved. We could cure some of the kernel zombies. The alternative is the status quo, where we're waiting for the proverbial sun to rise.
link7 comments|post comment

navigation
[ viewing | most recent entries ]
[ go | earlier ]