Log in

pointless pontifications of a back seat driver [entries|archive|friends|userinfo]

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

Fedora kernel exploded tree part deux: Snakes and assumptions [Oct. 6th, 2015|01:13 pm]

A while back I wrote about some efforts to move to using an exploded source tree for the Fedora kernel. As that post details, it wasn't the greatest experience. However, I still think an exploded tree has good utility and I didn't want to give up on the idea of it existing. So after scraping our "switch" I decided to (slowly) work on a tool that would create such a tree automatically. In the spirit of release-early and pray nobody dies from reading your terrible code, we now have fedkernel. The readme file in the repo contains a high level overview of how the tool works, so I won't duplicate that here. Instead I thought I would talk about some of the process changes and decisions we made to make this possible.

Git all the things

One of the positive fallouts of the previous efforts was that all of the patches we carried in Fedora were nicely formatted with changelogs and authorship information. Being literally the output of git-format-patch instantly improved the patch quality. When it came time to figure out how to generate the patches from pkg-git to apply to the exploded tree, I really wanted to keep that quality. So I thought about how to accomplish this and then I realized there was no need to reinvent the wheel. The git-am and git-format-patch tools existed and were exactly what I wanted.

After discussing things with the rest of the team, we switched to using git-am to apply patches in the Fedora kernel spec. The mechanics of this are pretty simple: the spec unpacks the tarball (plus any -rcX patches) and uses this as the "base" commit. Stable update patches are applied as a separate commit on top of the base if it is a stable kernel. Then it walks through every patch and applies it with git-am. This essentially enforces our patch format guidelines for us. It does have the somewhat negative side effect of slowing down the %prep section quite a bit, but in practice it hasn't been slow enough to be a pain. (Doing a git add and git commit on the full kernel sources isn't exactly speedy, even on an SSD.)

So after %prep is done, the user is left with a git tree in the working directory that has all the Fedora patches as separate commits. "But wait, isn't the job done then?", you might ask. Well, no. We could call it good enough, but that isn't really what I or other users of an exploded tree were after. I wanted a tree with the full upstream commit history plus our patches. What this produces is just a blob, plus our patches. Not quite there yet but getting closer.


This is where fedkernel comes in. I needed tooling that could take the patches from this franken-tree and apply them to a real exploded git tree. My previous scripts were written in bash, and I could have done this in bash again but I wanted to make it automated and I wanted it to talk to the Fedora infrastructure. This means it was time to learn python again. Fortunately, the upstream python community has great documentation and there are modules for pretty much anything I needed. This makes my "bash keyboard in interactive python session" approach to the language pretty manageable and I was able to make decent progress.

To get the patches out of the prepped sources, I needed to know mainly one thing. What was the actual upstream base for this build? That is easy enough to figure out if you can parse certain macros in kernel.spec. The one part that proved to be somewhat difficult was for git snapshot kernels. We name these -gitY kernels, where X increases until the next -rcX release. E.g. kernel-4.3.0-0.rc3.git1.1, kernel-4.3.0-0.rc3.git2.1, etc. That's great for RPMs, but the only place we actually documented what upstream commit we generated the snapshot from was in an RPM %changelog comment.

Parsing it out of there is possible, but it's a bit cumbersome and it is somewhat error prone. The sha1sum is always recorded, but it isn't guaranteed to be the newest changelog. Other patches and changelogs can be added before the kernel is actually built. Fortunately, we use a script to generate these snapshots. To make it trivial to figure out the sha1sum, I modified the script to record the full commit has to a file called gitrev in pkg-git. Now fedkernel can easily read that file and use it as the base upstream revision to apply patches on top of. Yay for cheating and/or being lazy.

The rest of the code deals with prepping the tree, using the git python module to do manipulations and generate patches, and applying them to the other tree. Python actually made much of this very easy to do and again I'm really glad I used that instead of bash.


So now that we modified a few things in pkg-git to make this easier, the assumptions basically fall out to be:

- Patches in the prepped source can be retrieved via 'git format-patch'
- The upstream base revision is determinable from kernel.spec and the gitrev file.

Pretty simple, right? Yes. Except the code isn't complete by any means and it requires a bit more manual setup. Like having existing pkg-git and linux.git trees that it can modify which contain all the branches and proper remotes set up already. That isn't really a huge issue, but it does mean when f24 is branched and rawhide becomes f25, we'll need to do some updates. Perhaps before then we'll have fixed the code (or some nice person will submit a pull request that does so.)

I've been using the tool to generate exploded trees for the past week or so. It seems to be working well, and I've published them at https://git.kernel.org/cgit/linux/kernel/git/jwboyer/fedora.git/ once again. There is a history gap there as the tree fell into disrepair for a while, but it should be kept current going forward.

Even if the code is terrible and hacky, writing it was a good learning experience. I hope to keep refining it and improving things in the TODO over time. If you want to pitch in, patches are always welcome. You can email them to me, submit a pagure.io pull request, or mail the Fedora kernel list as usual.
linkpost comment

A word on Fedora meetings [Sep. 9th, 2015|07:01 pm]

A few people have noted that when I chair a Fedora meeting, I seem to move quickly and remain strictly focused on the current set topic. I discourage broader discussion on semi-related topics and tend to finish meetings faster than most. There is a reason for this. It is because THAT IS HOW MEETINGS ARE SUPPOSED TO WORK.

There are many many articles about productivity and meetings. I am not an expert on this at all, but I do strictly follow my own set of rules for meetings derived partly from said articles but mostly from experience. They can be summed up as so:

1) The meeting better have an agenda. If it doesn't, I'm likely to not pay attention. The agenda should lend itself to having items decided upon and completed during said meeting. If there are hairy topics, the should be last. The vast majority of discussion of agenda items should have already taken place elsewhere, either on lists or in tickets.

2) The meeting should actually STICK to the agenda. A meeting is really not a place to bring up random or tangential topics for discussion. At most such things should be noted for further discussion and decision at future meetings, preferably during open floor.

3) The chair of the meeting is responsible for keeping people on topic and completing the meeting in a timely manner. Be polite, but firm.

4) If it is clear that a decision is not going to be reached on an item during the meeting, defer it for further discussion elsewhere.

That's it.

Meetings should be focused with clear goals. Fedora meetings taking place on IRC necessitates some amount of leeway because of the medium, but that does not mean meetings exist for the sake of meetings or for chat time. IRC is also a terrible place to have in-depth discussions on things, as people feel time pressured and the format of the dialogue can be difficult to follow.

So yes, my FESCo meetings (or any other that I chair) tend to be shorter than most. This is simply because I want the meeting to be productive and I don't want to waste people's time. And if you feel that I am wrong on several points here, that is totally fine with me. Just volunteer to chair the meeting next time ;).
linkpost comment

LPC 2015 Day 2 [Aug. 24th, 2015|03:10 pm]
I started day 2 of LPC with the Graphics, mode setting, and Wayland microconference. This was an extremely dense microconference with lots of information on what is missing in various areas and what has been in the works for some time. To be honest, I felt out of my depth several times. However, it was very clear that the participants in the session were making good progress. One side-note: nobody likes to use mics :).

The afternoon session was heavy in hallway track for me. I had some conversations with Josef Bacik around btrfs and Fedora (summary: not yet). I also spoke with him and another engineer from Facebook around how they handle the kernel across their various machines. It was an interesting high level look at what a company is doing at that scale.

I also touched base with Matthew Garrett on the secure boot patchset. This has been something that we've carried in Fedora since around Fedora 18. It hasn't really changed much at all, but the patches have failed to get upstreamed for a number of trivial reasons. There are a few follow-ups that need to be looked into as well though. Matthew plans on submitting them upstream again and I'm going to see what I can do to help them get merged. Hopefully the third (fourth?) time is a charm.

The remainder of the afternoon was spent catching up with some old colleagues and meeting a few new people from various companies. Conversation was wide ranging. We touched on technical topics as well as completely irrelevant things. However, I think the time was very well spent as one of the major purposes of conferences is to meet up with people face to face. Without that interaction, it is too easy to resolve a person to nothing more than an email address and that never works out well.

The evening event was held at Rock Bottom Brewery and it was delicious. We had some impromptu Fedora kernel team time over dinner and shared dessert. Once all the ice cream and brownie was gone, I turned in shockingly early. Apparently the jet lag and two late nights in a row were starting to wear on me.
linkpost comment

LPC 2015 Day 1 [Aug. 21st, 2015|12:00 pm]
I have the privilege of attending the Linux Plumbers Conference in Seattle this week. This is by far my favorite conference. The talks and microconferences are all of high quality and the events are very well organized.

The first day is shared with LinuxCon, which lends itself to some talks that span both audiences. The first talk I attended with "Everything is a file descriptor" by Josh Triplett. Josh gave a great overview of why file descriptors are a great mechanism and some of the more recent system calls that have been added to give userspace the ability to write less crappy code. Things like timerfd, and signalfd allow userspace to avoid some of the awkwardness that comes with using the more traditional interfaces that UNIX has provided. He also described work on clonefd, which is a new system call to allow userspace to get a file descriptor tied to a task in the kernel. This will have some interested benefits, such as being able to pass the fd to a process that is not in your process hierarchy and being able to poll for child/task exit without hanging in waitpid. At the end Josh threw out some possible future additions that haven't been implemented. Overall a very well done talk.

Following that, I sat in on Steven Rostedt's talk on the RT patchset. I've seen this talk or a version of it around three times and I'm always highly entertained. Steven likes to pack a ton of information in his talks. The overall success of the RT patchset has been very good, and they are down to some of the final really hard bits. Some of them they actually need help with to figure out the best solution in the subsystem in question (like the VFS). One of the questions from the audience was about lessons learned working on the patchset, the issues they've fixed in mainline because of it, etc. Steven said that is a great idea for another talk, so I hope he writes that up. I would love to hear it.

THe afternoon session started off with an overview of ACPI and where some of the ACPI 6 features are coming into play. Things like low power idle and other PM related features look to be headed towards us in hardware, and ACPI is of course being adapted to work with this. Overall a good overview.

Following that I did a bit of hallway track and then went to Daniel Vetter's talk on screwing up (or how to not) ioctls. Working in the DRM layer has provided him with lots of experience over the past few years on how to properly design your code to make it easier to use. One of the main points that was stressed several times was having testcases for everything. This is fairly obvious, but he pointed out that the testcases are better written when you aren't looking at the code. If you just look at the data structures and create boundary cases based on that, with all kinds of crazy input, you will often catch cases that the code itself doesn't cover. He had a good amount to say and it was entertaining.

The traditional kernel panel was the wrap up for the day, but I skipped that to catch up on some work. Then it was off to the evening event at the Experience Music Project museum. This venue was amazing and had a variety of exhibits ranging from a guitar collection to Star Wars costumes. It was very cool and the food was excellent. After spending perhaps a bit more than I expected in the gift shop, it was back to the hotel to try and sleep off some of the weariness that comes from travel delays and cramming information into your head all day.
linkpost comment

Failing Fast [Jul. 20th, 2015|10:48 am]

Users of the Fedora exploded kernel git tree might have noticed that it has been stale for a couple of weeks now. What they might not know is why that is, and when it is going to be fixed. The answer is somewhat complicated and I'll try and summarize here.

The Fedora kernel team recently tried shifting to using the exploded tree as the canonical location for Fedora kernel work. The benefits and ideas were written here, and most of those still stand. So I went to work on some scripts that would make this easier to do. The results weren't terrible. Things worked, kernels were still built, and the exploded tree was spit out (albeit at a different location). By some measures, this was a success.

Except it really wasn't. Some of the motivation behind the change was to increase participation and transparency around the Fedora kernel. However, while we worked through the new process we quickly realized that it was much more cumbersome to actually produce kernel packages. Since the primary output of our team is packages, that seemed like a pretty bad side effect. Similarly, transparency wasn't decreased but the change made things much more confusing. Changes in the exploded tree weren't synced 1:1 back to the Fedora package repo, so it appeared like big code drops there instead of discrete commits. Telling people to look at the exploded tree was an option, but in Fedora's package-centric world it was a bit out of place.

So what do you do when you've spent time making a change and it isn't working out? Well, you scrap it. There's little point in pushing on with something that even the primary author finds to be awkward. The payoffs weren't likely to materialize in any sort of timeframe that would make it worth the continued effort there. Was the time wasted? In my opinion, no. It answered some questions we've had for a while, and showed us that both the tooling and processes Fedora uses aren't really amenable to working with exploded sources.

Unfortunately, the exploded tree has been stagnant since we changed back. However, I don't think it will remain that way forever. We tweaked some of the things we do to build the kernel package such that taking that content and creating an exploded tree will be easier. With a bit more scripting, it should be possible to almost automate the creation on every successful build. I'll be working on that off and on for the next few weeks. Hopefully by Flock I have something cobbled together to get things back on track again.

It's been a while since I've had what I would consider a pretty big failure. I make mistakes all the time like everyone else, but those are generally small and on short-term things. In a discussion with someone, they asked me if I was bothered by this being a big bigger. I'm not. Personally, I don't care if I fail spectacularly or not as long as I learn something from it. I think I did, so the exercise was worthwhile to me. Failing fast is a really good way to work through some complicated scenarios, and is far better than staying stagnant out of fear of failure. Time to move on to the next idea!

(A couple of notes:)

It was pointed out on the list that RHEL tooling can build from an exploded tree, but it seems it does this with some odd hacks that we'd likely try and avoid in Fedora. The workflow between RHEL and Fedora kernels are also massively different. I would love to see some more commonality between the two, but not at the expense of forcing ill fitting process on either one.

Pagure, is where the exploded tree was hosted in the interim. We tried it with the idea that making multiple committers there would be easier than kernel.org. It is a neat service, but I'm not sure we'd really use many of the features it was designed around. The only complaint I heard was that browsing via the web interface was slow.
linkpost comment

Fedora 22 and Kernel 4.0 [Apr. 16th, 2015|10:36 am]

As Ryan noted yesterday, Fedora 22 is on track to ship with the 4.0 kernel release. So what does that mean in the grand scheme of things? In short, not much.

The major version change wasn't done because of any major feature or change in process or really anything exciting at all. Linus Torvalds changed it because he felt the minor version number was getting a bit large and he liked 4.0 better. It was really a whim more than any thing contained within the kernel itself. The initial merge window builds of this kernel in Fedora were even called 3.20.0-rc0.gitX until the 4.0-rc1 release came out.

In fact, this kernel release is one of the more "boring" releases in a while. It has all the normal fixes and improvements and new hardware support one would normally expect, but overall it was a fairly quiet release. So version number aside, this is really more of the same from our beloved kernel development community.

However, there is one feature that people (and various media sites) seem to have keyed in on and that is the live patching functionality. This holds the promise of being able to patch your running kernel without rebooting. Indeed, this could be very useful, but it isn't quite there yet. And it also doesn't make a whole lot of sense for Fedora at this time. The Fedora kernels have this functionality disabled, both in Fedora 22 and Rawhide.

What was merged for 4.0 is the core functionality that is shared between two live patching projects, kPatch and kGraft. kPatch is being led by a development team in Red Hat whereas kGraft is being developed by a team from SuSE. They both accomplish the same end result, but they do so via a different approach internally. The two teams met at the Linux Plumbers conference last year and worked on some common ground to make it easier to merge into mainline rather than compete with each other. This is absolutely awesome and an example of how new features should be developed upstream. Kudos to all those involved on that front.

The in-kernel code can accept patches from both methods, but the process and tools to create those patches are still being worked on in their upstream communities. Neither set are in Fedora itself, and likely won't be for some time as it is still fairly early in the life of these projects. After discussing this a bit with the live patching maintainer, we decided to keep this disabled in the Fedora kernels for now. The kernel-playground COPR does have it enabled for those that want to dig in and generate their own patches and are willing to break things and support themselves.

In reality, we might not ever really leverage the live patching functionality in Fedora itself. It is understandable that people want to patch their kernel without rebooting, but the mechanism is mostly targeted at small bugfixes and security patches. You cannot, for example, live patch from version 4.0 to 4.1. Given that the Fedora kernel rebases both from stable kernel (e.g. 3.19.2 to 3.19.3) and major release kernels over the lifetime of a Fedora release, we don't have much opportunity to build the live patches. Our update shipping infrastructure also isn't really geared towards quick, small fixes. Really, the only viable target for this functionality in Fedora is likely on the oldest Fedora release towards the end of its lifecycle and even then it's questionable whether it would be worth the effort. So I won't say that Fedora will never have a live patch enabled kernel, but there is a lot of work and process stuff to be figured out before that ever really becomes an option.

So that's the story around the 4.0 kernel. On the one hand, it sounds pretty boring and is likely to disappoint those hoping for some amazing new thing. On the other hand, it's a great example of the upstream kernel process just chugging along and delivering pretty stable quality kernels. As a kernel maintainer, I like this quite a bit. If you have any questions about the 4.0 kernel, or really any Fedora kernel topics, feel free to email us at the Fedora kernel list. We're always happy to discuss things there.
link1 comment|post comment

Fedora kernel position [Jan. 27th, 2015|10:22 am]

As you might have seen Paul blog about, Red Hat has an immediate opening for a Fedora kernel maintainer position on my team. This is actually a fairly rare thing, as we don't have a lot of churn in our department and most of the engineering positions we hire for are primarily RHEL roles. If you have kernel experience and love working on fast-paced and frequently updated kernels, then this might be a good role for you.

The job writeup is accurate in terms of what we expect, but it is also kind of broad. That is primarily because the role is too. Yesterday davej wrote a bit about how working on a Fedora kernel is like getting a 10,000ft view of everything. It's actually a really good analogy, and Dave would know as he did it longer than anyone. We deal with a lot of varied issues, on an even more varied set of hardware. This isn't a traditional development job. Being curious and willing to learn is key to enjoying a distro kernel maintainer role.

That being said, we're also looking at ways to make a bigger impact both upstream and in Fedora itself. Filling this position is a key part of that and I'm excited to see how it plays out. If you're interested in it, please don't hesitate to send me questions via email or on IRC. Also be sure to apply via the online job posting here:

linkpost comment

32-bit is the zombie of kernels. It needs your help. [Jan. 20th, 2015|09:49 am]

TLDR: Use 32-bit x86 kernels? Want to keep using them? Want to make sure they continue to work? Please help!

My kids recently convinced me to get them Minecraft. This in turn has caused lots of discussions about Minecraft. In an effort to be able to have anything close to resembling a coherent conversation with them, I've been playing the game myself a little bit. If you've been living under a rock like me, you might not know anything about it. I'll spare you all the never-ending details, but there is one part that recently got me thinking about Fedora and 32-bit and kernels.

See, in the game there are zombies. They're not particularly dangerous by themselves. They're slow, and they kind of moan and come after you but you can usually deal with them without really any effort. They only come out at night, and if you catch them outside at sunrise they burst into flames. Unless there's a large group of them, you basically ignore them.

In Fedora, there are x86 machines running 32-bit kernels. They're not particularly dangerous by themselves. They're slow, and they kind of stumble around a lot. If you shine a light on the dim corners of the kernel dealing with that code, it usually bursts into flames. Most upstream developers ignore them. Clearly they're a lot like Minecraft zombies, except there are always lots of them and they are never, ever the same.

This makes dealing with 32-bit kernels in Fedora actually fairly difficult. With upstream focusing almost entirely on x86_64, there isn't a massive amount of interest in solving 32-bit x86 kernel problems. That isn't to say that huge issues won't be fixed, but they are clearly not a priority (fortunately, they are also rare). There are other cases where the standard advice is to not use 32-bit kernels for things. Have more than 2GiB of DRAM? Don't use 32-bit kernels because PAE is a hack. Want to run VMs? Don't use 32-bit kernels. Transparent hugepages (or hugepages in general)? 32-bit is not your friend.

Then there is the variety of workloads people are using 32-bit kernels for. Some of them are old laptops that have crusty ACPI implementations. Some of the are 32-bit servers that are running constantly and stressing various things. Crazily enough, some people even run 32-bit kernels on 64-bit CPUs. That last one is a pet peeve of mine, but I won't dive into that here. The ISA variety is a headache as well, with some CPUs not supporting PAE so that we have to build different kernels for i686 and PAE-capable i686 machines.

When you take the above, add in the bug backlog we get from the just as widely varied x86_64 machines, the fact that our 32-bit hardware is rather limited for testing, the 32-bit x86 kernels in Fedora are simply pretty low on the priority list. We build them, we make sure we grab any fixes we see or are pointed to for them, but in the larger picture the time we spend on 32-bit specific issues isn't benefiting the majority of Fedora users. So the kernels linger on.

Not surprisingly, I'm not the first person to notice this. Just today I've had 2 discussions on what to do about i686 in Fedora, and Smooge posted his idea for a way forward. Others have had similar ideas. RHEL 7 does not include 32-bit kernels at all. I'm not going to comment on those proposals yet, but it does seem to at least confirm a bit of what we see on the kernel side of things.

So what can be done here? Should we kill the 32-bit x86 kernels? Should we kill one of them? Do we spend time on a solution I previously thought about a long time ago? I don't have answers for all of that at the moment. However, in listening to a very detailed dissertation on Minecraft zombie solutions from my son, his last solution was "or you could just cure the ones that are villagers". Apparently some of the zombies in the game can be cured. Some of them, the ones that are legitimately useful otherwise, can be saved.

Can we accomplish that in Fedora for 32-bit x86 kernels? There are most certainly sane uses of 32-bit, albeit on a reduced scale overall. So in the face of all the other challenges we have in dealing with this, I'm asking for help. We're asking for help. The kernel team has asked for help before, but it is understandably daunting for us to say "Hey! The whole kernel could use help! Should be fun!" This is a call to action on a much more limited scale. So if you use x86 32-bit kernels, and you want to see them better supported then speak up. Send us an email on the Fedora kernel list, dig through bugzilla for 32-bit kernel issues, find us on IRC.

Who knows, with a little community help and elbow grease, we could get some of these issues resolved. We could cure some of the kernel zombies. The alternative is the status quo, where we're waiting for the proverbial sun to rise.
link7 comments|post comment

It's not always technical [Oct. 9th, 2014|10:45 am]

Anyone that has been around Fedora long enough knows that FESCo is the technical steering committee. As Fedora committees go, I think it's one of the best. It works well, it remains on task, and it has typically been a good example of how to get things done in Fedora. But it's not infallible. Like everyone, sometimes even FESCo can be blinded by its own purpose. Yesterday was, in my opinion, one of those times.

We had a ticket on the agenda that really shouldn't have been there yet. A community member opened it and suggested that the FPC (Fedora Packaging Committee) wasn't functioning well and needed fixing. If you attended the meeting or read the logs, you probably saw me use words like "angry" and "absurd". However, it wasn't about the subject of the ticket itself. There's nothing wrong with someone opening a ticket on that topic and it's arguably under FESCo's responsibility. I was upset about how FESCo was (mis)handling it.

See, the ticket was opened about the FPC, but it didn't include any of the FPC members on CC. Then further proposals were made on how to fix the FPC, still without discussing them with the FPC. The proposals also suggested moving work to the Fedora Product working groups, and only one of those WGs was contacted. So by the time it got to the meeting, we had a proposal to fix another committee without bringing them into the discussion, and push work to other groups without talking to all of them to see if they even wanted to do this.

When I pointed this out in the meeting, the collective reaction was a shrug. It was very much (paraphrased) "oops, yeah we should have CC'd more people. That was bad. But we have a proposal so we're going to talk about it anyway". And that, again in my opinion, is not acceptable. When I asked another member privately what was going on, they said this was just normal FESCo. That I found flat out scary. FESCo doesn't govern from top down. It can't. We don't have actual resources to direct or money to pay people, so we very much rely on collaboration across the project.

Yes, there was a proposal. Yes, it was present in the ticket. Yes, even if we closed the ticket we'd likely have to copy a lot of it to a new ticket (or just reopen it), but the order of operations surrounding this was entirely backwards. It serves no purpose to have a technical discussion about something that hasn't been discussed with the stakeholders themselves. Even from technical point of view it's wrong because you don't have all the relevant information.

Some of this may stem from the fact that as Fedora has grown, our process isn't holding up. We have 5 more governing bodies now with the various Fedora.next working groups. We have more complicated technical challenges that have many moving parts and groups. With all of the additional layers, it's easy to forget someone. It's also easy to forget that things are going to be naturally slower when you have to pull more people into a discussion. But you can't forget to have that discussion in the first place.

In the end, FESCo decided to do the proper thing and ask the proposal submitter to start a broader discussion with the relevant parties. I'm glad that was the case, but it should have been a prerequisite before we even discussed it. I think we learned a lesson and we instituted a process change for tickets that should help with this going forward. That's good because I think we at times focus too much on the engineering side of things and ignore the fact that it impacts people. We just need to remember that in a technical committee things are not always technical.
linkpost comment

Fedora kernel git tree [Oct. 1st, 2014|06:48 pm]

As mentioned in my last post, a number of people over the years have asked about an exploded source git tree for the Fedora kernel. We've never done this, primarily because our build tool wouldn't be able to use it, and it didn't really have any perceived value for the maintainers. However, it's come up enough times that I thought I'd give it a shot. Creating it was easy enough, but whether it winds up being useful to anyone is something that you'll need to help with. This post will cover what it is, how it works, where it is, and possible future plans.

For the TLDR, see the bottom.

What is it: This is maybe the git tree you are looking for

If you've ever worked on packaging software with RPM (or apt or a number of other package managers), there are two basic ways to do it. You can use the upstream release tarballs and apply patches via a spec file, or you can use a snapshot of the upstream development repository as the "tarball". The former is how the Fedora kernel is packaged, and it presents a nice separation of upstream and the patches Fedora has added. However, that means that every time a new upstream kernel is released we wind up rebasing our patch set on top of it. When doing the initial creation of the git tree, I thought about a number of ways to handle this.

We could use feature branches for each patch/patchset and do a merge in the main branch. That would be how upstream development works for patchsets, but some of our patches are rather long-lived for various reasons. Doing merges like this over time would lead to a very tangled history and it wouldn't really provide much benefit to anyway.

Another way would be to start with a base version, add our patches, and then merge in the upstream kernel changes on top of that. This is essentially the opposite of the scenario above. It still results in a rather tangled history over time, and it actually sends the wrong message: this git tree isn't an upstream. It's supposed to be a downstream representation of what we've added/changed. We don't want people developing new kernel drivers/features against the Fedora git tree. We want them to develop against upstream.

Another factor in figuring this all out was our build system. As I mentioned, koji isn't setup to take random exploded source git trees as build input. It expects things from particular places and in particular ways, which is perfectly fine. To make this git tree a "source" for koji, we'd have to use it as a snapshot tree and not list out the patches and such in the spec file. There are distros that do this, and maybe that's suitable for them but I find it valuable to look at the SRPMs from a distro and be able to easily see what's changed [sideways looking emoji].

So in the end, that means the git tree became an exact mirror of what happens to the kernel RPM in Fedora. Namely, it rebases on top of whatever upstream is doing. Now typically this is a terrible thing to do to downstream consumers. It means they need to force update their tracking branches and such. I'd be very hesitant about it if not for the fact that it's mostly just an ease of use thing. Another point is that it's analogous to how linux-next operates (albeit for different reasons). Overall, I think this is the best trade off between "git tree exists" and "maintainer remains sane". The follow section should help illustrate why as well.

How it works: all your rebase are belong to us

How it works is pretty simple. The tree tracks the main upstream git repo in the master branch, and the Fedora content for a release is contained on the "rawhide" branch. As of right now, only the rawhide kernels are tracked here, mostly due to time but also somewhat to see if this whole experiment is worthwhile. So when we go to update the Fedora kernel package in rawhide (which is typically a daily build), we do:

git remote update; git checkout master; git merge origin/master; git checkout rawhide; git rebase master;

That sequence pulls down the latest from Linus onto the master branch, then we rebase the rawhide branch on top of the changes we just pulled. We fix up any issues during the rebase, and then we're set.

At this point we could call it good and just push it to the remote repository, but that didn't really feel worthwhile. I wanted to gain something out of this as well, so I started thinking about how I could actually use the tree. I came up with a couple of things.

Since we're rebasing the patches in git, we don't need to do it separately in the Fedora package repo. There's no sense in doing work twice. After some initial renaming of some patches and such, I now use the git tree to generate the patches we add to the spec file by using git format-patch master.. and a script to copy them to the working dir on my machine. This means we always have a nice fresh copy of the patches for that specific upstream base. It does mean that each patch typically gets one line of change (the sha hash of the commit) everyday, but I don't think that's a big deal. This actually saves me time now and it helps keep our patches fairly "clean". They all apply with git-am and most of them have changelogs and such. An overall improvement from before.

I thought the tree itself could still be more useful to people as well. As the last commit of each update, I include the generated kernel configs that we build the kernel with. Our config setup in the package is somewhat confusing to people that don't work with it daily, and it's not obvious how all the fragments go together. This has them all in one place.

Then I realized that isn't particularly helpful without a way to track things (remember, we rebase every time). To fix that, I add an annotated git tag for every build we do in koji. The annotation points to the build task URL in koji itself. Someone that wanted to could browse to a particular tag and have a directly link to the Fedora package the source represents. You can also diff between tags, etc. I added the tags starting around the 3.17-rc4 builds.

(IMPORTANT: the link does not mean that package was built from this git tree. It simply points to the build that best represents it. Official builds are done from the Fedora package repo via SPEC file and tarballs/patches as mentioned before.)

With those things in place, we push out the results to the master and rawhide branches for the day, and call it good.

But what about when other Fedora maintainers add, remove, or modify a patch? Thankfully that happens less frequently than most people think. When it does, git rebase -i is my friend. It lets me rebase the patches while editing or amending, etc. If you haven't used it before I would highly recommend learning how it works.

Whither yon git tree?

OK, OK. You've probably had enough of my blabbering so you can find the git tree here:


What's next?

To be honest, I'm not sure. The obvious thing would be to add the release branches (F20 and F21. F19 is dead to me.) so that will probably happen. Beyond that, I don't see much changing at this point. I'm very curious to see if people use this, and if they use it for something more than a web viewer of the tree. Time will tell!

Overall I'm glad I went through the exercise. It's been informative and beneficial, even if only in a small way.

What: exploded source git tree for Fedora kernel package (rawhide only for now)
How: rebases with every push; annotated git tags pointed to corresponding builds
Where: https://git.kernel.org/cgit/linux/kernel/git/jwboyer/fedora.git
Next: Up to you mostly!
link2 comments|post comment

[ viewing | most recent entries ]
[ go | earlier ]