You are viewing jwboyer

pointless pontifications of a back seat driver [entries|archive|friends|userinfo]

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

Flock 2014: Over but the work is just starting [Aug. 10th, 2014|02:23 pm]

It is traditional when you go to a Fedora conference to do blog posts about your experience there. I think they're great, and I've enjoyed reading those posted by others. There are some great talk recaps and it's always interesting to see how people are spending their time.

For one reason or another, I'm not one of those that manages to do this on a daily basis though. That usually means I try and do a week long recap of what happened, and the information is largely stale or redundant and I don't think it makes for great reading or communicating. I thought about this a bit and I think instead I'm going to talk about what I'm planning on doing now that Flock is over that came as a direct result of things that occurred while I was there. Hopefully that will be a bit more entertaining and really keep me honest with those plans.

Custom Kernel builds

A number of people came up to me and asked how to build custom kernels. The instructions we have on our wiki page are probably fairly valid, but I really expect that they are stale in certain areas that lead people to have trouble. The kernel maintainers never really build kernels the way the wiki page suggests, so it might be time for a revamp. I'll look this over and give some examples of common cases (enabling config options, adding patches) and hopefully this will lead to people having fewer issues.

A Fedora kernel git tree

We've often talked about having an exploded source tree. The problem with that is that it's of little benefit to the Fedora kernel maintainers. Koji builds from the SRPM and that's from the tarball, patches, and spec file. However, getting a kernel tree created is likely only difficult from the start, and keeping it maintained could probably be automated. It would have to rebase the branches, as that is literally what we do in the SRPM, but if people find it useful and it isn't overly difficult to create, we might as well try. Plus, it's a good way to spend a day or two when I need a break from the daily grind.

Packaging changes

A couple people spoke with me about some possible packaging changes for the kernel. Things like the kernel-source package that Fedora shipped long ago. To be honest, not all of these will probably happen but there are a few minor things we'll likely change.

Bug workflow

Unsurprisingly, this is one of the larger topics I left with things to work on. For starters, we want to try and automate some of the triage we do. It shouldn't be massively difficult to write something that parses an oops and figures out the correct part of the kernel it came from. We'll likely start with ABRT bugs, since those are mostly formatted in a uniform way and work from there (we might even be able to get ABRT to do this at some point as well.) We can also pull from the retrace server once that is able to filter out results. This is a pretty broad description of what to do, and that's because there are many options and directions we can take. Hopefully some automation will really help us get time back to work on different things.

Rawhide kernel handling

In my kernel talk, Owen asked me "Is the rawhide kernel the right kernel for rawhide?" He was referring to the fact that we build with debug options enabled, and this has performance impacts. Those impacts are less noticeable to a human now that we leave SLUB debugging off, but if you're using perf or some automated performance metrics it is going to be easy to see it as slower than a normal build. Even ignoring the performance aspect, there are still days where we'd like to get a build out for "adventurer" testing before it hits the main rawhide compose, but we don't have a great way to do that. I'm going to think about how we can use the kernel-tests repo, the autotest harness we have, and some of the other tools to see if there's a way we can improve both the rawhide kernel quality and the number of people willing to test rawhide kernels. Nothing concrete yet, but I think some of the work we've done over the course of this past year will really help.

Fedora Governance

This really has nothing to do with the kernel. The discussions I had over the course of the conference around governance were really good. The dedicated session Toshio and Haikel held was well attended, and participation was excellent. I'm looking forward to working with the broader Fedora community to hammer out the rest of the details in the plan and start executing on that soon. There's a lot of work to do, but out of everything at Flock it's the one thing I am most eager to jump into. I honestly wish I could spend more dedicated time on working on broader Fedora issues in general.

I'm sure there are things I've forgotten. If I talked with you about something that isn't on this list, send me a reminder. After a week of meetings and conference, my brain can get a little full.

As always, the best part of Flock was see old friends again and meeting new people. I had good conversations with everyone I spoke with, and it was a really great time. If you haven't already, be sure to thank the Flock organizers. Having pitched in a bit over the past 2 years, I can tell you that it isn't at all easy to get this conference planned, and it can be exhausting while it's running.

Looking forward to working with everyone over the course of this year to see what awesome things we can bring to the next Flock!
link2 comments|post comment

At the playground [Jul. 8th, 2014|03:25 pm]

A while ago, we had a thread on the Fedora kernel list where people were expressing a desire for easily accessible kernels that contained new and shiny technologies still under development. We've had a number of discussions like that over the years, and none of them ever seem to pan out to something remotely feasible. The patches in question tend to be larger, with no clear upstream path. The one thing we've learned from uprobes, execshield, and now Secure Boot is that carrying non-upstream patches is kind of a nightmare. It makes us different from upstream, creates a maintenance burden, and actually leads to those patches taking even longer to get upstream because there's no impetus once they're "in-distro". So we've been sufficiently hit with the cluebat enough to say "no major non-upstream patches" and to be honest that approach has been working well.

Except that precludes people from working on them, getting userspace ready for them, and improving them. There's a catch-22 there. The features aren't making upstream progress because nobody is using them, but nobody is using them because they aren't easily accessible. Not everyone is willing to build their own kernel just to play with a new toy. Fedora hasn't been willing to take the hit so they can easily use something that might not actually get upstream (see aufs). So what can anyone do?

The Fedora kernel team has thought about this for a while. How do we deliver something people are asking for without impacting the rest of the distro. The rawhide nodebug repo has shown that simply delivering a side-band kernel doesn't drastically increase the maintenance burden, but that isn't really a great comparison. That is literally the same kernel, just with the debug options disabled. However, the introduction of Coprs in Fedora gives us another option.

The kernel team first used a Copr to work on the kernel-core/kernel-modules packaging split before landing it in rawhide. The Cloud team was able to point their instances at that repo and test it out. This actually worked out really well and we fixed a number of bugs before they hit rawhide. The content of the kernel was still a 100% match to the mainline Fedora kernel, but the packaging difference was enough to prove the mechansim worked well enough.

So with that in mind, I've decided to create a kernel-playground Copr. The idea behind this is to provide easily consumable kernel builds with some of the things people have been requesting for a while now. Before we get to specifics, here's the playground rules:

* This is not a "supported" kernel. Bugs filed against the Fedora kernel component with the kernel-playground kernel will be closed. If the bug clearly lands in one of the features being carried, we'll make attempts to forward the information to the relevant upstream developers. If there are issues we can still discuss them, but we want to keep the Fedora distro and it's process/tracking isolated from this like we would with any other non-standard kernel.

* The kernel-playground build will roughly track rawhide, with selected features/patches added on top. I say roughly because it will not receive a build for every rawhide build done like the nodebug repo does. The tentative plan is to update it once per upstream -rc and final release.

* x86_64 only. In the future, other 64-bit architectures might be possible.

* There are no guarantees around feature/patch additions. We'll do our best to keep things integrated, but we might have to drop additions if they become too much work. There are also no plans to "graduate" features from the playground kernel to the standard Fedora kernel. "When it gets upstream" is still the default answer there.

* In all likelyhood, this kernel will crash on your machine at some point. It will probably even eat data. DO NOT RUN IT ON ANYTHING YOU RELY ON HEAVILY. Make sure whatever data you care about is backed up, etc. We provide this AS-IS, with no warranty, and are not liable for anything. (The same is true of Fedora kernels as well, but doubly so for kernel-playground.)

OK, now that we have that out of the way, let's talk about what is actually in kernel-playground. At the moment there are two additions on top of the standard rawhide kernel; overlayfs (v22) and kdbus.

Overlayfs is one of the top competing "union" filesystems out there, and has actually been posted for review for the past few releases. It has the best chance of landing upstream sometime this decade, and there has been interest in it for quite a while. I believe things like Docker would also be able to make use of it as a backend. I'll track upstream submissions and update accordingly.

kdbus is of course the thing that Lennart Poettering and Kay Sievers have been talking about at various conferences for a while now. It is the in-kernel d-bus replacement. It has not been submitted for upstream review yet, but systemd already has support for it and things seem to be progressing well there.

I'm open to adding more content if there's a reasonable enough purpose to do so. Some interest in the live-patching kernel solutions (kpatch/kGraft) has been expressed, though we won't be producing patch updates so that still requires heavy user involvement to actually be useful. Backported btrfs updates from the latest development tree has also been floated as an idea. If you have a feature/technology you'd like to see, shoot an email to the Fedora kernel list and we can chat about it.
link6 comments|post comment

Apr/May Fedora kernel release overview [May. 20th, 2014|10:42 am]

A brief release overview of the past several weeks below. If you have any questions, please chime in!


Currently at 3.13.11 but should be rebasing to the 3.14.4 kernel that has been in updates-testing very soon.


Currently at 3.14.4. Nothing pending. The upstream stable maintainer has been behind in doing stable releases and still has several hundred patches to wade through. More 3.14.y updates will be coming as he works through the backlog.

Hans has been working on several backlight issues in F20 (and F19). There are a number of models fixed, but several still remain. If you are having backlight issues, take a look at his blog post on how to debug these:

Many thanks to the reporters and testers on the existing bugs, and especially to Hans!

Rawhide (F21):

Currently at 3.15.0-0.rc5.git3.1. Continuing towards 3.15 final.

We've had several packaging changes in rawhide over the past few weeks. The kernel-core/kernel-modules split landed and has, from all indications, been much of a non-event after the first few days. That is good to see, since it should be minimal impact.

We also enabled two features that Kyle McMartin provided. The first is that the kernel packages will now have Provides(foo.ko) included for every kernel module the package provides. That lets us switch Requires from specific kernel packages (like kernel-modules-extra) to Requires on the specific kernel modules needed. With that we can move modules between packages more freely without breaking dependencies in the userspace packages that need certain modules.

Kyle's other feature was to compress the installed modules with xz. The kmod utility has been able to handle compressed modules for quite some time. This actually has significant space savings on the installed size, and the performance impact on module loading is pretty negligible. Combined with the kernel packaging split, this should make the Cloud people rather happy. Thanks Kyle!
linkpost comment

DevConf 2014 [Feb. 17th, 2014|01:52 pm]

I recently had the privilege of attending DevConf 2014, and it was really impressive. This was the first time I had ever traveled to the Czech Republic and I found the people to be extremely pleasant and helpful. The city of Brno was really easy to get around and the conference accommodations were nice.

The conference itself was more than I expected it to be, both in terms of scale and quality. That's not to say that I expected it to be poor quality, but at all conferences there are some talks or tracks that wind up being less worthwhile than you expect. That wasn't the case here. Every talk I attended was well done and very informative. Despite the conference organizers warning that the presenters were engineers and not public speakers, I thought everyone did a great job. Clearly this is a continuing trend because the conference was packed with people. It was almost to the point where a larger venue would be needed.

Day One

The first talk I attended was Thomas Graf's Linux networking stack overview. He took a very complex topic and presented it in a clear manner without overwhelming people. It was very well done, and I'll probably refer back to his slides when looking at networking issues for quite a while. I really like these kind of overview talks, because while I stare at kernel code all day long, it's hard to get a bigger picture of the whole stack if you don't work on it daily. Very helpful.

After that I attended Alex Larsson's talk on Docker. Docker was everywhere at the conference and the hype around it was pretty evident. Being a kernel guy not very much in tune with the higher level userspace stacks, I wanted to understand why people were excited and what usecases might be present for Fedora. Alex did a great job of talking to that hype and then was brave enough to give a good demo that worked. His talk was great and while I might not see myself using Docker in the future, it was definitely informative enough for me to explain Docker to someone else later on.

It was interesting to go from the Docker talk to Colin Walters' talk on OSTree. Where Docker seems designed to make it easy to swap out "stacks" of things, OSTree is very focused on building an immutable tree and having that be the basis for testing and deployment. The ability to build the trees on the server and deploy them on both bare-metal and virtual machines is interesting to me. I can certainly see this being very helpful as Fedora is making progress towards a release. It removes the ambiguity around what "version" of Fedora you're testing, since the trees are named and immutable whereas a typical system on Branched is a collection of packages that may or may not match the latest releases. I think this could also provide a great base for things like cloud-images, where you don't really focus on running an install for a great duration and upgrading it as you go. I really appreciated Colin's approach to the presentation as well. He presented both benefits and downsides to his ideas, which is not something you often see from someone presenting a new technology.

After that I did a bit of "hallway tracking". Catching up with people I haven't seen in a while, and meeting several that I've never met in person for the first time. As is typical with most conferences, the hallway track is very useful and this was no exception.

I went to the Continuous Integration with OBS talk, given by Adrian Schröter from SUSE. It was a very interesting presentation. Being able to see what OBS does differently and in-addition to koji is always good. I had looked at OBS somewhat in depth a few years ago, and the SUSE team has really continued to improve and polish it since then. Adopting some of the same features within Fedora would be beneficial, but whether that will be done is unknown to me.

The last talk of the day for me was Jeff Scheel's Linux on POWER. He did a great overview of the hardware advances in POWER recently, and what IBM is looking to accomplish with Linux on POWER. The push to get KVM running on POWER will hopefully help adoption there, as it gives others more familiar with Linux on x86 a common set of tools and interfaces to work with. To hear that the traditional IBM value adds of RAS features were also being worked on for Linux enablement was somewhat surprising and refreshing as well.

After that it was off to a dinner and then back to the hotel for sleep.

Day Two

Day two started with Thorsten Leemhuis's kernel overview talk. For those that have read Thorsten's online summaries of what goes into each kernel release, this was much in the same but with great additional detail around it. I really enjoyed the presentation. Even working on the kernel all day, I still found a few things that I had missed myself. Plus, having someone else's perspective on things is always refreshing.

Immediately after I stayed for Lukas Czerner's advanced filesystem/storage features talk. This was a great overview of some of the new and complex capabilities various filesystems can accomplish, as well as some of the new trends in storage coming down the pipe. It also served as a good lead-in to a later talk from Ric Wheeler on the storage side of things. Very well presented, and worth viewing the video for anyone that is interested in these areas.

I had intended to go to the kdbus talk next, but I had already seen the video from LCA on that subject and figured the talk would be much of the same. So it was outside for a bit of hallway track again. After that, I went to Ric's aforementioned talk on the new storage technology coming out. Specifically shingled drives (SMR) and persistent memory. It's interesting to see how PM is driving changes to the block and fs layers, as the IOPs numbers for those devices are very high. Combining PM with SMR seems like an interesting way to maximize the performance and capacity issues. It will be interesting to see how things play out in this space.

Later in the afternoon I attended Dan Walsh's Secure Containers talk. As usual, Dan had a great amount of information on the work he and others are doing to secure Linux containers, why you'd use containers, and what that entails. Of course, this naturally included mention of Docker. I'm curious how the adoption of container technology will drive changes in the market place over what the adoption of virtualization has done already. Thus far it seems Containers are primarily used for scaling and sandboxing at a lower cost than virt, where virt is needed for full isolation and higher security needs areas. Openshift and Docker certainly have a lot of hype around them, so I'm wondering if the security issue will be a less of a factor in which technology gets deployed going forward.

Right after Dan's talk I stayed for Kyle McMartin's Linux porting talk. This gave a great overview of the difficulties in porting free software (not just the Linux kernel) to a new architecture. The number of assumptions that have been made in various chunks of code is what really drives a lot of the work, and Kyle presented a good summary of the bulk of those issues. He finished off with some "war stories", which are always entertaining. I had seen bits and pieces of this talk at Flock last summer, but not the whole thing. It was well worth it, and had some good updates to it. I would recommend watching the video if you're interested in the 64-bit ARM work being done, or porting work in general.

That evening was the conference party, which was really nice. The food provided was tasty, and it was good to catch up with people. I retired a bit early as jet lag seems to catch up to me on the second day for some reason, but I still had a great time.

Day Three

Fedora day! To say this was Fedora day is somewhat misleading. There was talk about Fedora, implications of things in Fedora, and using Fedora for things throughout the conference. It was ever present and this was great. However, the final day of the conference was dedicated to Fedora and this is the main reason I came.

I didn't actually give a "state of the Fedora kernel" talk as I normally do at conferences. That talk is mostly the same, and frankly the presentations throughout the first two days on kernel topics were far more interesting. Instead, I was there to discuss and the related Working Groups.

Matt Miller kicked off the overview. This was a great way to start, since much of the day following revolved around what we're doing and why. When the video is available, I would highly recommend those trying to catch up on this to watch it in the entirety. Very informative. The questions at the end were actually one of my favorite parts. There were some great questions and it's clear that we need to do a better job of communicating reasoning, status, and goals in the future.

Following Matt's overview, we had a panel discussion with representatives from all of the various Working Groups. It was the first time the liaisons were all in one physical location, and it was very interesting to hear the questions and the answers each gave to them. Overall, I would say we're all on basically the same page, which is good given that a lot of things are still not clearly defined and we've been doing most of this work via email and IRC. Nice to see that distributed... er... development? works well even for things that don't relate to code. I was somewhat surprised that there weren't more specific questions on the Workstation product, particularly given some of the more energetic discussions we've had on the mailing lists so far. Was the summary of the product I gave was sufficient to qualm many fears, or were people so entirely skeptical of the whole thing that they think it will fail and doesn't matter? I have no idea. The questions we did get were well thought out and I hope that our answers were as well reasoned. Again, I highly recommend watching the video of this session. I really enjoyed participating and would definitely do it again, perhaps at Flock.

The last session of the conference I attended was the Meet Your FESCo session. I've been to several of these, and they're usually pretty interesting. After introductions they took questions and nobody seemed to want to break the ice. So I asked FESCo how they see the interaction between themselves and the Fedora Board working with the change being driven primarily through FESCo. The answers were interesting to me, in that most people seemed to want to stick with the status quo here. FESCo driving technical change/decisions and the Board acting as a more general project wide body. That differs somewhat from my opinion, but that's probably a discussion for another day.

Overall I thought DevConf was very worthwhile. I wish I could have gotten to some of the lightning talks on the first day, and some of the other talks in the afternoon of the second day, but there's only so much time. I hope to have the same opportunity next year, and I'm looking forward to being back in the Czech Republic for Flock in Prague this summer.
link2 comments|post comment

Saying NO less [Jan. 14th, 2014|02:30 pm]

This post isn't about my kids, but they're the prompt for it so bear with me.

My kids participate in an after school program that is designed to foster creative thinking. They do spontaneous problem solving and "outside-the-box" kind of thinking. Watching them do this and hearing about the ideas they come up with has been pretty interesting so far. Their solutions to problems aren't always practical, but it's their interaction with their team members that I really noticed.

They are the youngest two kids in the group, by at least 2 years and in some cases 4. Sometimes when the coach gives a problem, the older kids say "oh, we can't do that because " or "that won't work because " etc. The coach is great and works really hard to get them to think harder, but she said my kids very rarely do that to begin with. Instead they just throw out all kinds of ideas. Need to simulate water? Use lots of blue M&Ms. Need to make something look smelly without actually making it stink? Use pipe cleaners to create comic style scent waves coming off of your body. Some just aren't feasible to do, but then the group discusses the ideas and figures out _why_ they aren't really practical. And a lot of the time the ideas are great. All it took was an unjaded (and sometimes naive) point of view.

That got me thinking about Fedora and open source in general. Fedora is 10 years old now, and despite the stuff we're currently working through, we're pretty entrenched in our ways. We have RPM and packaging guidelines and processes. They've worked to produce 20 versions of Fedora, and there is a lot of momentum behind them. However, that doesn't mean they're the best long term thing going forward.

We have developers writing applications that bundle things. We have people wanting to make app containers that are portable so they're distro-agnostic. We have things flying in the face of "Unix tradition". The response to a lot of this stuff within Fedora (and other distros) has been mixed, but there is quite a bit of "we can't do that", or "it's too hard", or . Even the stuff is met with a lot of resistance based on the grounds that it's a rather large and significant undertaking. There are lots of people saying "it can't be done" or that it won't matter. They may be right. They may also be wrong.

Even within the kernel team, I tend to have knee-jerk "no" answers. Usually with good intentions, but often without really exploring it in detail. It doesn't really help expand my knowledge base and it probably leaves proposers of things with a sour taste in their mouths. I don't know anyone that likes being told a blanket "no" without an explanation.

Thinking about all of this made me wonder if we're missing some great ideas because we've lost that unjaded point of view. So I think from now on I'm going to try pretty hard to consider suggestions with a more open mind. If I think something isn't really practical or feasible, I'll try and respond with reasoning as to why or I'll respond with what it would cost to do and let the proposer work that out. Maybe this won't really lead to anything new, or maybe it will lead to doing something new that ultimately fails. But at least we have the possibility of making something better or learning from a mistake. I guess I'd rather have that than sit around being stagnant and eventually irrelevant. Here's to thinking outside the box.
link6 comments|post comment

The kernel is not special [Dec. 16th, 2013|02:46 pm]

Lately I've seen a lot of weird references to the kernel come up in Fedora discussions. These range from basing things on its upstream release cycle, to somehow holding it up as an example of some strange patch/development model. It's really kind of baffling and also kind of frustrating. So here's the secret about the Fedora kernel: it isn't any more special or different than any other package in Fedora. It's a package.

The Fedora policy for kernel patches is "upstream first, backport when relevant", like most of the rest of Fedora. There is no development done directly on the Fedora kernel at all. Patches that are included in the kernel package are already mostly backports, and the few that aren't are either dumb things to silence warnings or change defaults. There are exceptions here and there for major features, but those are almost always posted by their authors upstream at the same time (and the authors are rarely the Fedora kernel maintainers). We don't want to carry major out-of-tree features for any longer than we have to for all the same reasons you don't want to carry a ton of patches against any other upstream release. It winds up benefiting far fewer people to do so, and increases the maintenance burden on Fedora.

The upstream release cycle is fairly regular, which is a good thing. However, that is done to benefit upstream, not anyone else. It's also rather short, so basing something else on when the upstream kernel releases would likely introduce a lot of significant churn for really no reason. If you're developing a Feature for a Fedora release that requires kernel changes, we're going to tell you to get them upstream. If you do that with sufficient planning, then we're likely to get that upstream release in the Fedora release anyway. If for some reason it doesn't line up, then we can look at carrying it for a while. But to base some other release schedule on the upstream kernel release is really kind of odd. Just don't do it. Plan for whatever thing you're trying to release by figuring out what problems/features you're trying to solve, how long it's going to take, and how much testing you need.

The Fedora kernel also has a lot of bugs in bugzilla. Maybe more than most packages, but what _isn't_ different is that we're working as best we can to resolve them. Holding up the Fedora kernel as a do or don't example of bug handling is really no more relevant than any other package. Some bugs we don't get to. Some we have no clue on. Some we fix. Some we fix and then there's a regression. All of these things happen (or can happen) in every other package in Fedora. We aren't doing anything magical here. (If you have bugzilla magic, PLEASE LET ME KNOW.)

The kernel can cause your machine to not boot. This is true! However so can glibc, dracut, shim, grub2, grubby, systemd, lvm, dbus, and a number of other important packages. Like all of those, we strive to avoid breaking things so people can use their computers as they'd like[1]. Most of the time we're successful. Sometimes we aren't. In the times we aren't, it's normally not a Fedora specific issue and instead is something we need to fix upstream. Exactly like the other packages.

"But the kernel gets major version updates in the middle of a release", you say. It does. This is, admittedly, slightly different than how most packages work in Fedora. We've discussed this several times elsewhere, but really the best way to think about it is that it's done because of scale. Each upstream release gets between 9,000-10,000 commits. That's a lot of bug fixes sitting there. Just adding backport after backport quickly gets you a long way towards that upstream release anyway. Plus, going through each of the releases and grabbing the fixes is not feasible for the team we have. There are three of us. Remember, there are on average about 1,250 people contributing to each upstream release and there were almost 4k different people over the course of last year[2]. Which kind of leads me to my last point.

I truly believe that one of the few things that is "special" or significantly different from other packages is that it is one of the most successful examples of an Open Source project that exist. That is a testament to the people in the upstream community, the commitment from the contributing companies, and the willingness to solve the difficult problems and adapt as needed. There are great examples of these qualities in other upstream projects too, but the Linux kernel has been doing it for a long time and doing it consistently well. So if you want to talk about the kernel being different or special, please do it in that frame of reference. Focus on the upstream community. Otherwise, it's just code and it's just a package.

[1] Except DaveJ. His "usage" of computers is basically breaking them in every way possible, so we kind of don't care about how he wants to use things. Chances are he already found the breakage anyway. He suffers so you don't have to. Buy him a drink (or another computer for the one he just broke).

[2] Numbers fudged from the "Who Writes Linux 2013" report by the Linux Foundation.
link4 comments|post comment

Flock 2013 [Aug. 16th, 2013|09:19 am]

Flock was a great and exhausting conference. I can honestly say, it is one of the best Fedora conferences I've been to. Now, being one of the people helping out with it (in a small fashion), that might sound biased but I've had many other people tell me the same thing. I'm really glad people found it worthwhile, fun, and productive. It makes the work put into it much more rewarding.

I gave the normal state of the union talk for the kernel. It is generally the same talk we give at every FUDCon, but I decided to actually present real data on the bug counts and release behaviors we've been seeing for a while. I had some graphs that showed total bugs, incoming bugs, and closed bugs over the past 18 months or so, across all releases and within F17, F18, and F19. The charts all showed similar trends, but I think it was good to get a visualization of it.

I attended a few sessions as well, with the Census talk and hackfest being two of my favorite. Census is a replacement for the old Smolt project aimed at not only collecting the data, but making it useful and extremely easy to use. One of my favorite quotes came from the project maintainer, Nathaniel McCallum. We were discussing ideas for what to collect with PCI and USB information, and he at one point he said "This is a hackfest. Josh, do you want to start writing that now?" I'm sure the look on my face was slightly confused. Write code? At a conference? But then I dived in and got it done in a relatively short amount of time (for a kernel hacker working with python). In fact, it took me more time to figure out how to use the workflow features of Github than it did to write the code. I must be getting old. I just wanted to email my patch somewhere.

The end-of-conference session with FESCo was great to sit in on. There was lots of good discussion around some of the bigger idea sessions that happened during the conference, like Matthew Miller's "Fedora Next" rework. Having FESCo and the community members in the same room for high-bandwidth conversation provided a much clearer and organized flow of information than you get from an IRC meeting. IRC works well enough most of the time, but I'm still a firm believer in face to face conversation.

That same face to face communication let the Fedora kernel team have some really good and productive conversations as well. We talked a lot of things over and I think we're in a good place going forward. There is a lot of work to be done, and some of it won't immediately be apparent to end users, but I think the pay off will be much better in the long run. The next few days are going to be spent collecting the ideas we talked about and writing them down so I don't forget.

I'd like to thank Red Hat, the Flock sponsors, and the Flock staff for putting on a great conference. It's not easy to do, but I think it's entirely worth it.
linkpost comment

Rawhide kernels and kernel-install [May. 17th, 2013|01:28 pm]

This week we took a patch from Harald Hoyer to the Rawhide kernel.spec to switch to using the kernel-install tool to install the kernel onto the system. This new tool is provided by systemd and is intended to be distribution agnostic, with the eventual goal of getting into the upstream kernel Makefile as well.

Previously, the Fedora kernel package would call a script called 'new-kernel-pkg' to do the boot loader configuration addition, initramfs creation, and depmod steps in the RPM %posttrans script. That script is provided by the grubby package, which despite its name doesn't mean it only works with grub. It works with grub, grub2, silo, yaboot, lilo, and a variety of other bootloaders. However, to the best of my knowledge, it is a Fedora specific package. The other distributions all use their own flavor of initramfs/bootloader tooling for one reason or another.

The kernel-install tool provided by systemd is aiming to replace all of those tools. It was created in part to support the Boot Loader Spec, which aims to create a cross-distro way of managing boot loaders and boot loader entries. The intention is to make dual/multi-distro booting work well without requiring a lot of manual hacks or configuration. If you have questions on this, I would encourage you to talk to Harald Hoyer or Kay Sievers as they're driving that work.

At the moment, Fedora's kernel-install is patched to still call new-kernel-pkg if it exists. That should easy the transition for existing installs and result in machines still working. About the only fallout should be that you won't be able to install Rawhide kernels on older releases of Fedora. The RPM requirements specify that you have systemd >= 203 and dracut >= 027. Of course, there still may be bugs. If you're running Rawhide and hitting issues when you install or remove kernels, let us know.
linkpost comment

Fedora and Ubuntu Kernel Config Comparison [May. 9th, 2013|02:29 pm]

Every once in a while, I crawl out from under the rock that is bugzilla and I try and look around at what others are doing in the distro kernel space. Today I was curious how Fedora and Ubuntu compare in how they configure the kernel. I've long thought that for all the focus the kernel gets, it should be the most boring package in an entire distro. It should work, work well, and that is about it. It isn't there to differentiate your distro. It's there to let your distro run. So, will my personal belief stand up, or would I find something in the configs that proves one "distro" is better than another? Let's dive in.

I won't give a line by line comparison because 1) that would be very long and 2) that would be utterly boring. You don't need to know that e.g. Ubuntu enables one scsi driver that Fedora doesn't or vice versa. Instead, I'll try to focus on various functional areas and give an overview.

The kernels

For comparison purposes, I used the config from linux-image-3.8.0-19-generic Ubuntu 13.04 package and the 3.8.11-200.fc18 kernel from Fedora. These are the same base upstream kernel version, and both of them are from the latest stable release of their respective distros. I used the x86_64 (amd64 in Ubuntu speak) build, as I found that to be the most relevant in this day and age. I could have used the kernels under development, however those are more apt to change and I thought it would be better to compare things out there in wider use.


The vast majority of the config is pretty much identical. It's what you would expect from a distribution config. There is a wide range of support for all kinds of devices and buses, with much of it being provided by modules instead of built in. This should be expected, given that they're coming from the same upstream code base and really there is only one sane option for most config options.

Low-level options

A few of the low-level options did differ though. One of the more notable ones is that Ubuntu has CONFIG_NR_CPUS set to 256, whereas Fedora has it set to 128. On the face of it, one might think both values are overkill as people aren't booting on machines with 128 cores very often. However, Ubuntu has transitioned to having a single kernel between it's desktop and server variants (starting with 12.04 I think), so it does make some sense for them to have it set higher. Fedora can revisit our choice if we get a sudden clamoring for massive machine support. I await my mini-supercomputer test system.

Another area where we differed was in the NUMA options. Ubuntu enables CONFIG_X86_NUMACHIP, CONFIG_MOVABLE_NODE, CONFIG_MEMORY_HOTPLUG, and CONFIG_MEMORY_HOT_REMOVE. The CONFIG_NODES_SHIFT value was higher on Ubuntu as well, allowing for more NUMA nodes. Again, given their server variant that probably makes sense for them. Fedora just doesn't deal with that class of machine on a regular basis.

The HZ area also had some differences. Fedora defaults to HZ_1000 compared to Ubuntu's HZ_250. That means the timer tick is 1000 times per second on Fedora, versus 250 on Ubuntu. Typically a higher HZ value is used for better interactive response. On a server system where there isn't typically a desktop or user waiting for interaction, a smaller HZ setting allows more work to get done as there are few timer interrupts to process. Of course, both distros set CONFIG_NO_HZ, which means the timer interrupts are only triggered on an as-needed basis when the CPU is idle. I would expect that both will eventually also go with the new completely tickless options that were introduced in the 3.10 kernel merge window after they have matured a bit.

Ubuntu also enabled CONFIG_FAST_NO_HZ whereas Fedora doesn't. That option lets CPUs with pending RCU callbacks enter the dynamic tick state (idle) by extending the grace period. That slows down synchronize_rcu calls, but by letting the CPUs enter idle state more it improves energy efficiency. Sounds like a great thing, and it really probably is. Fedora disabled this after having it enabled when it was first introduced and hitting numerous issues with it. The option has been upstream for a few kernel releases now, so it's probably something we should revisit.

Default choices

For the most part, the default choices were the same. A couple of the areas where they diverged were in the default I/O scheduler, and the default CPU frequency governor. Ubuntu uses the deadline scheduler by default compared to Fedora's CFQ (Completely Fair Queueing). Ubuntu uses the performance governor instead of the ondemand governor that Fedora uses. To be clear, all of the options are provided by both distros. They just differ in the default setting.


The security stack is probably one of the larger areas of difference between Ubuntu and Fedora. Fedora is solidly an SELinux camp. It is the only LSM we enable and is therefore naturally the default. Ubuntu on the other hand enables all of them, including SELinux. I was kind of surprised by this, as I expected they would only enable their default LSM which is AppArmor. I wouldn't argue that this makes Ubuntu more secure than Fedora by any means, but it does give users a greater choice in what they use for MAC/DAC security.

Module options

Fedora and Ubuntu both produce signed modules. I was surprised Ubuntu was using this given their statement of it not being needed for Secure Boot purposes. So perhaps they're using it for some other purpose. Looking at their choice of using CONFIG_MODULE_SIG_SHA512 that might well be the case. Fedora uses CONFIG_MODULE_SIG_SHA256 because that is the common hash implementation used in UEFI.

Ubuntu also enables CONFIG_MODVERSIONS. This generates a CRC value for each exported symbol prototype. When a module is loaded, it compares the CRC with that in the kernel and fails the load if they do not match. It is a simple form of ABI check for modules. They also enable MODULE_SRCVERSION_ALL, which embeds a "srcversion" field. Arguably, this lets people build modules against different kernels and then load them while knowing where the module came from.

The old, the obscure, and the crappy

Ubuntu seems to have a tendency to enable a lot more old stuff than Fedora. They have support for acorn, atari, ultrix, and sysv68 partitions. They build DECNET and Parallel IDE support. ARCNET. Things one would not normally expect to find, much less want to use in this day and age. Of course, Fedora has some weird stuff too. We still enable the OSS sound stack and Ubuntu appears to have ditched this entirely.

They also seem to have a lot of "embedded" focused options enabled. Things like full MTD support, all of the random keyboard drivers, SPI, MFD, Regulator, and GPIOLIB. They have JFFS2, F2FS, and VXFS filesystem support. They enable various SoC options for things like sound and media drivers. This seems a bit at odds with their desktop and server focus, but if one wanted to boot an Ubuntu kernel on some random x86 based SoC thing they might be able to.

Fedora also tends to only enable the main DRM drivers that support KMS and very few framebuffer drivers. Ubuntu enables everything almost. Personally, I'm fine with our KMS or go away approach. Oddly, Ubuntu doesn't enable CONFIG_LOGO. WHY DO YOU HATE THE PENGUINS?

Lastly, Ubuntu enables way more staging drivers than Fedora. If you aren't aware, staging is where drivers come in to get cleaned up before being officially mainlined. They literally taint the kernel as TAINT_CRAP, and we tend to stay away from them. Ubuntu is also carrying AUFS, the union fileystem and ALX, an ethernet driver for new Atheros chips that has had a pretty horrible review history upstream.


So did my personal theory of "kernels should work and that's it" hold? I think it did mostly. There are no shocking differences between the two distribution kernels. The few areas where we differ that might matter are worth reviewing on both sides, but at the end of the day none of those options are going to make any kind of massive improvement. I consider this to be a worthwhile exercise, but one I won't repeat for a while. I can only stare at kernel configs for so long before it makes even bugzilla start looking interesting.

If you've stuck with me thus far, thanks!
link22 comments|post comment

kernel64 [May. 7th, 2013|03:05 pm]

It all starts with a rant

Last week Linus ranted a bit about PAE kernels:

For those that don't know, PAE (physical address extension) is an addition to 32-bit x86 CPUs that allow them to address more than 4GB of memory. That sounds like a good idea on paper, but it is kind of hacky. The address lines allow addressing up to 64GB of memory, but the virtual memory space is still limited to 4GB. It causes the page tables to be larger which impacts performance. It also, as Linus mentions, causes lots of weird corner case behavior for memory management.

Fedora and PAE

Fedora has been building PAE kernels since Fedora Core 6, which is quite a while now. However, not everyone needs PAE support so we also still build a normal i686 kernel. In fact, most people that have 32-bit CPUs aren't running with more than 4GB of memory anyway, so why bother building PAE at all? Well, namely PAE brings in NX.

NX is another x86 feature that stands for 'Never Execute'. Namely, it is a way for the operating system to mark a page as containing either code for execution, or storage for data. It's a hardware assist for helping with things like buffer overflow exploits, etc. Now, there have been software implementations of this for a while and Fedora carried one for a long time called execshield. Fedora carried the execshield patches on the non-PAE i686 kernel for quite some time, however it's an additional out-of-tree patch and since it is an emulation of what NX does in hardware, it's naturally going to be slower. So for quite some time, anaconda has had code to detect if the CPU supports PAE and it will install kernel-PAE by default if so.

The issue

So if kernel-PAE exists, and it is used by default if the CPU supports it, what is the problem? Well, the problem is that PAE is a technology that makes a crappy situation on 32-bit CPUs tolerable and it totally irrelevant on 64-bit CPUs. For whatever reason, there are people that still install the 32-bit version of Fedora on their new shiny 64-bit machine. Maybe they like retro-computing. Maybe they're stuck using proprietary software that is only available in 32-bit versions. Maybe they really hate how python sucks on 64-bit because OMG pointers. Maybe they're masochists. I don't know, and it doesn't really matter. What does matter is that when this happens on machines with a larger amount of memory, they've worked themselves into those corner cases that Linus mentioned. Plus, it's just plain inefficient and it bothers me.

(I'll also note that people using actual 32-bit CPUs with PAE and large amounts of memory is another issue, but I can't fix everyone. If they want to do that, then they're going to be stuck with the problems.)

A possible solution

After lamenting and discussing this with a few other Fedora developers and the kernel team, Matthew Garrett suggested just building another kernel flavor for 32-bit Fedora called kernel64 which is a 64-bit kernel packaged in an i686 RPM. The intention would be to have this available in the 32-bit Fedora repositories, and for new installs of 32-bit Fedora on x86_64 machines to use this. Now, this doesn't mean tricking the users and installing the 64-bit version of Fedora when they think they're installing the 32-bit version. It would be for the kernel only. The rest of the install would still be i686 binaries and libraries, and things should operate normally as 32-bit processes.

What's old is new?

Those of you familiar with PowerPC at all will probably recognize this. When Fedora had ppc as a primary architecture, we did something similar. We put both 32 and 64 bit kernels in the ppc/ repository, and installed the .ppc64.rpm version by default on a 64-bit CPU. There were also a bunch of other .ppc64 packages in that repository, as Fedora didn't ship a split out ppc64 version of the repositories at all and for the most part 32-bit userspace made sense even on 64-bit CPUs. 32-bit PowerPC doesn't suffer from the lack of registers that x86 does.

That was done until Fedora 16 and the Fedora PowerPC secondary arch team split out ppc and ppc64, and now ppc is entirely unsupported anyway. Oh well. So much for that sidebar.

What would it take?

In order to get a 64-bit kernel in the 32-bit repositories, we'd either need to do some kind of special mash magic like ppc used to and bring in the .x86_64.rpm version from the 64-bit repositories, or we'd need to actually build a 64-bit kernel as a .i686.rpm. To do the latter, we'd need a 32-bit gcc that accepts -m64. Currently, the gcc in Fedora doesn't support that at all. Assuming that could be done, then building it shouldn't be too hard.

That leaves anaconda and yum. We'd need to patch anaconda to do the 64-bit kernel install on 64-bit CPUs. We'd probably also need to convince yum that it's OK for uname to return x86_64 and still have a completely 32-bit userspace. I seem to recall yum having some special handling on ppc for this, so it's possible it wouldn't be too hard.

Are you going to do this?

I'm glad you asked! I have no idea. It's something that might actually turn out to be a win for people, but it is hard to say without forging ahead and doing it. That's partly why I'm writing about it first. I also don't know how difficult the anaconda or yum parts would be. I wouldn't be at all upset if Fedora stopped shipping i686 separately, but I just don't think that's going to happen any time soon. Maybe this is worth the effort in the meantime. If so, I'll have to cook up a Plan for it in Fedora 20 or 21.
link7 comments|post comment

[ viewing | most recent entries ]
[ go | earlier ]