You are viewing [info]jwboyer's journal

pointless pontifications of a back seat driver [entries|archive|friends|userinfo]
jwboyer

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

Fedora 15 kernel plans [May. 15th, 2012|09:23 pm]
[Tags|]

The F15 release has been an oddity and somewhat of a headache from a kernel perspective. We've stuck with the strange 2.6.4x naming scheme to prevent userspace from breaking on the 3.0 scheme, and for the past 5 months it seems that almost nobody is testing things from updates-testing. It's as if nobody is using F15 at all. That does mean a fairly low number of new bugs reported, but it still has maintenance burden for us and we get concerned when updates sit around for weeks without getting to stable. Maybe we worry too much. Anyway...

With F17 rapidly approaching release, we briefly discussed the plan for the final month of F15. Looking at the usage rate, the upstream kernel status, and the overall timeframe before F15 EOL, we decided that we'll stick with the 3.3.y kernel series for the remainder of its lifetime. There should be one or two more 3.3.y releases upstream as the 3.5 merge window opens and we'll pick up a number of fixes that way. I'll be on F15 duty for the final month and will be backporting any important fixes that aren't picked up by the upstream stable series.

This will break commonality with F16 and F17 in base kernel version. That commonality has served us very well throughout the F15 lifetime, but rebasing it to 3.4 is of limited value at this point. I would be surprised if a 3.4 rebase even got to the F15 stable updates repo before it went EOL to be honest. Fortunately, we can stick with the 3.3.y series without having to worry about it drying up or going unsupported upstream for long durations of time.

So there you have it. F15, in my opinion, was a decent release overall and I used it for quite a while. I believe F16 was better and F17 improves on that yet again, but that should be the norm for releases in Fedora. For those of you still on F15, you might want to look at upgrading soon. If you don't, at least throw some bodhi karma our way so we can put F15 out to pasture gently.
link3 comments|post comment

koji bisecting kernel builds [May. 10th, 2012|03:13 pm]
[Tags|]

I've discussed a number of times the fact that we rebase the kernel in Fedora quite often. I'm not going to rehash that whole discussion, but it does have some side effects. One of the worst is that it inevitably introduces regressions. Those can range from "my doohickey no longer beeps when I cat some random string to it from a shell script I wrote in 1994" to "my machine doesn't boot". These are obviously of the same importance.

Now, if you as a user were constantly running rawhide you would notice these regressions immediately after your daily kernel update of the latest git snapshot and reboot. Except you aren't running rawhide, are you? No, you aren't. Instead, you seem to think you have better things to do than be guinea pigs for the upstream kernel developers and you're probably running the latest stable Fedora release. If we're lucky, you're running the Beta of the one under development. That means you don't see these regressions until we do a major rebase, and by then there are hundreds of commits and dozens of kernel builds between the kernel you were using before and the one you just hit a problem on.

When this happens, we often suggest finding the first build in koji that exhibits the issue. Those of you familiar with git (or, you know, math) would recognize this as doing a bisect. Normally we'd look at the two reported versions that represent good and bad kernels, and we'd go digging through the koji website and post a bunch of links to various kernels. Then the user would click, download, install, reboot, test, repeat until they found the first one that failed in their particular way they are interested in. After that, the kernel team would look through the range and see if we can narrow down which commit might have caused this.

That whole process can be kind of tedious, and usually relied on the kernel developers taking the time to find the links, etc. So instead of doing it by hand each time and having almost no log of what the user actually did, I decided I'd try and write a tool to help us do that. After hacking around and re-learning (or maybe learning in the first place) python, I've come up with a simple tool to do just that. It is aptly, and unimaginatively, named 'koji-bisect', is released under the GPLv2 license, and lives here:

http://fedorapeople.org/gitweb?p=jwboyer/public_git/koji-bisect.git;a=summary

A few caveats. 1) In the spirit of 'release/fail-early-often' I'm posting about it now so interested parties can laugh at my python coding skillz and give it a try. 2) It doesn't install the kernels for you, but it will download them so you don't have to. 3) Did I mention my python is bad? It is. 70% of the time it took me to write this was staring at docs.python.org. Alas, I guess that's what I can expect having my head firmly in C for so long.

A small example of it's usage. Let's say you are semi-diligently updating your kernel in F17 and you hit a problem with kernel-3.3.5-2.fc17. The last working kernel you had was kernel-3.3.0-8.fc17. So you essentially have the entirely stable series of releases between the good and bad kernels. You grab the tool from git and do:

[jwboyer@zod koji-bisect]$ ./koji-bisect.py --start
successfully connected to hub
Must have a good or bad index
[jwboyer@zod koji-bisect]$ ./koji-bisect.py --good kernel-3.3.0-8.fc17
Marking kernel-3.3.0-8.fc17 as good
[jwboyer@zod koji-bisect]$ ./koji-bisect.py --bad kernel-3.3.5-2.fc17Marking kernel-3.3.5-2.fc17 as bad
successfully connected to hub
downloads/kernel-3.3.2-5.fc17/kernel-tools-devel-3.3.2-5 |  97 kB     00:00 ... 
downloads/kernel-3.3.2-5.fc17/kernel-debug-devel-3.3.2-5 |  15 MB     00:16 ... 
downloads/kernel-3.3.2-5.fc17/kernel-headers-3.3.2-5.fc1 | 1.6 MB     00:00 ... 
downloads/kernel-3.3.2-5.fc17/kernel-debug-3.3.2-5.fc17. |  53 MB     00:31 ... 
downloads/kernel-3.3.2-5.fc17/python-perf-3.3.2-5.fc17.x | 136 kB     00:00 ... 
downloads/kernel-3.3.2-5.fc17/kernel-debug-modules-extra | 2.7 MB     00:05 ... 
downloads/kernel-3.3.2-5.fc17/kernel-modules-extra-3.3.2 | 2.6 MB     00:02 ... 
downloads/kernel-3.3.2-5.fc17/kernel-tools-3.3.2-5.fc17. | 217 kB     00:00 ... 
downloads/kernel-3.3.2-5.fc17/perf-3.3.2-5.fc17.x86_64.r | 882 kB     00:00 ... 
downloads/kernel-3.3.2-5.fc17/kernel-3.3.2-5.fc17.x86_64 |  51 MB     00:41 ... 
downloads/kernel-3.3.2-5.fc17/kernel-devel-3.3.2-5.fc17. |  15 MB     00:05 ... 
-------------------------------------------------------
kernel-3.3.2-5.fc17 is now available for install.
[jwboyer@zod koji-bisect]$

Now in the downloads/ directory you have a shiny kernel to install and test. So you do that, but it still didn't work. You can either just call the script with --bad on that new test version, or you can take a look at what's left first (or both!):

[jwboyer@zod koji-bisect]$ ./koji-bisect.py --list
kernel-3.3.0-8.fc17         good build
kernel-3.3.1-1.fc16
kernel-3.3.1-1.fc17
kernel-3.3.1-2.fc16
kernel-3.3.1-3.fc16
kernel-3.3.1-3.fc17
kernel-3.3.1-5.fc16
kernel-3.3.1-5.fc17
kernel-3.3.2-1.fc16
kernel-3.3.2-1.fc17
kernel-3.3.2-4.fc16
kernel-3.3.2-5.fc16
kernel-3.3.2-5.fc17         current build
kernel-3.3.2-6.fc16
kernel-3.3.2-7.fc17
kernel-3.3.2-8.fc17
kernel-3.3.4-1.fc16
kernel-3.3.4-1.fc17
kernel-3.3.4-3.fc16
kernel-3.3.4-3.fc17
kernel-3.3.4-4.fc17
kernel-3.3.4-5.fc17
kernel-3.3.5-1.fc16
kernel-3.3.5-2.fc16
kernel-3.3.5-2.fc17         bad build
[jwboyer@zod koji-bisect]$

You'll notice that there are f16 builds listed as well. That is actually on purpose, as we grab ALL the builds and sort them via RPM Epoch-Name-Version-Release ordering. In our contrived example, we don't really need the f16 builds, but often times we have to install builds from different releases in order to continue bisecting. E.g. if you hit a bug in a kernel-3.3.0 rebase on F16 you'd need to use the -rcX builds from F17. Anyway, if you don't want to see them you can use the --dist option to limit what's used based on disttag.

So we mark the test one bad (or good), and rinse and repeat until eventually the script tells you:

[jwboyer@zod koji-bisect]$ ./koji-bisect.py --list
kernel-3.3.0-8.fc17         good build
kernel-3.3.1-1.fc17         current build
kernel-3.3.1-3.fc17         bad build
[jwboyer@zod koji-bisect]$ ./koji-bisect.py --bad kernel-3.3.1-1.fc17
Marking kernel-3.3.1-1.fc17 as bad
Build kernel-3.3.1-1.fc17 is the first bad build
[jwboyer@zod koji-bisect]$

You then dutifully tell the Fedora kernel team that kernel-3.3.1-1.fc17 caused your regression and we go off scratching our heads.

There is a lot more that can be done with this. Perhaps we can get it so that once it find the first bad kernel, it does a vanilla build of that to see if it's a Fedora patch that broke something. And if that vanilla build fails too, maybe even dip into a git bisect. However, all of that is stuff that can be added over time.

So if you're interested, grab it from git and give it a whirl. If you want to submit patches, send them to me! If you want to ridicule my python skills, or how simplistic the approach is so far, do that too! I'll either use it as constructive criticism to improve things, or as fuel for my hate-powered motivational engine. I win either way.

Now, excuse me while I go off and try to figure out a bug I found while trying to do the example demo.
link2 comments|post comment

What's in a version? [Apr. 24th, 2012|01:44 pm]
[Tags|]

Some of you may have noticed that Fedora 16 and Fedora 17 basically have the same kernel. The astute of you might have also noticed F15 does as well [1]. All three main Fedora branches are currently sitting at the Linux 3.3.2 release, with Linux 3.3.3 already committed to the git repository. So if they're all based on the same kernel version, that means they're all identical, right? And if so, why bother upgrading, right? Well, not quite. There are some differences, and I thought I'd take a minute to explain what those are and why.

The base vanilla kernel version is, as previously mentioned, the Linux 3.3.3 stable release. F17 has been on the 3.3 kernel for quite some time now. F15 and F16 were both rebased to 3.3.0 approximately a month ago. Normally we would have waited for the 3.3.1 release before rebasing, but two factors made us decided to rebase on the initial release. The first was that the entire wireless networking stack and drivers from the 3.3 kernel were being used in the F16 3.2.x series of kernels via the compat-wireless mechansim. A bunch of bugs were filed and a large number of them were fixed, so we were fairly comfortable with that in the 3.3 series. The second factor was that we have been having a fairly decent experience with the rest of 3.3 in F17. That isn't to say we didn't hit issues [2], but they weren't "ARGH, MY COMPUTER IS ON FIRE" kinds of things.

Ok, now for the differences. First and foremost, the config options in F15 are much more restricted. For the most part, we have stuck with the config settings that were there when F15 first shipped. F16 has a number of changes (old drivers disabled, etc), and F17 has even more. Given that nobody seems to be using F15 anyway [3], I'm going to focus on the differences between F16 and F17.

F17 has:

- A much larger number of config options turned off. This came from the great kernel module purging of FUDCon 2012 that Dave did.
- A smaller number of extra options enabled. This includes:
    - Support for GMA500 and GMA3600 graphics (poulsbo and cedartrail)
    - SCSI virtio drivers
    - The r8712U wireless driver
    - A number of ARM specific changes
    - NR_CPUS set to 64 instead of 256 (because, honestly, do you really have 256 CPUs?)

There are also a number of additional patches that were added to F17 for various reasons. These include:

- Mac support:
    - Various HFS enhancements to make booting on Macs easier
    - The Apple GMUX device driver
    - VGA arbiter support to allow switching of default devices on multi-GPU systems
- Graphics:
    - RC6 is now enabled by default with the i915 driver for better power management
    - The Virtual GEM provider driver is added to allow Adam Jackson's work on llvmpipe software rendering to work.
- The new NFS idmapper support is enabled by default
- SELinux has a 'ptrace_child' capability that allows processes to ptrace their children if the "deny_ptrace" option is set in the SELinux policy

F17 also has grown the kernel-modules-extra subpackage, which contains a number of lesser used or esoteric device drivers. Overall this has worked well enough thus far in F17 that we plan on keeping it for the final release.

The only change I'm aware of that isn't suitable to go back to F16 is the NFS idmapper changes, as the userspace utilities in F16 cannot cope with that.  However we try to keep a stable release, well, stable so we've avoided adding the extra goodies.  Besides, you all need reasons to upgrade to F17 when it comes out.

If you have questions or comments, feel free to ask either here or on the Fedora kernel list.  We'd be happy to discuss things.

[1] F15 is on 3.3.3 in git, but we're still sticking with the 2.6 naming because its userspace is old-ish and we don't want to break the world by doing proper kernel version naming.  So it's called 2.6.43.3.  Woo.

[2] The biggest issues with the 3.3 rebase in F16 were 1) ALSA regression for various Thinkpad models, 2) Some machines didn't boot because of an overlooked error condition in the memblock subsystem, 3) the ath9k driver didn't work in 3.3.1.  Item 3 got fixed by 3.3.2, item 2 is fixed now, and item 1 should hopefully be fixed with the 3.3.4 stable release.  If not, upstream has been very responsive so we'll continue to work it.

[3] We have gone through the entire F15 3.3 rebase without having a single 3.3 kernel pushed to the stable updates repository.  They keep getting submitted to updates-testing and never getting enough karma to make it out of there.  We've even set the stable karma value at 1 and it doesn't get promoted.  That includes security fixes as well.  Either people _really_ like 3.2.12, nobody is using F15 anymore, or if they are they can't be bothered to log karma on the updates.  I choose to believe that people are smart and upgraded to F16 by now.
link1 comment|post comment

FUDCon Day 2 [Jan. 15th, 2012|05:15 pm]
[Tags|]

Ah, Barcamp. I had never heard of a barcamp before the first FUDCon I attended, and I really like this format. Having the 30 second pitches for presentation topics is great and the voting means most people wind up satisfied with some kind of talk they want to attend. I think it also puts some of the onus on the attendees because if the conference is going to go over well it needs to have great participation from the members.

After the barcamp presentations and voting, Jared Smith gave a nice presentation on how Fedora as a project has done over the past year and where he thought we might improve. I found the year review particularly helpful because I was essentially a ghost in the project until mid-July. I agreed with most of his points on where we could improve, though the picture on the last slide reminded me of a different area we don't focus on other than what he said. He used a picture of a classroom as a metaphor for "mentoring" and while I do think we can improve there, it immediately struck me more as "schools". I think we reach adults well, but we really don't even try to introduce high school age students to open source. There are some notable exceptions, but that is mostly because they found Fedora and not due to any outreach from us. Anyway, I'm sure there are any number of areas where we can improve.

After Jared's talk the barcamp sessions started. I attended a Secondary Arch talk by Denis Gilmore, which went well. Then lunch followed, and then a bit of spare time to review some kernel changes and patches. After the small break, I attended Dan Walsh's talk on Secure Linux Containers. I found this to be really interesting and I'm looking forward to seeing what Fedora 17 brings in this area.

Next up with the fairly traditional State of the Fedora Kernel talk. Dave Jones lead the session with a review of where things are at with each release, what problems we're seeing, and what we're intending to do with the release branches from a broader picture. I chimed in here and there and I thought the "presentation" part went fairly well. Having multiple people to cover things adds some different perspectives and provides a bit of "backup" in case you start to get brainlock. Then we had questions and we tried to answer them to the best of our abilities. This lead to an interesting discussion on hibernate and how it appears to be rather broken right now, both in Fedora and upstream. We're still unsure what to do there, but something will need to happen before long.

After the kernel pitch was over, we stayed in the same room to catch the Security BoF. That wound up being a great overview of some of the current and upcoming security items being thought about for Fedora. As usual, Dan was great and and Eric Paris provided some good insight as well. Then it was off to the GPG key signing where we had about 35 people show up. It went rather smoothly and then it was back to the hotel briefly before FUDPub.

FUDPub was in the student center. The bowling filled up quickly and I'm not very good at pool, but they had a great TV room with video games and football on. Sometimes after a long day of trying to cram a bunch of new information in your head, you just want to sit down for a bit and turn off your brain. Football is great for that and Tom Callaway, Bill Nottingham, Jeff Darcy, and a bunch of others were in there as well. When the first game ended, it was back to the hotel.

The hotel proved to be a great "hallway" track as well. There was a lot of laughter, a well played prank by Adam Williamson, and we capped the night off with a wonderful, uh, light show. It's a bit hard to explain but definitely a great time.
link1 comment|post comment

FUDCon Day 1 [Jan. 14th, 2012|11:05 am]
[Tags|]

The first official day of FUDCon was hackfests and workshops. There were a few official ones to choose from and several impromptu gatherings in the empty rooms. Most of my day was spent wandering between rooms and meeting various people. Then I decided to take advantage of the "hackfest" nature of the day and spent some time updating all three releases to the latest stable kernel release. Fedora 15 and 16 got updates to 3.1.9 and Rawhide got a brief update to 3.2.1 to hold us over until we start rebasing to the upcoming 3.3 kernel.

After a brief stop in the Secondary Arch workshop to say hi to a few people, we headed out for lunch. After we returned, Dave Jones and I had a conversation about a lot of the random obsolete and widely unused options and drivers we have enabled in the kernel still. Apparently Dave was feeling particularly inspired and spent the rest of the afternoon taking a chainsaw to our config settings. The result of his mania should be in Rawhide today. In general, this shouldn't really be noticeable to anyone unless they decide to look at the installed kernel size. However, in the event you update today and something breaks please let us know. If you're at FUDCon, you can even try and track us down but we might hide from you.

After the days activities were done, it was back to the hotel and then attempting to make dinner plans. That resulted in Seth Vidal and his brother Logan, Matt Domsch, Dave Jones, Mike McGrath, Bill Nottingham, and myself heading to a local Indian restaurant. Now, I live in a place where the diversity of the local cuisine is rather... limited so I've only eaten Indian once or twice before and have never been impressed. This time I thought at least I was with people who could steer me in the right direction. I went with the chicken vindallu and it was delicious. Of course, it was 8pm before we got our food so I probably would have eaten anything but it really was great.

Back to the hotel, and then a beverage with Justin Forbes, Peter Jones, and Bill and off to bed. FUDCon barcamp and FUDPub is today, so it is going to be a long one. Dave and I are giving a Fedora Kernel talk at 15:00 today, so stop by room 230 if you want to hear about the kenrel or have questions.
linkpost comment

FUDCon Day 0 [Jan. 13th, 2012|05:26 pm]
[Tags|]

The day of traveling to FUDCon is really pretty boring to read about. I mean, who really wants to read about me walking through airports, getting annoyed at old people that insist on carrying on bags they can't even pick up and put in the overhead bin, and having to go to the bathroom as soon as the plane takes off? Nobody, right? Good. We'll skip all that and get to the part where I actually spoke to people instead of standing in the corner like the scary lumberjack guy with a look that said "I don't know you, go away."

Spot arrived at the airport to pick up about 12 of us in a rather large van, which was of course immediately dubbed the FUDBus. We piled in and headed off to the hotel. Once we arrived, I met up with Dave Jones and we proceeded directly to the pre-FUDCon "bar"camp. Much good discussion was had about all kinds of kernel related things. A bit later Dan Walsh and Eric Paris joined us, then Nathaniel McCallum. As usual, it was great to put some faces to names and then I crashed.
linkpost comment

(no subject) [Dec. 16th, 2011|12:28 pm]
[Tags|]

It's been a while since I posted about the overall state of the Fedora kernel.
Seeing as we're winding down for the year, I thought I'd take a moment to
briefly discuss where things are at, what trends we're seeing, and maybe what
people can do to help.

Fedora 14 is now EOL. That means there will be no further updates pushed, which
also includes no security updates. The final F14 kernel was
kernel-2.6.35.14-106.fc14. We're really not all that sad to see it go. The
2.6.35.x kernel was showing it's age and it's 'longterm' status seems to be in
limbo of sorts. If you're still clinging to F14 for one reason or another, we
strongly encourage you to upgrade to F15 or F16.

Fedora 15 and Fedora 16 are now based on the latest stable releases of the 3.1
kernel. At the moment, that is 3.1.5, which is of course called 2.6.41 in F15.
Keeping them on the same kernel version helps keep down the maintenance burden,
and allows us to remain a bit more relevant to upstream when working on
problems. We're trying to be a bit cautious when updating, so we tend to update
F16 first and wait until that update moves from updates-testing to updates
before we submit the similar update for F15. So far, that has worked fairly
well in preventing surprises from popping up on the most stable release. I hope
that trend continues.

Rawhide is chugging along with the -rc releases of what will be the 3.2 kernel.
At the moment, there is nothing earth-shattering that I can think of queued up
in this release. It's the usual set of fixes and enhancements. The two major
features users might have seen from it are the inclusion of the brcm80211
wireless drivers into the mainline tree, and support for the ALPS touchpads that
are found on various Dell (and other) laptops. However, we've already
backported those to the F16 kernel. Overall, the 3.2 release seems to be
shaping up fairly nicely.

That's not to say we don't have problems. There are the usual set of odd bugs
that seem to hit here and there but aren't prevelant. That is somewhat expected
and par for the course these days. However, we seem to have some certain areas
that are causing a larger number of users problems.

The Intel wireless drivers in 3.1.x and 3.2 seem to be hitting an unforunate
string of problems. Sometimes they don't work at all, sometimes they oops,
other times they appear to work but then stop. There is a lot of work going on
upstream for these problems, and when they are reported they get fixed rather
quickly due in no small part to the efforts of Stanislaw Gruszka. Hopefully
things will quiet down on this front soon.

An area where we seem to be having lots of issues is hibernate. Sometimes
machines that used to hibernate suddenly don't, or they can't resume from
hibernation. However, it seems to mostly manifest itself in filesystem or
memory corruption after resuming. Eric Sandeen has been scratching his head
over bug 744275 for a while now, trying to come up with some kind of common
trends so he can recreate. If you see this bug, please read over his latest
request in the bug and let him know.

Suspend is in a similar state of affairs. Machines are either failing to
suspend or failing to resume. While this used to be a fairly common thing a
few years ago, the suspend path had gotten fairly reliable. Seeing a rash of
problems now is somewhat of a regression, one that we hope to reverse. The
biggest hurdles are that suspend (and hibernate) issues can be fairly hard to
debug. We have some tips here: https://fedoraproject.org/wiki/KernelCommonProblems#Suspend.2FResume_failure that can help isolate which driver/device is
causing problems. Those are always worth trying if you are having suspend
issues. Aside from that, sometimes it takes a kernel bisect to figure out the
commit that caused the problem.

What can you do to help? First and foremost, report the bugs. The kernel
has a ton of bugs reported against it and while I might seem insane for
encouraging even more bugs to wade through, it's necessary. We can't fix bugs
that we don't know about. So if you have a bug you've been ignoring because
it's not really hampering you in a major way, report it anyway. You might get
lucky and have a relatively easy bug to fix.

Another helpful thing is helping other users in the bugs. Do you feel fairly
comfortable with git, installing kernels, and looking at kernel oops output? If
you do and you see a bug that is fairly contained (e.g. 3.1.4 works but 3.1.5
doesn't), then you might be able to help that user do a git bisect to figure out
which commit causes the issue. That is often the biggest hurdle to get over for
a bug, so any help in that regard is most welcome.

And of course, running the rawhide kernel and working with upstream for bugs you
see is always helpful. The more bugs found and reported during the release's
development window, the fewer users are impacted when Fedora incorporates that
release into the distro. Working with upstream eliminates the middle man and
improves kernel quality for everyone. I really wish we had more rawhide kernel
users.

There I go being all long winded again. Hopefully this at least gives a good
overview of where things stand and what we can work on. If you have questions
for the kernel team, feel free to email the Fedora kernel list. For those that
are able to attend, we'll also be at FUDCon Blacksburg in January so stop by
and see us. (NOTE: Matthew Garrett is unable to attend. Magic laptop fixings
on-site will not be possible at this FUDCon.)

---

Do you enjoy digging into a variety of problems across the kernel subsystems?
Have you worked with the Linux kernel extensively? Are you interested in
working as a Fedora kernel maintainer? Then you might be in luck. We have an
immediate opening on the Fedora kernel team. If you think you would be a good
fit, please send a resume to Tom 'Spot' Callaway, Dave Jones, or myself.
linkpost comment

And then there were two (kernels) [Nov. 7th, 2011|08:57 pm]
[Tags|]

The Fedora kernel team resolved a few weeks ago to get back to rebasing the kernel to the latest stable release as soon as reasonably possible. There are multiple reasons for this, but primarily we get two main benefits. The first is that each new kernel release typically fixes a number of bugs and supports a wider variety of hardware than the previous release. (Yes, there is the possibility of regressions. We're aware of this.) The second is that the team has fewer kernels to support across all of the releases.

In keeping with this resolution, davej committed the rebase of F15 to the 3.1 kernel today. We kept the 2.6.4x naming that was started with the 3.0 rebase, so this naturally becomes 2.6.41. That means F15 an F16 are both based on 3.1 at this point, though I will point out that we won't be pushing 3.1 into F15 until 3.1.1 is released and built on F16. So now we're down to two kernels to focus on in Fedora: 3.1.x for F15/F16 and what will be 3.2 for F17/rawhide.

Now, I know I will get comments about stable releases and large version bumps and regressions being introduced (see above where I already said that). These are all fine points, and they are not to be dismissed out of hand. However, we simply get more bugs fixed by rebasing than we ever would if we did not. The number of bugs that were closed when F15 went from 2.6.38 to 3.0.x was significant. I can think of perhaps 3 regressions that came from it, only one of which really gave me any sort of pause.

And here's another kicker that most rebase critics seem to dismiss but is, in my humble opinion, extremely important. Upstream kernel developers don't want to look at problems on stale kernels. They might say they care, and they go through valiant efforts to get stuff backported and into -stable releases, but let's be honest. They are off working on the stuff that will go into linux-next (which isn't even what is supposed to go into the not-yet-released kernel), fixing the bugs they missed after the not-yet-released kernel merge window closes, and trying to make sure the developers and users sending in reports to the various mailing lists aren't being ignored. They are in a word, busy.

Asking upstream about something seen on a kernel that is 3 releases old isn't likely going to get very far, and if it does the first question asked is usually "Can you hit this on kernel?" Keeping Fedora close to where the developers are working is, again in my opinion, quite worthwhile. The code base is relatively fresh in the collective upstream mind, and we're much more likely to trigger "oh... that might be XYZ" responses than hear "2.6?? I can't even remember the last time I booted a 2.6 kernel."

So anyway, I'm pretty relieved we're getting back to doing frequent rebases. I'd like to start gathering some more statistics on how it goes as well, so that we can measure the impact and make sure we aren't turning "Onward!" into "Unsafe at any speed". It will be doubly useful because some amount of extrapolation can be done to reflect on upstream as well. I know davej has some bugzilla statistics, but I'm not sure they're quite as fine grained as they could be to track this particular thing. I think I found my next side project.

(Some of you may have noticed I keep saying 2 kernels, and I don't mention the fact that F14 is still on 2.6.35 which would really mean 3 kernels. Well... you remember that guy that from Goonies that lived locked up in the basement? Strong, loveable, but a touch slow in the head? Yeah. That's basically F14 at this point. We feed it a candy bar in the form of a security fix every now and then, but that's about it. When it goes EOL in December, I won't really miss it.)
link2 comments|post comment

I might have a problem.. [Oct. 28th, 2011|12:45 pm]
[Tags|]

Recently the Fedora PPC secondary arch team announced an F16 Alpha release. Now, I don't really have a whole lot to do with this. I started the PPC secondary arch effort back in the F13 timeframe, built most of F13 and F14, got some new builder machines acquired, and then basically got busy enough that I was happy to hand it off to someone else. Karsten, Phil, Brent, and Mark all did a great job working through building and getting the images created. Kudos to them.

Since this shiny release was sitting there, and my iMac G5 was still plugging along on F12 I decided I should install it. Burned a DVD, popped it in, and was greeted with the you don't have enough memory error. Well, anti-climatic. Then I got a crazy idea. What if I could make it look like my machine had more memory than it did? So then I proceeded to figure out what anaconda was looking at to determine it, and devise a plan to hack around the check without rebuilding anything. (Like I said, crazy.)

It turns out that this is entirely possible. I let anaconda boot, error out, and then I switched to tty2 which is where they leave a nice shell for you. Then I █████ █████████████, edited it, and ████ mounted it again. So the 'memory' problem was solved, but I needed to get anaconda to run again without rebooting. This is fairly non-obvious but eventually I figured out you had to call ███████████. Except if you do that, it complains that it has already been run and starts yet another shell. To avoid that, you have to remove ███████████████. Did that, called it again and anaconda started. Woot, right? Sort of. Turns out it didn't switch the controlling tty to tty1, so I had to figure out how to get input to anaconda to get it to do the steps. This involved reading the anaconda screens and switching to tty2 to type in whatever was needed.

After a lot of ctrl+alt+{F1,F2}, it finally started a minimal install. This eventually completed and left me with a nice minimal install. All that was left to do is install a bunch of packages from the DVD and reboot and things worked nicely.

A morning spent reviving a dead machine. For really no other reason than because I can. I should probably get help.

NOTE: The steps used here were edited to protect the author from being tracked down and beaten by the anaconda developers for teaching people how to do things that are guaranteed to cause bugs that they'll never be able to fix.
link2 comments|post comment

Rawhide kernel rebased to 3.2 [Oct. 27th, 2011|07:51 am]
[Tags|]

The rawhide kernel has been rebased to the initial wave of commits that
will eventually become Linux 3.2. I haven't built the kernel in koji
yet, but I will probably do so tomorrow or Monday, depending on how well
a local build works when I test it.

There was a large driver movement in the ethernet drivers that caused a
bunch of config option changes. I think I got them all right and we
should still have all of the ethernet drivers we had enabled before
still being built. However, if you upgrade and notice a driver you were
using is now missing please file a bug and let us know!

Aside from the ethernet driver movement, there are of course a wide
variety of changes and fixes coming in. The linux-next tree for 3.2
submission was the second largest ever, weighing in around 11,000
commits that need to get merged. This is mostly due to the kernel.org
compromise causing the 3.1 release to drag on. Fortunately, Linus is
being a bit of a stickler on what he will pull for 3.2 for a variety of
reasons. Hopefully that helps us as we continue to bring in snapshots
of the merge.

Hang on, it might be a bit of a bumpy ride :).
linkpost comment

navigation
[ viewing | most recent entries ]
[ go | earlier ]