|Fedora updates pushing: Behind the scenes
||[Jul. 24th, 2009|03:56 pm]
Just before F11 release, we enabled deltarpms for updates. There were some bumps in the first few days, but we got through it and the people rejoiced. Everyone was happy and the Fedora updates world had a victory in terms of end user gains.
Then time went by. Updates kept getting submitted by maintainers, and they noticed they were pushed to users less and less frequently. Some asked on the list, and rel-eng (mostly me) blamed deltarpms. This was not an untruth. Generating deltarpms is a pretty intensive task, and the larger the RPMs in question, the longer it takes to actually generate them. So our illustrious Infrastructure team took note and increase the DRAM and number of CPUs the releng box had. This has proved to be most helpful, and our box no longer gets kernel OOMs if the rawhide and updates mashes happen to be going at the same time. However I still didn't think something was right.
It was taking 2-3 days _per_ push. What does a push entail? Bodhi moves koji tags on the submitted updates, then it mashes f10-updates, f10-updates-testing (no drpms), f11-updates, and f11-updates testing, and when all that is done it performs the repomd.xml, bugzilla update, and announce email steps. Now, that's a lot to do, but 2-3 days seemed really excessive.
The first thing we noticed was that it was taking 6-10 hours to mash just f10-updates. Over time, the updates repo grows and it takes longer to mash than a fresh release. However, 6-10 hours was longer than f9-updates ever took. It seemed about 2x longer than it should take, and there were no deltarpms to take into consideration. So we poked. We pondered. We noticed that all the mtimes on the updates were from the day of the mash, and not from the time the package was created. What did that mean? Well we found out that the releng box had an odd mount situation that prevented the normal hardlinks that occur. That doesn't sound like a huge issue, but it has pretty bad impacts.
First, it means mash has to copy all the relevant RPMs to a different location on the same NFS mounted filesystem. SLOW. Then when the updates push wound up completing, the mirrors would basically be slammed trying to update every single RPM in the push due to the mtimes changing. Couple that with the fact that the master mirrors were having major issues at the same time, and this meant that even our top tier mirrors were continually out of date. Ew.
We fixed that ASAP and noticed an immediate improvement in f10-updates-testing and f10-updates mash times. Right back to the normal 2.5-3 hours for f10-updates. Awesome! Everything was cake and gravy and we were all happy again, right?
It fixed the f10 release updates, and significantly helped the mirror situation, but mashing the f11-updates and f11-updates-testing repos still took forever. The overall push times were now 1-2 days (depending on various factors), which was a good improvement, but it still seemed off to me. So we dug into it more. And sure enough, we found another issue. I noticed that during the f11-updates mash, we were creating drpms for package combinations that already had drpms present. When you take into account that this included huge things like wesnoth-data and openjdk and other large RPMs, it was very easy to see why it was taking so long to mash the repos.
This was a bit baffling to me, because it is obviously stupid to simply not reuse the existing drpms. I looked in the mash code (and confirmed with the author) and it did seem that mash was set up to copy any pre-existing drpms that were still valid so we could avoid regenerating them. But this code wasn't working. We slapped some debug stuff together and figured out that there was a bad assumption in the drpm handling code in mash from day 1. Namely, drpms have their own type of signature in the drpm header and this is what was being compared to the RPM sig. They never matched, so the existing drpms were never used even though they would produce a fully and properly signed RPM at update time. So we worked around that for now.
The next day, updates pushes went from 1-2 days (which was already decreased from 2-3 days), to 8-9 hours. HOURS! We can theoretically do more than one updates push in a 24 hour period now if we have to. This makes it possible for us to be more responsive to items which may need to be pushed out rapidly (security updates for example). It also has the pleasant side effect of not making me want to cry every time I start an updates push or read the LWN.net daily security updates article.
Is everything cool beans now? Well, no. I think there is a lot more we can do to make the back-end of the updates system better. But it's certainly great to see us get back to a state I consider usable.
If you're enjoying the goodness that is deltarpms, and have noticed the improved turnaround time, give some kudos to all those that helped fix things up: Mike McGrath, Ricky Zhou, Seth Vidal and Bill Nottingham (I am probably missing a few people, and if so I apologize. It was a long few weeks.)
I owe all of you a round of beer. Just show up at my curling club and I'm buying.
Appreciate the speedup. It certainly helps a lot for me as a package maintainer and a remix (that uses yum-presto by default)
I translated the article into Spanish. It should be live today at 4pm (CST).
Thanks for writing about the updates process, I found it very interesting!
Not only a great job at identifying and fixing the problem, but also fantastic work making the problem and its solution understandable to me, and the three people in the Fedora Project who understand less about this than I do. Superb!
P.S. Sorry it took me until Monday to get back to my RSS reader and find this genius article.