This week in Anaconda

Posted on May 16, 2011 by Chris Lumens in work.

Over the past two weeks, I’ve been working on figuring out where all the memory goes during installation. As many people have no doubt noticed, we’ve increased the memory requirements a good bit for F15. This is mostly due to my single large initrd change. wwoods is working on further refinements for F16 that should lower that requirement by a good bit. But, that alone doesn’t tell the story of install-time memory usage.

The first thing I needed was data. So, I wrote a little program to poll /proc/meminfo and record statistics, as well as check which programs are using the most memory. A little experimenting showed my initial frequency of once every ten seconds was not at all fine-grained enough, so I cranked it up to once per second. My graphs were still showing some bizarre behavior towards the end, so I bumped memory available to the VM way up and got rid of swap to remove a variable.

All those changes got me a graph that I was happy with. So then I set out to perform a full install and get good data. I tested a VM with 2 GB of memory, installing from a DVD, doing media check, changing language to German in the GUI, and doing the default package set. Here is my graph, annotated with times at the obvious spikes:

Apologies for the tiny graph thumbnail there. Please just open it in a new tab to see the enormous one.

The A and B spikes are due to anaconda running localedef during installation. Turns out it’s much smaller to ship the source files for locale data and build them at install time than it is to ship the compiled locales. Spike A is us doing localedef during loader for English, which seems unnecessary. Spike B is because I selected German later on in the GUI.

Spikes C through H are various packages running update-mime-database as part of their %post script. These aren’t big spikes, but they’re noticeable enough in an otherwise boring stretch. It seems to me like this could be run once as part of something’s %posttrans script to save a bit of time and effort.

The enormous spike is I, followed by J. These are both from the selinux-policy-targeted package’s %post running semodule. I have some data on this by running the script through valgrind, but that’s a topic for another day.

Spike K is firefox (or something else, but firefox certainly does this) running gtk-update-icon-cache from a %posttrans script, and spike L is the kernel running depmod out of its %posttrans as well. While both pale in comparison to what’s happening earlier, it seems like both are using an awful lot of memory for what should be simple tasks. Some investigation here is warranted.

Two things are unlabeled. First, the graph starts out with a baseline usage of somewhere around 175 MB. This is due to the giant initrd being loaded into memory. Doing so sets a pretty high base, but like I said earlier, it’s being worked on. Second, there’s a large climb starting at 20:47:00. This is repo setup and dependency resolution. As you can see, the memory consumed there is never freed so the big selinux-policy-targeted spike is made even worse. This is either anaconda or yum doing something goofy.

Update: At least part of this is rpm - not yum. After some further tracing, I determined approximately 100 MB of memory usage comes from the populateTs call in anaconda’s yuminstall.py. In yum, that doesn’t do anything particularly interesting so I dug down into the call into rpm. It looks like the memory is getting consumed by storing all the provides and file lists from each package in memory. I bet this portion of the graph would change with the number of packages to install. I don’t yet see a way around this.

Also unlabeled is the fact that everything between about 20:47:00 and 21:03:00 is out of anaconda’s control. This time is repo setup and depsolving (which is largely yum) and then package installation (which is whatever crazy things each individual package wants to do).

So, where do we go from here? First, decreasing baseline memory usage will do a lot of good. Second, figuring out what can be done in semodule is critical. Until we figure that out, there’s not much point in doing anything else in anaconda. That is far and away our biggest driver of memory consumption and it’s not even under our control. If you have swap configured, it’ll be getting hit like crazy at this point. Third, we need to figure out if depsolving memory can be freed at any point. That would also help lower the severity of spike I.

I suppose it’s also worth testing a minimal install and an everything install to see what other crazy packages there are. It’s also worth me getting this patch committed to anaconda so we can occasionally re-run and make sure we haven’t introduced any new problems. At the least, seeing what happens after we’re no longer using the giant initrd will be interesting.