Archive for January, 2008

Use flags

Saturday, January 19th, 2008

This is something that’s been talked about for a long time, but never been implemented. Now with general agreement that it should happen from list discussion and the developer meeting this morning, we need to think about how. I’ve volunteered to take the lead on this one, so I’d like to write up my ideas for how it should go.

Firstly, I want to have all the flags signalled in (Build)Dependencies. In most cases, the flags will be associated with dependencies, but for those that aren’t (relating to enabling/disabling bundled libraries, say), I want them to be specified on their own. To that end, I’d like to redefine the rule format to consist of a current dependency specification, followed by a list of use flags that activate the rule: GTK+ >= 2.8 ?gtk, gui. I’ve chosen a question mark to offset the flags here, but the specifics aren’t important – it’s necessary to mark them somehow, so that they can be listed on lines of their own for the no-dependency cases. It’s possible that we will want more complicated rules in the future (?foo && !bar), but I think this is enough to start with.

Storing the flags in Dependencies means CheckDependencies can handle all the work transparently to the rest of the system – Freshen and Manager will automatically get the correct list of dependencies for the enabled flags, and ChrootCompile will build the environment correctly. It’s due to be merged into Compile itself in this cycle too, which makes this feasible.

Secondly, the Recipe file: we like to keep the format declarative for the common cases, so I plan to create some magic variables to deal with the usual situation of just adding a configure flag. with_gtk="--with-gtk=$gtk__path" would add that configure flag; in many cases, that wouldn’t even be necessary, because it would be detected automatically within the chroot environment. For the more complex cases, a function using_gtk() would be defined in the recipe to perform the necessary customisations. If it turned out there were a lot of those functions doing a particular task, we could add another variable to handle it, in line with the recipe philosophy“The plan is to make Compile as smart as it can and the recipes as small as possible.” – Hisham, 2004. Most recipes would only need the variables, if anything, above mentioning the flags in Dependencies.

When a recipe was compiled, a file Resources/UseFlags would be created listing the flags enabled for the build. This would allow finding which packages needed to be rebuilt to take full advantage of changes in the flags on a system. It would also allow recipes to test for the flags enabled in their dependencies, if the need arose, and prompt for recompiling the necessary package. I’m not planning on implementing that at this point, but rather adding functionality as we need it.

We would need to adopt a default set of flags to use for binary packages and ISO releases, which I can foresee a little debate over. I guess they’ll be largely determined by the packages we fit onto the ISO. That could be interesting if we do end up with larger LiveUSB or DVD releases, though. In any case, choosing the flags to use is almost the entire problem. ChrootCompile will make building packages with them simple, and with any luck the work someone (AndrĂ©?) has been doing on distributed package building will remove the bottleneck we’ve had there in the past.

I’m excited to be finally getting these in to the tools; they make the recipe system that much more powerful, and they’re something I’ve been hankering for for a while. I expect we’ll have some more discussion about how to implement the flags, and probably change things somewhat from what I’ve outlined here – I gather some people want a more procedural system, or a greater level of assumption about the flags, and there’ll probably be a bit of give and take over that. I heard whispers of an interesting Makefile-like approach at one point, which I’d like to hear more about. Regardless, I’ll be glad to see them make it in in whatever form, and I’m looking forward to getting the support into Freshen as well (more news there sometime too, I imagine).

Freshen goes Python

Monday, January 14th, 2008

There will be a new release of Freshen, the package management soon (meaning, in the next few weeks). The old code was starting to run into its limitations, and especially with the new Compile functionality planned for the future it was time to start afresh. It was showing a lot of the heritage of the original Perl version, that you actually piped the output of find into – there was a lot of guesswork built in to deal with the assumptions made then.

Given that I was going to have to re-implement things anyway, it made sense to write it in Python and use the same versioning and dependencies code used by the other scripts. I mentioned that during the 014 postmortem on IRC, and with some help from AndrĂ© in explaining the scripts I have an early working version in CVS now. It depends on some functionality introduced in CVS Scripts to have any reasonable performance, but you don’t have to be runningt them on your system – just add a symlink Freshen/lib/python2.3/ -> Scripts/bin/CheckDependencies to have the new code used. Try it out – bug reports are welcome.

There are some architectural changes in the new codebase (aside from the obvious). Most importantly, it no longer produces a graph of programs on your system. This has a number of implications, perhaps most notably that there will no longer be topological sort failures when the graph is not acyclic. The list is instead resolved program at a time and joined. This means that running multiple (distinct) queries on the same system state won’t have the same speed it used to, but examining short lists should complete much faster.

There is a cache of generated lists in ~/.Settings/freshen_cache, which makes consecutive identical queries nearly instant (especially useful in the list, review, and upgrade case). It only lasts ten minutes, but you can force the use (-c) or non-use (-C) of that cache on the command line. The cache varies by generation mode and the list of examined programs. For the moment, there are two modes: --thorough, and the default; the difference being that the default does not include revision-only updates. There may be more in future.

There will also be other features in the future; I plan to have rough feature-compatibility with the 2.x series, at least in as far as they’re still useful. Another more technical point is switching to a generator-based updates list, as soon as I can figure out how to cache that well and a good progress indicator; it should feel a bit speedier that way.

Again, you’re encouraged to try it out – check it out from CVS with the instructions on Savannah, and run it with bin/Freshen from the tools/Freshen directory (making the symlink mentioned above if necessary). It is usable, but I don’t suggest doing a whole system upgrade with it just yet – keep an eye on it while it works.

More posts to come as development proceeds.