Posts Tagged ‘GoboLinux’

Scripts 2.10.2 and Compile 1.13.3 released

Tuesday, April 27th, 2010

More releases, all bugfixes this time.

Scripts has a fix for the man directory path in PrepareProgram (actually used by Compile), and handles dependency conversion better. It also includes updates to the database used by the CommandNotFound system as usual.

The only notable behaviour change is a special case in the useflag code: having -INSTALLED in the environment variable will now disable automatic flags from occurring at all, rather than applying the change at the end of the process. This helps debugging of recipes and is useful for targeted compilation, and the previous more consistent behaviour doesn’t seem to have a use case.

Compile has a single direct bugfix, affecting a case where a failed direct dependency would not cause the build process to terminate. It also benefits from the changes made within Scripts.

The packages are on the master now and will propagate to the mirrors shortly. You can use `InstallPackage Scripts 2.10.2` and `InstallPackage Compile 1.13.3` to install them. Please report any bugs you encounter in the bug tracker and request help either on the mailing lists or the forums.

Thanks to all those who contributed to this release, particularly “Baffo32” who sent patches to Scripts.

LCA day 5 and the GoboLinux talk

Saturday, January 23rd, 2010

The last day brought the reason I was really there, my GoboLinux presentation in the very last speaking slot. Before then I made it into Sarah Sharp’s USB 3 talk, the photo management BOF, which was less useful than anticipated, and Rusty’s Wiimote presentation. The latter was both amazing and adorable, and well worth watching when the videos come out. The video of the first time it actually worked left me open-mouthed from both directions. That was the last talk I got to since I spent the next couple of hours preparing and practicing.

My slot was right before the closing ceremony and not too badly populated given that it was against at least two others I wanted to see. I don’t think it went too badly, though I did fixate on minor topics at a couple of points, and there were some good questions at the end. Since we were before the ceremony the cutoff was rigorously enforced and we actually ran out of time for them all, but I did have a chat with a couple of interested people at the stage while I was packing up.

Interestingly, and unexpectedly, an article turned up on TechWorld the next day about my talk and about GoboLinux. Some of the quoting is interesting, but the gist of it is there. The slides are also available (I think they’ll be on the conference site too at some point; at least they were collected on a USB stick afterwards), as is a more formally written-up version of the presentation. It doesn’t have all the content the talk did, since I added a few parts, including the whole Rootless section, in the aftermath of Monday’s Distro Summit discussion. Video should be available in a couple of weeks.

The closing ceremony was good enough, but ran long (and started late, so we didn’t really need to have rushed the questions so much…). Mostly that was to fit in thanks to the organisers and helpers, who really did do a fantastic job, so no complaints. Next year’s conference is in Brisbane. I’m not sure whether I’ll go or not yet, but it was definitely valuable this time so I will consider it. No speaking maybe ever again though. The Penguin Dinner that night was all right but also lagged a bit. The entertaining part of it was the Life Flight Trust donations race, which had a fivefold increase in the donation volume in just a couple of hours. I left early to get some actual sleep, so I’m not sure how high it got by the end, but it was approaching $20,000 last I saw.

I was hoping to find out the total at the open day today, but it didn’t appear to be up anywhere. I did get a lot of brochures and blurbs instead, and there was free stuff everywhere too. I didn’t end up with much of that; it was probably more appreciated by my later companion, who got all klepto (but not as klepto as she wanted to be, which seemed like a wasted opportunity). Most appreciated was the koala pen from the LCA2011 booth, so good marketing on their part. It was cute.

That’s a wrap. Later I’ll write up something on the various community engagement presentations I attended, which I think we can get some value out of. There were some others I didn’t make it into that I’ll watch when the videos become available. Right now I’m appreciating the opportunity to do something non-conference-related for a while.

First talk over at

Monday, January 18th, 2010

I made my first presentation at today, in the Distro Summit miniconf. I talked about the Aliens system, and it wasn’t a disaster, which is a plus. It went reasonably well, though I missed out on a couple of points I was planning to hit since I went off-script a bit. All the important content made it in, and there were plenty of good questions afterwards (& during). The speaker immediately after me had travel problems and didn’t make it, so there was an extended discussion jumping off from my talk that I think covered a lot of useful territory. It was a good miniconf to be at.

I appreciated the copious power points around to keep the battery charged, and the wireless network I managed to get working today (except in the Civic Suite where the miniconf was, annoyingly [edit: and they’ve fixed it! Amazing.]). Kudos to the organisers. The tech people from the conference were also fantastic helping me get the projection up and running, even though it was completely my fault it wasn’t working. He was, in fact, a genius full of information, like he said, so people got to see my fairly information-free slides. Thankfully I’d turned up early to make sure it was going to work.

Those slides are available now, as is a written-up version of it that has more content than I was able to fit into the talk. I had to go through it with a hatchet last night and today to make it fit in the twenty-minute slot and some good parts had to go. Honesty dictated that I couldn’t leave out any of the limitations and drawbacks so it was mostly the rest that got the chop. The main talk is on Friday, and there were a few points that came out of the discussion today that I’ll work into it as well. It turns out everything is being webcast live, which I didn’t find out until after I spoke today, but you can watch Friday’s if you like (Renouf 1 feed, 1545L/0245Z). There’ll be recordings up too, apparently later in the week, which’d be an impressive feat, but at some point over the next few weeks or months anyway.

Now I have to try to plan out what I’m going to go to tomorrow. It’s not going to be easy.

Scripts 2.10.0 and Compile 1.13.0 released

Monday, December 28th, 2009

Another release, and a minor version bump for both tools. The major update in these releases is the inclusion of the /System/Aliens feature in both. There has also been a change in the default mirror configuration that you will need to update your GetAvailable.conf for.

The /System/Aliens system incorporates third-party packaging systems into the system, allowing both recipes and packages to depend on components from these alien systems (RubyGems, CPAN, and LuaRocks at this point). Using these features will require building the newest versions of Ruby, Perl, and LuaRocks — if you aren’t using them, you don’t need to do that, and they should be installed automatically when required. Dependencies of the form “LuaRocks:json” are handed off to the alien system to install.

With the final release, these dependencies are now available for use in recipes in the store. Remember to specify compile_version=1.13.0 in any recipe using them. Contributions of new wrappers for other systems are also welcome.

Another important change here is in the default mirror configuration: has disappeared, and a new mirror has been added to replace it. If you are using you don’t need to make any change, but otherwise you should swap in the new mirror in /System/Settings/Scripts/GetAvailable.conf.

UpdateSettings will prompt you about this change when you install the Scripts package. While the package mirror is now the master it is not included in the configuration by default. The recipe store remains on and there is no need to change your settings.

Other than those two important notes, there are also the usual bugfixes and minor feature improvements. There is support for more archive types in Compile (xz and cpio), a bugfix for Cabal recipes, and a few variables should now be correctly set in some more corner cases.

Many of the changes relate to improving functioning with the /System/Index structure that will be in 015, but most work on that is occurring in the “015” SVN branch. There will be another tools release in the new year (probably another minor version bump) that will incorporate all of those changes and be used in 015. There may also be a 2.10.1/1.13.1 point update early next month to address any issues found in these releases, but not merging the 015 branch.

You will be able to install using `InstallPackage Scripts 2.10.0` when the mirrors sync (assuming you’re not using, but until then you can use:




to update. Be sure to merge the changes in GetAvailable.conf when installing Scripts. Please report any bugs you encounter on the bug tracker and ask for help on the lists or the forum.

Thanks to everybody who contributed code, bug reports, and feedback during this release cycle.

An overview of /System/Aliens

Thursday, July 23rd, 2009

/System/Aliens is a new feature that will debut in the 015 era, helping with the integration of the growing number of third-party package managers into the system as a whole. It’s prompted a few questions lately from those who weren’t around in the dev meetings where it was hashed out, particularly once we decided to submit a paper on it for next year. It’s a really great feature that deserves explanation, so I’ll try to give an overview here with less technical detail than the paper and more GoboLinux-specific information. I have a working implementation locally, and it will go into trunk as soon as the feature slush for 015 is lifted (hopefully, soon). It should be in a Scripts release shortly after that, and possibly on a CD by January.

The goal of the Aliens system is to pull these various systems under the umbrella of our package manager, so everything within them is automatically available and there’s no need for repackaging. There are a lot of these systems nowadays – as well as the venerable CPAN, RubyGems, LuaRocks, Haskell’s Hackage, and myriad others are in play and growing in popularity. Often those are the only way libraries in their respective languages are published now. I can’t think how many times installation instructions have been “use gem install foo-bar“. Ultimately these systems (mostly) use standardised build and installation systems underneath, so we’ve been able to find the underlying tarball URL and create recipes for them, usually just the standard couple of lines. There’s an overhead in doing that though, and they inevitably get out-of-date. There are ways of automatically importing the entire tree and repackaging them up for the distribution, but they have their own complications and force everything to go through the distribution’s package manager, which can mask useful functionality the alien system provides.

The other temptation for users has been to go entirely over to the third-party system for those libraries, setting up their own tree either in /usr/local (in other distributions), or slightly more cleanly inside a /Programs entry here. Sometimes even over the top of distribution-provided files in /usr, which causes no end of conflicts later on. While at least it’s reversible and mildly clean inside a dedicated /Programs entry, it’s untidy and breaks dependency tracking. If a program depends on Ruby-GTK+, that will have to be wrapped in a distribution package and installed – even if it’s already around via RubyGems, and potentially conflicting with it. Aliens aims to avoid that.

When Aliens is around, alien packages can be installed in their ordinary way, using the familiar gem or luarocks or cabal install … command, and any of the functionality usually provided by those systems can be accessed. All the advantages of a separate tree remain, in this case the entire tree under /System/Aliens/{RubyGems,LuaRocks,Cabal,CPAN,…} being given over to the alien package manager to deal with as it likes, and our standard configurations for those languages being set up to look in there as well. Better though, a dependency from a program in the system package manager can be satisfied via the alien system. If a program depends on Ruby-GTK, it need only declare – within its ordinary dependencies – that it requires “RubyGems:ruby-gtk >= 2.10” and that will be satisfied on installation.

That’s achieved through just a couple of hooks into CheckDependencies and the installation tools. A dependency of the form Foo:Bar is recognised as one on package bar within the alien package manager Foo, and CheckDependencies will hand over to the Aliens subsystem to check that. If the dependency is satisfied, nothing more needs to happen. If not, it will remain in the dependency list and will be installed as soon as it is reached, using the alien package manager.

Each alien system supported will have a simple wrapper layer, using a defined protocol for answering queries and receiving commands. This wrapper protocol is simple and extensible so that adding new supported systems in the future will be easy. Some systems may not support all of the functionality Aliens can make use of – they will simply miss that feature, rather than being fully incompatible. At the simplest, a system need only support checking whether something is installed, and installing it.

Dependencies in the other direction, as from Ruby-GTK+ to GTK+ itself, may be handled one of two ways. Firstly, by defining a mapping of such dependencies as part of the Aliens system: this is not an enormous bother, since it need only be done the once for those items actually depended upon, but it is some. Alternatively, the system dependency may simply be specified before the alien one. If other distributions implemented this system (as we’d hope they might), they might prefer one way or the other of those two. We will probably begin with the second and move towards the first as a simpler migration path.

While I’ve made it sound pretty simple, there are a few technical complexities under the rug that are wrapped up by the Aliens subsystem. From a user level though, they get to rely on the distribution not duplicating or conflicting with their installation, while still having the full range of packages available to them. Users who move between distributions, or to *BSD, Mac OS X, or Windows, may also prefer to have the familiar language-specific interface available to them. If they don’t want to know about it, they won’t ever have to see it or notice it exists. For recipe authors, it’s many fewer dependencies they’ll run across and have to wrap up just to get an application up and running. For the developers (who are also in the first two categories), it’s another world-class feature that makes our lives easier by removing the need for coding so many special cases. For the developers of the packaging systems themselves, Aliens aims to embrace rather than extinguish their works, putting their domain-specific knowledge to work in providing the best way to install software in that language. And for authors of libraries, it lets them publish their creations the way they want, with a functional and consistent installation experience for end-users.

If there’s something I missed or didn’t cover, or you have any questions, comments, suggestions, post a comment, mail the list, or grab me on #gobolinux on Freenode (mwh).

Validating Scripts after a fresh installation

Saturday, June 20th, 2009

On #gobolinux last night an issue came up that has appeared occasionally before – the signature format for packages has changed between the 014.01 release and now, so the Scripts package won’t validate when you upgrade. The change was necessary to make for security, but it creates a problem for new installations trying to bring their tools up-to-date.

For those that don’t care for the fuss, and are willing to take the (very small) risk, you can just run SymlinkProgram Scripts 2.10.1 to activate the new package after installation fails with the validation error. Alternatively, you can upgrade piecewise through intermediate releases to get the fullest possible validation. The first stop should be 2.9.1, which will allow validating the latest package but does not include the version-validating code introduced for another problem. That will also be good enough for most people – the chance of a compromised mirror is pretty slight.

However for complete security you would need to validate the entire package as installed using a trusted signature from the CD release. Here’s a quick script to do that – it builds a verifiable checksum and validates it against the live system. The core code is embedded in a signed block using my key, with a small piece of code outside to verify the signature and execute the code automatically (not necessary, but the GPG command line to use the GoboLinux keyring is pretty long, so this makes it easier – you can copy the internal block out and validate manually if you prefer).

HopValidate will validate the Scripts 2.10.1 package using a known good checksum, preserving the chain of trust all the way. For full security, you should review the unsigned portion of the code, which uses the system keyring to validate the rest.

The manual command to decrypt is gpg –decrypt –no-default-keyring –keyring=/Programs/Scripts/Current/Data/gpg/goboring.gpg HopValidate, or –verify to verify the signature only. If you’re using this on Rootless, you’ll need to adjust the wrapper script and the command to use your local path to the Scripts directory – there’s no autodetection to keep the code easily reviewable.

[ed. 2010-01-31: Updated for 2.10.1]