The contents of this website are Copyright (c)2005 by Brian Manning <brian at antlinux dot com>. Please do not reuse any of the content on this website without permission from the author.

AntBuild should end up being a set of Perl scripts that have instructions on how to build and package source software, kind of like a Portfile in BSD.

TODO: look up the format of a Portfile, see how much it has in common with the below formats.

depsh.pl Page :: Application Package Building :: What to build? :: Building the Kernel Package and initrd image :: Windows Packages :: Creating SQZ Package Files getdeps.pl Requirements ::

filemodtest.pl Commands

 perl -T filemodtest.pl --debug all --testfile /usr/bin/ssh

Package Building

• create a set of core commands that would tell antbuild what to do, kind of like a simple scripting language; squash, comment, make, unpack, geniso.
  • Module::Build in Perl
    • Cookbook examples
  • App::Build is for applications instead of just Perl modules
  • Cons is an alternative to Make that's pure Perl
    • Cons FAQ, which includes some practical examples on how to use cons
  • you would feed antbuild files that contain lists of commands to perform to build a single package, a group of packages, or an entire system including generating the ISO image
  • create a command shell that could be used interactively as well
    • use Term::ReadKey for command line completion and history
• how does jigdo-lite scan and copy package files from ISO images?
• are there Perl tools for manipulating ISO images?
• Use PAR.pm to create Perl ARchives for deployment
• use a listing of files in the current package to build a new package
  • observer initrd: ls -lR --time-style=+%s
• use file atime/ctime/mtime to determine if a package needs to be rebuilt; if the time is different than what is already in the package, use checksums to see if it really is different, then query the new package version if it is different and rebuild the package(s)
• make an alsabuild.sh script, that rebuilds the alsa kernel modules and libs/utils if needed (detects for libs/utils to see if they need to be built)
• build UML kernel for imp on eightball using make-kpkg
• use alsabuild.sh script on imp after new kernel package is installed
• use init-style naming convention for packages to determine if/when a package gets mounted after boot:
  • p00-base.sqz
  • p01-modules.sqz
  • p40-networking.sqz
  • p70 and above are not mounted at boottime
• sign packages with gpg key?
• make sqz packages of .wav files, see if they compress as well as encoded MP3 files
• xar - eXtensible ARchiver; uses XML for archive description, and binary data in what's called 'the heap'
• put the commands use to build the package and the commands used to compile the source for the package into the package description file/directory
• Things to add to the package metadata "signature":
  • string with 'sqz' in it (3 bytes)
  • md5sum of metadata (32 bytes)
  • length of metadata. (12 bytes) The length of the metadat does not include these three sets of values
  • when appending the package metadata to the end of a =.sqz= package file, md5sum the original package file and also =md5sum= the metadata as well, then add that number to the end of the metadata (an md5sum is a fixed 32 bytes), and make the the length number for the size of the metadata as the last 'object' in the .sqz file.
  • calculate the size in bytes of the package description file, append the size of the package description to the end of the package file as a block of 10 bytes (0000000000)
• write an SQZ explorer that runs as an applet that autoruns from the CD ROM. It should allow users to explore (and repackage?) sqz packages from the CD-ROM on macs/windows/linux
• look into using prelink ( debian package source at http://packages.qa.debian.org/p/prelink.html) to speed load times
• strip all binaries strip -s (strip all symbols)
• download Sid packages from Debian mirrors, freeze the packages at a certain date and then use the downloaded files for new image buildouts from then on

What to package?

• the Perl module File::Magic is a wrapper around the Unix file(1) command, and could be used in determining what each file is (script, binary, library, text file, etc.)
• File::MMagic is another Perl file type detection tool, uses an external magic file
• File::Type

 %libs = {
 libfoo.so => libfoo.so.0.0,
 libfoo.so.0.0 => package::libfoo,
 libbar.so.0 => libbar.so.0.0,
 libbar.so => libbar.so.0.0,
 libbar.so.0.0 => package::libbar };

• Cygwin has a section on what makes up the contents of their packages.
• have the package file layout format include the type of file that needs to be created, the location on the image (or device major/minor for devices) and the location on the ramdisk image where the symlink/device/FIFO gets created

 file:uid:gid:perm:/path/to/file/on/image:/path/to/file/on/ramdisk
 sym:/path/to/file/on/image:/path/to/symlink/on/ramdisk
 dev:uid:gid:perm:type.major.minor:/path/to/device/on/ramdisk
 dir:uid:gid:perm:/path/to/dir/on/image:/path/to/dir/on/ramdisk
 hard:/path/to/file/on/image:/path/to/hardlink/on/ramdisk
 fifo:uid:gid:perm:/path/to/fifo/on/ramdisk
 sock:uid:gid:perm:/path/to/socket/on/ramdisk

• Try the BuildingInitRamFs page, where the gen_init_cpio binary can build initramfs CPIO archives.

Kernel Packages and initrd images

The plan is to have a package description file/directory with scripts and instructions that will automate the building of a kernel and an initrd package. The scripts could be reused on a running installation for repackaging.

• also see the BuildingInitRamFs page, where the gen_init_cpio binary can build initramfs CPIO archives.

For a given release kernel, create a list of instructions on how to:

• download the kernel from a mirror
• download patches to that kernel
  • use wget with an HTTP mirror (http://www.us.kernel.org), which should allow it to read file sizes and dates correctly so as to only mirror files that have been added/changed
• apply kernel patches from a directory of prepared patching scripts/instructions (see below code)
• tweak specific parts of the kernel (max_loop for example)
• build the kernel and kernel modules
  • use chroot/fakeroot/find to get the kernel makefiles to put the kernel modules all in one place for packaging. Once you have a modules package, you can copy out specific modules for specific boot scenario, or use the whole modules archive when running from CD
  • or use tar or the build script to pull the files out of the kernel archive. The files are all named *.ko, and their sources in the kernel should be their destionations under /lib/modules/<kernel version/kernel
• build the initrd image that contains the kernel modules needed for a specific boot scenario
• package the kernel and modules onto floppies, or into an isolinux archive that could be unpackaged in the destination for the ISO filesystem to be built

 foreach my $patchfile (<patchesdir/*.patch>) {
   system("cd $patchdir; patch -p? < $patchfile");
 } # foreach my $patchfile

Windows Packaging

• GTK for Windows got moved to it's own page
• tools that need to be ported
  • mkisofs
  • mksquashfs
• use Colinux (http://www.colinux.org/) to allow reading of the SQZ files and remastering of the image under windows. mebbe even get colinux to run some of the games and things.
• Create a Perl::Tk standalone Windows app that lets people remaster the CD image under Windows/Linux. Will use =cdrecord.exe= in Windows, and the system cdrecord package under Unix OS's
• use bash.exe for writing Windows shell scripts
• Is there a Windows version of whiptail or dialog that could be used for scripting instead? (See also MileStones page for more details on dialogs that are planned for the system)
• mebbe use reactos (http://www.reactos.com/) to do windows thingies under linux/by themselves with no OS on the machine

Rough outline to creating a sqz package

Copy of the mksquashfs README

• create the sqz with all of the correct files
• calculate the md5sum of the new sqz file
• append a package description file to the end of the sqz; include:
  • original sqz file size
  • original sqz md5sum
  • package name
  • project that the package belongs to
• collect the package descriptions in one place, like Debian does in /var/lib/dpkg/info with *.list files
• use the package description to build the file...
  • reverse the order of the package file to build the package
  • package script has a --warn option that will warn() instead of die() when a syntax error is encountered in the package description file

 filetype:/path/to/packagefile:perms:/linkpath/on/host

Squashed file magic:

 head -c 10 modules-2.6.2-antlinux.2004.045.1.sqz | od -a
 0000000   h   s   q   s   x stx nul nul   `  us

Change filenames from .sqz to .sqs to match their file magic

Output of otool -L /usr/bin/ssh

 /usr/bin/ssh:
   /usr/lib/libcrypto.0.9.7.dylib 
     (compatibility version 0.9.7, current version 0.9.7)
   /usr/lib/libbsm.dylib 
     (compatibility version 1.0.0, current version 1.0.0)
   /usr/lib/libz.1.dylib 
     (compatibility version 1.0.0, current version 1.2.2)
   /System/Library/Frameworks/Security.framework/Versions/A/Security 
     (compatibility version 1.0.0, current version 222.0.0)
   /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos 
     (compatibility version 5.0.0, current version 5.0.0)
   /usr/lib/libSystem.B.dylib
     (compatibility version 1.0.0, current version 88.0.0)

getdeps.pl Requirements

Note: getdeps.pl is an older version of DepSh. This is just here for completeness.

getdeps.pl is a script that builds a list of dependencies for a given set of binaries. You feed the script a list of binaries you want to build packages for, and it will determine what libraries the binary|binaries links to, as well as what Debian packages those libraries come from.

• sort output of getdeps.pl based on the number of binaries that link to that library
  • See this mailing list post for a good idea of what kinds of information could be gathered.

getdeps.pl Psudeocode

• Script start, read config via command line and config file (AppConfig Perl module)
• use the file lists in /var/lib/dpkg/info/*.list to determine what files are in which packages; this is probably all dpkg -S does anyways
• sort getdeps.pl output based on the number of files that are linked against that library