Tuesday, January 31, 2012

'Engineering' versus 'Science'

As part of my work as a PhD student, I do research in software engineering, in particular software deployment. My research consists of all kind of activities: I talk to all kinds of people (academic people, people from companies), I write and maintain various software tools, such as Disnix, I give presentations and I do various experiments. Most importantly, I write papers which report about various pieces of my research. Usually these papers are supposed to be presented at academic conferences.

Writing papers is not always an easy job. Conferences typically only accept high quality submissions for presentation. Papers are peer-reviewed by members of a program committee and submissions are in competition with each other. Usually, a small percentage of good papers are eligible for presentation. Some papers of mine got accepted immediately, others got rejected and received criticism.

In some of the reviews I have received so far, I get complimented with things like:

This is a remarkable piece of engineering work.

Although this may sound like a compliment, this typically means something bad, because conferences want valuable scientific research and not engineering work. In all these cases I've received such "compliments", the paper was rejected.

Although I understand that a paper has to conform to a certain quality level and not every story is suitable for presentation on a software engineering conference, such compliments bother me a bit, because I don't know exactly what they mean, nor I have an exact idea what the reviewers want me to do with these kind of criticisms. And another big question that I have is: When do we consider something engineering and when is it really science?

Software engineering conferences

There are many software engineering conferences around. We have sub-field conferences about specific domains, such as software testing, reliability engineering, programming languages, reverse engineering and repository mining. Furthermore, we have a number of top general conferences in which every software engineering task is potentially eligible for presentation. The International Conference on Software Engineering (ICSE) probably is the most well known top general conference in our field.

Most of the attendants of these conferences are academic people, although there is also some industry participation. For example, I have heard that the industry participation of ICSE is less than 20%, while this used to be around 50% in the past.

Difference between engineering and science

In the software engineering domain, the difference between science and engineering is not so clear to me. A common perception is that scientific research is what we should do in academia, engineering is what industry does.

Lately I have been looking for answers and (for fun) I dived into the history of the ICSE, the most popular software engineering conference in our field. I've looked into the proceedings of ICSE 1976, which was the first edition of the real ICSE (it seems that ICSE 1975 was actually NCSE: National Conference on Software Engineering). Furthermore, in this year the industry participation (if what I've heard is correct) is around 50%.

When I looked at the proceedings, the following paper caught my interest: 'Research Paradigms in Computer Science' by Peter Wegner from Brown University. In this paper, the author explores 4 influential definitions of computer science. Interestingly enough, this paper also reports about research in engineering:

Research in engineering is directed towards the efficient accomplishment of specific tasks and towards the development of tools that will enable classes of tasks to be accomplished more efficiently.

Furthermore, the paper also makes a distinction between a research engineer and a practicing engineer:

The problem-solving paradigm of the practicing engineer generally involves a sequence of systematic selection or design decisions which progressively narrow down alternative options for accomplishing the task until a unique realization of the task is determined.

The research engineer may use the paradigms of mathematics and physics in the development of tools for the practicing engineer, but is much more concerned with the practical implications of his research than the empirical scientist or mathematician.

Academic contributions

After reading this paper, assuming that this properly defines what research in engineering is about, I have the following idea about academic software engineering research contributions:

  • You shouldn't merely solve a problem for which you have to engineer something. It should be related to software engineering processes.
  • Your contribution should fill a gap within in a software engineering process. For example, it should offer an improvement or a new insight everybody should know about. Furthermore, this contribution must be novel.
  • The contribution should be significant enough.
  • You need strong motivation why this contribution is needed, which is not always obvious. You also need to look at what has been done before.
  • You need to prove what you claim is valid (or at least strong enough) and you need to describe how it compares to (possibly) related work.
  • And of course, you need to explain what you did

If you just solve an arbitrary problem by means of software engineering, without filling a gap, it is engineering.


If I take what I have written in the previous section into account and look back to what I've done before, I don't think I have written a pure engineering paper so far. I think that in these cases probably something else went wrong. This is what I think that could have go wrong:

  • The motivation is unclear or misunderstood.
  • Perhaps the paper may contain too many implementation details.
  • There is something wrong with the evaluation method or the evaluation is not strong enough.
  • Reviewers have different expectations about submissions. Maybe this is also due to the fact that software deployment is a very uncommon research subject.
  • A solution is/looks too simple

I find it a bit disappointing that I have to guess about these reasons myself. If I get a bad review, I would appreciate it if reviewers also make themselves clear why they think it's engineering, so that I can do something with their feedback (or preferably: use concrete criticisms as I have shown above). I think that's what peer-reviewing is about, right?


I'd also like to elaborate a bit more on engineering in this blog post. Although academic conferences want scientific contributions, is doing engineering work a bad thing? In my opinion it's also a good thing to do engineering work (even as a researcher), for the following reasons:

  • Typically, many software engineering problems arise in practice. Being in the field gives you a lot of experience and a better understanding of particular software engineering problems.
  • Many software engineering contributions are implemented as tools. Tools need to be constructed from various components by using various means. So in order to fill a gap in software engineering processes, you also have to do some software engineering.
  • Tools need to be applied in real world scenarios. In such cases, you may have to study existing systems, extend/integrate your tooling, or even implement work-arounds because you're tooling assumes ideal behaviour.
  • There may be lessons that you have learned in your research that every software engineer should know about.

In my period as a PhD researcher, I have done many engineering contributions, next to the scientific papers I have written. Typically, I use my blog to report about these, such as applying Disnix for deploying .NET services or comparing NixOS with GoboLinux. Although these blog posts aren't really useful scientific contributions, they are if fact more appreciated by the people I work with. Furthermore, these blog posts have also attracted many more readers than all my academic papers combined.

Last year I have also presented pieces of my work at FOSDEM, the Free/Open-Source Developers' European Meeting attracting thousands of developers from all over the world. For some reason, my talk attracted quite a lot of people and I also received much more interesting questions than I have ever received on any academic software engineering conference.

So what's the big deal anyway?

Next to the discussion of the meaning of science and engineering, I'd like to point out that engineering is not something bad, although it may not always be something reviewers of academic conferences are interested in. Engineering aspects, however, are very useful for industry people. They typically care more about what benefits new tools could offer and how they can apply these techniques in their environment instead of seeing a mathematical proof.

There have been many discussions among academic conference organisers about the question why the industry participation is so low these days and how this could be improved. I think one of the big reasons is a mismatch of interests. There are of course more reasons, but this is probably not something you could blame academic people for.


In this blog post, I have tried to discover what scientific research in software engineering is and what ordinary engineering is. I have also expressed my opinion about engineering and concluded that it's not useless, although it may not be something that program committee members of academic conferences are interested in.

Finally, I want to make myself clear that I have nothing against ICSE or any other academic conference. This blog post is not an ICSE rant. I find the ICSE a good conference, which I have attended with pleasure in 2009 and 2011. I have attended a number of very good presentations and I have also met quite a number of great people. Also I'm currently collaborating with a few researchers that I have met there.

(Ok! Ok!... I'm not going to lie about the fact I'm a bit disappointed that our paper for this years' edition has been rejected and I agree that certain parts of the paper can be improved. Furthermore, this blog post is not elaborate about this rejection, but about a phenomenon occurring within the academic software engineering research community that bothers me a bit.)

Tuesday, January 17, 2012

Porting software to AmigaOS (unconventional style)

As I have mentioned in my previous blog post (in which I reflected over the last year), the Amiga is still not quite dead yet and (for some reason) my blog post about my good ol' Amiga has attracted much more visitors than I expected. In fact, for a long time this was the most interesting story I have written, while most of my other blog posts are mostly about the research that I'm currently doing.

Nowadays, there is still an active community developing Amiga software. For that reason, I have decided to pay some attention to this subject once more. This time I'm going to make it a bit more related to the research I'm currently doing. If you are curious to see how I achieve that goal, please read on...

Geek Gadgets

One of the interesting efforts done in the past is a project called Geek Gadgets, in which the GNU toolchain and many other UNIX utilities are ported to AmigaOS. You could see this effort as something similar to Cygwin, providing a UNIX environment and GNU toolchain on Microsoft Windows.

Interesting packages included in this project are:

  • ixemul.library. A BSD 4.3 kernel running under AmigaOS, which internally translates a number system API calls to AmigaOS kernel calls. The API is specifically modelled after NetBSD.
  • libnix. An ANSI C API library directly using the AmigaOS libraries, instead of the Unix compatibility layer. This library is better suited for developing native AmigaOS applications not requiring any UNIX functionality.
  • GNU Compiler Collection. The free/open-source/ubiquitous compiler suite used on many platforms, such as Linux, Cygwin, FreeBSD, OpenBSD, NetBSD and Mac OS X.
  • Korn Shell. Geek Gadgets provides the pdksh shell as a Bourne-compatible shell instead of bash, which is commonly used on other GNU systems. Probably because it's smaller and faster.

These tools make it possible to port Unix-like applications to AmigaOS as well as writing native AmigaOS applications, using development tools commonly found on free software systems, such as Linux.

Quite a number of applications are ported using this toolset such as TeX, Emacs, the X Window System and SDL, a library which is commonly used on Linux to develop games and other multimedia applications.

Using Geek Gadgets

Below, I have included a screenshot showing the Amiga Workbench and an Amiga Shell. In this Amiga shell, I have opened a Korn Shell session to show the contents of the Geek Gadgets toolset and the version of the GCC compiler.

As you may notice in the screenshot above, the /gg directory is the Geek Gadgets root directory in which a typical UNIX-like filesystem hierarchy is defined with its associated files and utilities.

The Amiga filesystem is organized quite differently as a UNIX filesystem. Whereas a UNIX filesystem is a tree with only one root directory, an Amiga filesystem has many roots. For example, on the Amiga a path can be identified by Assign:RootDirectory/Subdirectory/File, in which an assignment can represent a disk drive identifier, disk label or a random assignment which you can attach to any folder in the filesystem.

Like Cygwin, the ixemul kernel also provides a UNIX to native filesystem mapping, by defining assignments as root directories. For example the /gg directory represents the GG: assignment in which the Geek Gadgets files are stored.

Deploying Geek Gadgets

Deploying a Geek Gadgets environment was a bit trickier than I expected:

  • The latest stable version is much too old, dating from 1998. The Geek Gadgets documentation is quite outdated and does not accurately describe what packages you need from the latest snapshot.
  • In order to use any Geek Gadgets tools, I need to manually set up the Geek Gadgets environment each time, by executing the GG:Sys/S/GG-Startup script and by raising the stack size, because otherwise I can get mysterious lock-ups/crashes. Of course, I could also add these instructions to S:User-Startup, but I don't want the Geek Gadgets to spoil up my regular system when I use native AmigaOS applications.
  • Geek Gadgets uses a traditional UNIX filesystem organisation, with all its disadvantages. For example, packages are deployed in an impure manner because their contents are scattered across the filesystem. I don't like it very much to let development packages pollute my filesystem.

Most importantly, my research as a PhD student is about software deployment in which I try to deal with such inconveniences. For Amiga readers not knowing anything about it, one of our key aspects is the Nix package manager, which we use to build software components from declarative build specifications. Furthermore, Nix has several distinct features such as the ability to store components in isolation, atomic upgrades and rollbacks and garbage collector which safely removes components no longer in use.

Nowadays, I use the Nix package manager for virtually everything that I develop and I have to compile, because it stores all builds isolated and it never pollutes my system. Furthermore, I can also easily use the Nix package manager to build several variants of the same package (e.g. for i686-linux and x86_64-linux), just by modifying several build parameters in the Nix expression for that particular package.

Building AmigaOS applications with Nix

So an interesting question that I have raised to myself is: "Can I use my research about software deployment to deal with these deployment inconveniences of Geek Gadgets on AmigaOS?".

An answer that you probably would expect me to give is that I have ported the Nix package manager to AmigaOS, but unfortunately that isn't possible because Nix requires much more memory than an Amiga has. Furthermore, the ixemul library does not completely support the POSIX API and does not support symlinks, which makes it impossible to run the Nix package manager properly.

Instead, I have developed a Nix function, which should be invoked on a Linux machine. This function launches UAE, the Ultimate Amiga Emulator, to perform a build and stores the build result in the Nix store of the host system. To develop this Nix function I had to create a basic disk image and a function that starts UAE with this disk image in which the actual build is performed.

Creating a disk image

Creating a basic disk image is a straightforward process:

  • UAE must be configured to use a host filesystem directory as a hard drive partition. In the UAE GUI this can be enabled by clicking on 'New filesystem...' button in the 'Harddisks' tab.
  • The Amiga Workbench 3.1 diskettes are required to perform a hard drive installation of the Amiga Workbench. The install disk contains a wizard which takes care of these steps. I performed an 'Immediate user installation' in which I only installed the English language locale, no printers and an American keymap.
  • I have also installed the LhA archiver, because this is the most widely used archiver on the Amiga. I have obtained this version from Aminet: http://aminet.net/package/util/arc/lha. This file is a self extracting archive, which can be installed by running the following instructions from the command-line interface:
    Protect lha.run +e
    Copy lha_68k C:lha

Installing the Geek Gadgets environment is a little trickier though:

  • I used this simple README file from Aminet as a reference describing the base packages I need: http://de4.aminet.net/dev/gg/0README-GG.txt
  • I did not download the Geek Gadgets packages from Aminet, but instead I downloaded the latest snapshot, which I found here: ftp://ftp.back2roots.org/pub/geekgadgets/amiga/m68k/snapshots/990529/bin.
  • Instead of gcc, I have downloaded egcs. Additionally, I have downloaded a2ixlibrary and gawk, which are frequently required to build packages. libg++ is not included in the latest snapshot and is apparently also not required.
  • I've extracted all the tarballs on the host system into the GG subfolder of the Amiga drive, because this is much faster. I had to fix to bin/sh symlink to the ksh manually, because the symlink in the tarball is incorrect.
  • A GG: assignment is required so that Geek Gadgets knows where to find its files. This assignment can be made by adding the following line to the S:User-Startup file:
    Assign >NIL: GG: DH0:GG
  • If you want to create native AmigaOS applications, you need to copy the Amiga system headers into GG:os-include. I have found a collection of Geek Gadgets compatible os-include headers here: http://ftp.back2roots.org/back2roots/cds/fred_fish/geekgadgets_vol1_9610/ade-bin
  • In order to automatically perform builds from Nix expressions, we need to start the build process right after the system is booted. To make this possible I commented out the last two lines from the S:Startup-Sequence, to disable the Workbench and to boot into command-line mode:
    ; C:LoadWB
    ; EndCLI >NIL:

    I have added the following lines to the S:User-Startup script to initialize the Geek Gadgets environment and to automatically start the build process:
    Execute GG:Sys/S/GG-Startup
    Stack 200000
    sh build.sh

Writing a Nix function

Now that we have a basic disk image containing the Amiga Workbench and the Geek Gadgets utilities, we need to write a Nix function that automatically performs the build steps. I will list pieces of the code of this Nix function here, with some explanation.

The AmigaOS build function, takes 4 arguments where 1 argument is optional:

{name, src, buildCommand, buildInputs ? []}:

The name parameter specifies the name of the component, which also appears in the resulting Nix store path. The src parameter is used to specify the source code that must be compiled. This parameter could refer to a tarball, or a directory containing source code. You could also bind this parameter to a function, such as fetchurl to remotely obtain a source tarball. The buildCommand parameter is a string which specifies how the source code can be built. The buildInputs parameter can be used to install additional packages into the Geek Gadgets environment, such as libraries or other build tools which aren't included in the basic disk image.

The next line imports all the Nix package build functions into the scope of our AmigaOS build function, so that we can conveniently access UAE and other utilities:

with import <nixpkgs> {};

We also need to specify where we can find the disk image containing the Workbench and Geek Gadgets. The UAE emulator also needs a kickstart ROM image. The following code fragment specifies where to find them (you need to replace these paths to match your situation):

  amigaBase = ./amigabase;
  kickRom = ./kickrom/kick.rom;

The following lines invoke the stdenv.mkDerivation function, which we need to specify how we can actually perform a build. The following code fragment inherits the name from our function header. Furthermore, it adds the UAE emulator in our environment so that we can boot our disk image and perform the compilation. We need procps to kill the emulator, once the build has finished:

stdenv.mkDerivation {
  inherit name;
  buildInputs = [ uae procps ];
  buildCommand = ''

The remainder of the function specifies how the build should be performed. As we're not allowed to mess up the original base image (it's read-only anyway during a Nix build), we need to create a new one in our temp directory of our build. Fortunately, most parts of the AmigaOS filesystem can be used in read-only mode, so we only need to symlink these system directories. The following code fragment creates symlinks to all static AmigaOS directories:

    mkdir hd
    cd hd
    for i in ${amigaBase}/{C,Classes,Expansion,Fonts,L,Libs,Locale} \
        ln -s "$i"
        # Symlink icons
        if [ -f "$i.info" ]
            ln -sf "$i.info"

The Geek Gadgets environment however, must be copied to our new temporary hard drive, because we may need to install other packages in this environment. Additionally, we need to fix the permissions of the copied GG directory, because in the Nix store it has been made read-only and we require a writeable GG directory:

    cp -av ${amigaBase}/GG .
    chmod 755 GG
    cd GG
    for i in `find . -type d` `find . -type f`
        chmod 755 $i

If we have any build inputs (such as library packages), we need to unpack them into the Geek Gadgets environment:

    for i in ${toString buildInputs}
        cp -av $i/* .

The next step is copying the source code into a location in which a build can be performed. For this purpose we create a temp T directory in our AmigaOS filesystem. If the source code is a directory, we need to strip the hash off it and we need to make it writable again:

    cd ..
    mkdir T
    cd T
    stripHash ${src}
    cp -av ${src} $strippedName
    if [ -d "$strippedName" ]
        chmod -R 755 "$strippedName"

Then we have to execute the instructions defined in the buildCommand somehow. We generate a shell script called buildinstructions.sh containing the contents of the buildCommand string. This script is executed at boot time by the Korn shell.

    echo "src=$strippedName" > buildinstructions.sh
    cat >> buildinstructions.sh << "EOF"
We can't directly invoke the build instructions, because we also have to determine whether a build has succeeded or failed. The following code fragments creates an additional script called: build.sh, which invokes the buildinstructions.sh script. After the build instructions script has finished, it determines whether it has succeeded or failed and writes the status into a file called done which is stored in the resulting Nix store path:

    cat > build.sh << "EOF"
    ( sh -e buildinstructions.sh
      if [ $? = 0 ]
          echo "success" > /OUT/done
          echo "failure" > /OUT/done
    ) 2>&1 | tee /OUT/log.txt
We now configured a bootable disk image, which automatically performs a build. The next step is to provide a number of settings to UAE so that our temporary disk image can be used and that a build is performed properly:

    cd ../..
    # Create UAE config file
    export HOME=$(pwd)
    cat > .uaerc <<EOF
    config_description=UAE default configuration

The code fragment above defines our temporary build directory as HOME directory (In a Nix build environment this variable is set to a non-existent path: /homeless-shelter in order to remove side effects). Furthermore, we generate a UAE configuration file with the following properties:

  • We disable the GUI, because this prevents us running UAE non-interactively.
  • We need to specify where the Kickstart ROM image can be found. This property is defined earlier in the kickRom variable.
  • We disable sound support, because we don't need it and it may cause lock ups in some cases.
  • The Geek Gadgets tools require quite an amount of RAM to run properly, so we've configured the UAE instance to use 2 MiB of Chip RAM and 8 MiB of Fast RAM.
  • Normally the UAE emulator tries to accurately match the speed of a 7 MHz Motorla 68000 chip, which makes building software very slow compared to nowadays' standards. Fortunately, UAE can also use the host CPU as efficiently as possible, which we do in our build function to make builds significantly faster.
  • We also configured two hard drive instances. The HD0: drive corresponds to the disk image we have created earlier containing the Workbench and Geek Gadgets. The OUT: drive corresponds to the Nix store output path, in which we have to store our build result.

After configuring UAE, we have to run it. Since UAE is a GUI application requiring a X11 display server, we have to specify the DISPLAY environment variable:

    mkdir -p $out

    export DISPLAY=:0
    # Start UAE
    uae &

There is no way to shut UAE down from an Amiga session, so we wait until we see the done file in our output directory. Once this file exists we kill the UAE process:

    while [ ! -f $out/done ]
        sleep 1
    # Kill the emulator
    pkill uae
    sleep 1

    # Check the build status
    [ "$(cat $out/done)" = "success" ]

After executing all the previous steps, the build has either succeeded or failed. If a build has succeeded, we can remove the done file, as well as some temp files created by UAE:

    cd $out
    rm done
    rm -f `find . -name _UAEFSDB.___`

Using the Nix function

The following code fragment shows how this Nix function can be used to build GNU Hello for AmigaOS:

import ./amigaosbuild.nix {
  name = "hello";
  src = fetchurl {
    url = mirror://gnu/hello/hello-2.1.1.tar.gz;
    sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
  buildCommand = ''
    tar xfvz $src
    cd hello-2.1.1
    ./configure --prefix=/OUT
    make || true
    make install

The expression above looks almost identical to the regular Nix expression building GNU Hello. Important to point out is that we store the output of the GNU Hello world in the /OUT directory, which corresponds to the OUT: drive. The GNU Hello build apparently requires help2man to generate a manual page. This tool is not present in our environment, and to make the build succeed I've added || true to the make instruction, so that it always succeeds. By running the following command line instruction:

$ nix-build hello.nix

The GNU Hello component gets build for AmigaOS, which looks like this:

I have also tried compiling a native AmigaOS application, which creates a very simple Intuition GUI window. I wrote the following test program for this purpose:

#include <exec/types.h>
#include <intuition/intuition.h>
#include <intuition/intuitionbase.h>
#include <intuition/screens.h>

#include <clib/exec_protos.h>
#include <clib/dos_protos.h>
#include <clib/intuition_protos.h>

#include <stdio.h>


struct Library *IntuitionBase = NULL;

struct TagItem win_tags[] = {
    {WA_Left, 20},
    {WA_Top, 20},
    {WA_Width, 300},
    {WA_Height, 120},
    {WA_CloseGadget, TRUE},
    {WA_Title, "Hello world!"},

void handle_window_events(struct Window *window)

int main(int argc, char *argv[])
    IntuitionBase = OpenLibrary("intuition.library",
        fprintf(stderr, "Error opening intuition library!\n");
        return 1;
        struct Window *window = OpenWindowTagList(NULL, win_tags);
        if(window == NULL)
            fprintf(stderr, "Window failed to open!\n");
        return 0;

And this is what the result looks like:

So apparently it works fine!


This blog post covers one of the craziest experiments I have done so far. I have developed a Nix function, which builds software for AmigaOS using the Geek Gadgets toolset by creating a disk image and by using UAE.

After reading this blog post (in case you're still interested :-) ), you may wonder if I actually have a real use case for this. The answer is: YES, although I'm not going to reveal what it is yet. Probably it will take a while, because I currently don't have that much spare time for fun projects like these.

I have decided not to commit this function to Nixpkgs, because it's quite hacky and it depends on stuff (like the Amiga Workbench) which I cannot distribute freely. It's probably not so hard to recreate this function yourself. You can freely use the code I have written under the MIT license.


  • In order to use this build function you need an Amiga Kickstart ROM image and the Amiga Workbench diskettes. If you don't have these, the legal way to obtain these is by ordering one of the Cloanto's Amiga Forever discs. Alternatively, you could try AROS, a free/open-source operating system reimplementing the AmigaOS APIs, although I have no experience with it so far.
  • Performing builds in virtual machines inside a Nix expression is nothing new. We have similar approaches in our Hydra build farm, in which we generate OpenSUSE, Ubuntu and Fedora machines to build RPM and Debian packages from Nix expressions. NixOS virtualization provides additional advantages, which I have described here.


UPDATE: I didn't expect this, but it seems that there are people around who actually want to use this. I have recently put some effort in it to make it a bit usable. The source code can be found on the nix-amigaosenv GitHub page, which contains updated instructions. Furthermore, I have made some enhancements, so that builds can be performed much faster and cheaper.