Flask in a Rump Kernel

Rump Kernels

[Updated for Python 3.5.1]

Rump Kernels are better explained by the FAQ than myself, so I suggest reading through it. But essentially a Rump Kernel is a kernel with everything but your bare essentials removed. Don't need a file system? Then don't bundle the file system drivers. What about the drivers you do need? You get them from an Anykernel!

The idea of an Anykernel is that individual drivers can be taken and used in a Rump Kernel. NetBSD is an Anykernel. What this means is that a Rump Kernel can use the drivers directly from the NetBSD source tree. You don't need to build your own drivers to use a Rump Kernel. You can use real-world drivers to build production ready Unikernels (wait... unikernels?).

At this time, a Unikernel is built with a script called Rumprun. Rumprun takes your application, libc, the rumpkernel, and hypervisor layer, then builds the final product (unikernel) that will interact with things. The Unikernel is what you end up running in Xen or QEMU.

So... back to Python remember?

Yes. A Unikernel provides you with a "POSIX-y" environment to operate in. This lets most software run right out of the box. Which means we can probably get Python up and running without too much trouble.

In fact, Python already exists in the Rumprun package system.

The Python Package

The Rumprun package system is similar to the BSD ports structure. A source distribution is specified and downloaded. Patches are applied to the source. Then everything is compiled into a Unikernel.

In the case of Python there were a couple hurdles to overcome.

• Cross Compilation. The build-rr toolchain lets us build binaries for the Rump Kernel environment. The Python config scripts need to be patched in order to make this cross compilation option available.
• Static builds. Python needs to be compiled statically. We are creating a single, bootable, Unikernel. We can't rely on dynamic libraries being availble at runtime.
• Bootstrapping Python. Python actually needs Python to build Python. This is problematic when we are cross compiling. The target Python (and tools) that we are building can't run on the host platform. This is solved by building Python twice. The first time we build for the host, and save the required binaries. The second time we cross build for the target, and use the host binaries to help.

The Python package makes these cross compilation updates. It also modifies the Modules/Setup.dist file to build several modules statically along with Python.

To build the Python package you will need a working toolchain, start here. I will assume that you have a working build environment for the rest of the tutorial.

Now grab the rumprun packages repo, enter the python package, and start the build.

git clone git@github.com:rumpkernel/rumprun-packages.git
cd python
export PATH=/path/to/app-tools:$PATH
make

You're going to see a lot of errors like this one:

collect2: error: ld returned 1 exit status

But you will also end up with a Python ISO in the images/ directory. To test out your Python package installation, please follow the directions in the README.md file. This will walk you through rumpbake and how to build the bootstrap Python script.

Running Flask

At this point, you should have a Unikernel booting up and running the Python "Hello World" example. Now our goal is to:

• Build a Python environment with the modules we need
• Include that environment in our unikernel
• Setup networking

Building the Python environment is fairly straight forward. Using the same version of Python as your Unikernel (probably 3.4.3) create a virtual environment and install any packages you'd like available to your Unikernel. In this case we're just installing Flask.

pyvenv-3.5 helloworld-env
source helloworld-env/bin/activate
pip install flask

To include this environment in the Unikernel, we want to build an ISO containing our site packages. We will end up mounting this ISO inside the Unikernel. Just like we did for the static Python build.

genisoimage -r -o py3env.iso helloworld-env/lib/python3.5/site-packages

Setting up networking requires us to create a TAP device. This will let the host machine pass traffic to our Unikernel. We'll setup 10.0.120.100, and then our Unikernel will listen on 10.0.120.101.

sudo ip tuntap add tap0 mode tap
sudo ip addr add 10.0.120.100/24 dev tap0
sudo ip link set dev tap0 up

We will avoid doing anything special for our Flask application. Just a basic example will be enough for now.

# main.py
from flask import Flask
app = Flask(__name__)


@app.route("/")
def hello():
    return "Hello World!"


if __name__ == "__main__":
    app.run(host='0.0.0.0')

Now we are able to start our Unikernel. The following script should be familiar. We are following the steps from the README.md file from earlier. However we also setup networking and mount our Python environment in the site-packages directory that our Unikernel sees.

#!/bin/sh

PATH=/home/rday/Development/os/rump/qemu/bin:$PATH:/home/rday/Development/os/rump/rumprun/app-tools

cython --embed -v -3 -Werror main.py
x86_64-rumprun-netbsd-gcc main.c -o main.o \
  -I../Python-3.5.1/pythondist/include/python3.5m \
  -L../Python-3.5.1/pythondist/lib \
  -lpython3.5m -lutil -lm -lz -lssl -lcrypto

rumprun-bake hw_virtio main.bin main.o

rumprun qemu -i \
      -I if,vioif,'-net tap,ifname=tap0,script=no'\
      -W if,inet,static,10.0.120.101/24 \
      -b ../../python.iso,/python/lib/python3.5 \
      -b ../../stubetc.iso,/etc \
      -b py3env.iso,/python/lib/python3.5/site-packages \
      -e PYTHONHOME=/python main.bin

Give it a shot. Run the script, and visit http://10.0.120.101:5000. You should get the "Hello World" message to pop up!