Development workflow
A workflow for developing Qubes OS+
First things first, setup QubesBuilder. This guide assumes you’re using qubes-builder to build Qubes.
Repositories and committing Code
Qubes is split into a bunch of git repos. These are all contained in the
qubes-src
directory under qubes-builder. Subdirectories there are
separate components, stored in separate git repositories.
The best way to write and contribute code is to create a git repo
somewhere (e.g., github) for the repo you are interested in editing
(e.g., qubes-manager
, core-agent-linux
, etc). To integrate your
repo with the rest of Qubes, cd to the repo directory and add your
repository as a remote in git
Example:
$ cd qubes-builder/qubes-src/qubes-manager
$ git remote add abel git@github.com:abeluck/qubes-manager.git
You can then proceed to easily develop in your own branches, pull in new commits from the dev branches, merge them, and eventually push to your own repo on github.
When you are ready to submit your changes to Qubes to be merged, push
your changes, then create a signed git tag (using git tag -s
).
Finally, send a letter to the Qubes listserv describing the changes and
including the link to your repository. You can also create pull request
on github. Don’t forget to include your public PGP key you use to sign
your tags.
Kernel-specific notes
Prepare fresh version of kernel sources, with Qubes-specific patches applied
In qubes-builder/qubes-src/linux-kernel:
make prep
The resulting tree will be in kernel-<VERSION>/linux-<VERSION>:
ls -ltrd kernel*/linux*
drwxr-xr-x 23 user user 4096 Nov 5 09:50 kernel-3.4.18/linux-3.4.18
drwxr-xr-x 6 user user 4096 Nov 21 20:48 kernel-3.4.18/linux-obj
Go to the kernel tree and update the version
In qubes-builder/qubes-src/linux-kernel:
cd kernel-3.4.18/linux-3.4.18
Changing the config
In kernel-3.4.18/linux-3.4.18:
cp ../../config .config
make oldconfig
Now change the configuration. For example, in kernel-3.4.18/linux-3.4.18:
make menuconfig
Copy the modified config back into the kernel tree:
cp .config ../../../config
Patching the code
TODO: describe the workflow for patching the code, below are some random notes, not working well
ln -s ../../patches.xen
export QUILT_PATCHES=patches.xen
export QUILT_REFRESH_ARGS="-p ab --no-timestamps --no-index"
export QUILT_SERIES=../../series-pvops.conf
quilt new patches.xen/pvops-3.4-0101-usb-xen-pvusb-driver-bugfix.patch
quilt add drivers/usb/host/Kconfig drivers/usb/host/Makefile \
drivers/usb/host/xen-usbback/* drivers/usb/host/xen-usbfront.c \
include/xen/interface/io/usbif.h
*edit something*
quilt refresh
cd ../..
vi series.conf
Building RPMs
TODO: Is this step generic for all subsystems?
Now it is a good moment to make sure you have changed kernel release name in rel file. For example, if you change it to ‘1debug201211116c’ the resulting RPMs will be named ‘kernel-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm’. This will help distinguish between different versions of the same package.
You might want to take a moment here to review (git diff, git status), commit your changes locally.
To actually build RPMs, in qubes-builder:
make linux-kernel
RPMs will appear in qubes-src/linux-kernel/pkgs/fc20/x86_64:
-rw-rw-r-- 1 user user 42996126 Nov 17 04:08 kernel-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm
-rw-rw-r-- 1 user user 43001450 Nov 17 05:36 kernel-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm
-rw-rw-r-- 1 user user 8940138 Nov 17 04:08 kernel-devel-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm
-rw-rw-r-- 1 user user 8937818 Nov 17 05:36 kernel-devel-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm
-rw-rw-r-- 1 user user 54490741 Nov 17 04:08 kernel-qubes-vm-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm
-rw-rw-r-- 1 user user 54502117 Nov 17 05:37 kernel-qubes-vm-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm
Useful QubesBuilder commands
make check
- will check if all the code was committed into repository and if all repository are tagged with signed tag.make show-vtags
- show version of each component (based on git tags) - mostly useful just before building ISO. Note: this will not show version for components containing changes since last version tag.make push
- push change from all repositories to git server. You must set proper remotes (see above) for all repositories first.make prepare-merge
- fetch changes from remote repositories (can be specified on commandline via GIT_SUBDIR or GIT_REMOTE vars), (optionally) verify tags and show the changes. This do not merge the changes - there are left for review as FETCH_HEAD ref. You can merge them usinggit merge FETCH_HEAD
(in each repo directory). Ormake do-merge
to merge all of them.
Copying Code to dom0
When developing it is convenient to be able to rapidly test changes. Assuming you’re developing Qubes on Qubes, you should be working in a special VM for Qubes and occasionally you will want to transfer code or RPMs back to dom0 for testing.
Here are some handy scripts Marek has shared to facilitate this.
You may also like to run your test environment on separate machine.
Syncing dom0 files
TODO: edit this script to be more generic
#!/bin/sh
set -x
set -e
QUBES_PY_DIR=/usr/lib64/python2.6/site-packages/qubes
QUBES_PY=$QUBES_PY_DIR/qubes.py
QUBESUTILS_PY=$QUBES_PY_DIR/qubesutils.py
qvm-run -p qubes-devel 'cd qubes-builder/qubes-src/core/dom0; tar c qmemman/qmemman*.py qvm-core/*.py qvm-tools/* misc/vm-template-hvm.conf misc/qubes-start.desktop ../misc/block-snapshot aux-tools ../qrexec' |tar xv
cp $QUBES_PY qubes.py.bak$$
cp $QUBESUTILS_PY qubesutils.py.bak$$
cp /etc/xen/scripts/block-snapshot block-snapshot.bak$$
sudo cp qvm-core/qubes.py $QUBES_PY
sudo cp qvm-core/qubesutils.py $QUBESUTILS_PY
sudo cp qvm-core/guihelpers.py $QUBES_PY_DIR/
sudo cp qmemman/qmemman*.py $QUBES_PY_DIR/
sudo cp misc/vm-template-hvm.conf /usr/share/qubes/
sudo cp misc/qubes-start.desktop /usr/share/qubes/
sudo cp misc/block-snapshot /etc/xen/scripts/
sudo cp aux-tools/qubes-dom0-updates.cron /etc/cron.daily/
# FIXME(Abel Luck): I hope to
Apply qvm-tools
TODO: make it more generic
#!/bin/sh
BAK=qvm-tools.bak$$
mkdir -p $BAK
cp -a /usr/bin/qvm-* /usr/bin/qubes-* $BAK/
sudo cp qvm-tools/qvm-* qvm-tools/qubes-* /usr/bin/
Copy from dom0 to an appvm
#/bin/sh
#
# usage ./cp-domain <vm_name> <file_to_copy>
#
domain=$1
file=$2
fname=`basename $file`
qvm-run $domain 'mkdir /home/user/incoming/dom0 -p'
cat $file| qvm-run --pass-io $domain "cat > /home/user/incoming/dom0/$fname"
Git connection between VMs
Sometimes it’s useful to transfer git commits between VMs. You can use
git format-patch
for that and simply copy the files. But you can
also setup custom qrexec service for it.
Below example assumes that you use builder-RX
directory in target VM
to store sources in qubes-builder layout (where X
is some number).
Make sure that all the scripts are executable.
Service file (save in /usr/local/etc/qubes-rpc/local.Git
in target
VM):
#!/bin/sh
exec 2>/tmp/log2
read service rel repo
echo "Params: $service $rel $repo" >&2
# Adjust regexps if needed
echo "$repo" | grep -q '^[A-Za-z0-9-]\+$' || exit 1
echo "$rel" | grep -q '^[0-9.]\+$' || exit 1
path="/home/user/builder-R$rel/qubes-src/$repo"
if [ "$repo" = "builder" ]; then
path="/home/user/builder-R$rel"
fi
case $service in
git-receive-pack|git-upload-pack)
echo "starting $service $path" >&2
exec $service $path
;;
*)
echo "Unsupported service: $service" >&2
;;
esac
Client script (save in ~/bin/git-qrexec
in source VM):
#!/bin/sh
VMNAME=$1
(echo $GIT_EXT_SERVICE $2 $3; exec cat) | qrexec-client-vm $VMNAME local.Git
You will also need to setup qrexec policy in dom0
(/etc/qubes-rpc/policy/local.Git
).
Usage:
[user@source core-agent-linux]$ git remote add testbuilder "ext::git-qrexec testbuilder 3 core-agent-linux"
[user@source core-agent-linux]$ git push testbuilder master
You can create ~/bin/add-remote
script to ease adding remotes:
#!/bin/sh
[ -n "$1" ] || exit 1
if [ "$1" = "tb" ]; then
git remote add $1 "ext::git-qrexec testbuilder 3 `basename $PWD`"
exit $?
fi
git remote add $1 git@github.com:$1/qubes-`basename $PWD`
It should be executed from component top level directory. This script
takes one argument - remote name. If it is tb
, then it creates
qrexec-based git remote to testbuilder
VM. Otherwise it creates
remote pointing at github account of the same name. In any case it
points at repository matching current directory name.
Sending packages to different VM
Other useful script(s) can be used to setup local package repository hosted in some VM. This way you can keep your development VM behind firewall, while having an option to expose some yum/apt repository to the local network (to have them installed on test machine).
To achieve this goal, a dummy repository can be created, which instead
of populating metadata locally, will upload the packages to some other
VM and trigger repository update there (using qrexec). You can use
unstable
repository flavor, because there is no release managing
rules bundled (unlike current and current-testing).
RPM packages - yum repo
In source VM, grab
linux-yum repository
(below is assumed you’ve made it in ~/repo-yum-upload
directory) and
replace update_repo.sh
script with:
#!/bin/sh
VMNAME=repo-vm
set -e
qvm-copy-to-vm $VMNAME $1
# remove only files, leave directory structure
find -type f -name '*.rpm' -delete
# trigger repo update
qrexec-client-vm $VMNAME local.UpdateYum
In target VM, setup actual yum repository (also based on
linux-yum, this time
without modifications). You will also need to setup some gpg key for
signing packages (it is possible to force yum to install unsigned
packages, but it isn’t possible for qubes-dom0-update
tool). Fill
~/.rpmmacros
with key description:
%_gpg_name Test packages signing key
Then setup local.UpdateYum
qrexec service
(/usr/local/etc/qubes-rpc/local.UpdateYum
):
#!/bin/sh
if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then
exit 1
fi
real_repository=/home/user/linux-yum
incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN
find $incoming -name '*.rpm' |xargs rpm -K |grep -iv pgp |cut -f1 -d: |xargs -r setsid -w rpm --addsign 2>&1
rsync -lr --remove-source-files $incoming/ $real_repository
cd $real_repository
export SKIP_REPO_CHECK=1
if [ -d $incoming/r3.1 ]; then
./update_repo-unstable.sh r3.1
fi
if [ -d $incoming/r3.0 ]; then
./update_repo-unstable.sh r3.0
fi
if [ -d $incoming/r2 ]; then
./update_repo-unstable.sh r2
fi
find $incoming -type d -empty -delete
exit 0
Of course you will also need to setup qrexec policy in dom0
/etc/qubes-rpc/policy/local.UpdateYum
.
If you want to access the repository from network, you need to setup HTTP server serving it, and configure the system to let other machines actually reach this HTTP server. You can use for example using port forwarding or setting up Tor hidden service. Configuration details of those services are outside of the scope of this page.
Usage: setup builder.conf
in source VM to use your dummy-uploader
repository:
LINUX_REPO_BASEDIR = ../../repo-yum-upload/r3.1
Then use make update-repo-unstable
to upload the packages. You can
also specify selected components on command line, then build them and
upload to the repository:
make COMPONENTS="core-agent-linux gui-agent-linux linux-utils" qubes update-repo-unstable
On the test machine, add yum repository (/etc/yum.repos.d
) pointing
at just configured HTTP server. For example:
[local-test]
name=Test
baseurl=http://local-test.lan/linux-yum/r$releasever/unstable/dom0/fc20
Remember to also import gpg public key using rpm --import
.
Deb packages - Apt repo
Steps are mostly the same as in the case of yum repo. The only details that differ:
use linux-deb instead of linux-yum as a base - both in source and target VM
use different
update_repo.sh
script in source VM (below)use
local.UpdateApt
qrexec service in target VM (code below)in target VM additionally place
update-local-repo.sh
script in repository dir (code below)
update_repo.sh
script:
#!/bin/sh
set -e
current_release=$1
VMNAME=repo-vm
qvm-copy-to-vm $VMNAME $1
find $current_release -type f -name '*.deb' -delete
rm -f $current_release/vm/db/*
qrexec-client-vm $VMNAME local.UpdateApt
local.UpdateApt
service code
(/usr/local/etc/qubes-rpc/local.UpdateApt
in repo-serving VM):
#!/bin/sh
if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then
exit 1
fi
incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN
rsync -lr --remove-source-files $incoming/ /home/user/linux-deb/
cd /home/user/linux-deb
export SKIP_REPO_CHECK=1
if [ -d $incoming/r3.1 ]; then
for dist in `ls r3.1/vm/dists`; do
./update-local-repo.sh r3.1/vm $dist
done
fi
if [ -d $incoming/r3.0 ]; then
for dist in `ls r3.0/vm/dists`; do
./update-local-repo.sh r3.0/vm $dist
done
fi
if [ -d $incoming/r2 ]; then
for dist in `ls r2/vm/dists`; do
./update-local-repo.sh r2/vm $dist
done
fi
find $incoming -type d -empty -delete
exit 0
update-local-repo.sh
:
#!/bin/sh
set -e
# Set this to your local repository signing key
SIGN_KEY=01ABCDEF
[ -z "$1" ] && { echo "Usage: $0 <repo> <dist>"; exit 1; }
REPO_DIR=$1
DIST=$2
if [ "$DIST" = "wheezy-unstable" ]; then
DIST_TAG=deb7
elif [ "$DIST" = "jessie-unstable" ]; then
DIST_TAG=deb8
elif [ "$DIST" = "stretch-unstable" ]; then
DIST_TAG=deb9
fi
pushd $REPO_DIR
mkdir -p dists/$DIST/main/binary-amd64
dpkg-scanpackages --multiversion --arch "*$DIST_TAG*" . > dists/$DIST/main/binary-amd64/Packages
gzip -9c dists/$DIST/main/binary-amd64/Packages > dists/$DIST/main/binary-amd64/Packages.gz
cat > dists/$DIST/Release <<EOF
Label: Test repo
Suite: $DIST
Codename: $DIST
Date: `date -R`
Architectures: amd64
Components: main
SHA1:
EOF
function calc_sha1() {
f=dists/$DIST/$1
echo -n " "
echo -n `sha1sum $f|cut -d' ' -f 1` ""
echo -n `stat -c %s $f` ""
echo $1
}
calc_sha1 main/binary-amd64/Packages >> dists/$DIST/Release
rm -f $DIST/Release.gpg
rm -f $DIST/InRelease
gpg -abs -u "$SIGN_KEY" \
< dists/$DIST/Release > dists/$DIST/Release.gpg
gpg -a -s --clearsign -u "$SIGN_KEY" \
< dists/$DIST/Release > dists/$DIST/InRelease
popd
if [ `id -u` -eq 0 ]; then
chown -R --reference=$REPO_DIR $REPO_DIR
fi
Usage: add this line to /etc/apt/sources.list
on test machine
(adjust host and path):
deb http://local-test.lan/linux-deb/r3.1 jessie-unstable main