Whonix Source Code Introduction

From Whonix
< Dev
(Redirected from Dev/SourceCodeIntro)
Jump to navigation Jump to search

Introduction into Whonix source code.

Introduction[edit]

This chapter is dedicated to give an introduction into the Whonix source code. If you prefer to read and understand the source code just by reading scripts you may skip this optional chapter. It can be quite difficult to get started with hacking existing big complex projects.

build-steps[edit]

This is a high level overview. Whonix-Example-Implementation can currently be build in nine steps. (There is also a derivative-maker script, which automates these eight steps for your convenience.)

  • *_prepare-build-machine
  • *_export-libvirt-xml
  • *_create-debian-packages
  • *_create-raw-image
  • *_install-packages
  • *_run-chroot-scripts-post-d
  • *_convert-raw-to-qcow2
  • *_convert-raw-to-vdi
  • *_create-vbox-vm
  • *_export-vbox-vm
  • *_create-report

help-steps[edit]

Boring methods, which are required by the build-steps above.

  • delete-vbox-vm
  • mount-raw
  • chroot-raw
  • unchroot-raw
  • unmount-raw
  • pre gets sourced by all other scripts.
  • variables gets sourced by all other scripts.

derivative-maker[edit]

Is a script, which simply runs all other build-steps. Actually it is "optional". It has very little functionality besides running all other steps. You are free to run all steps one by one. That is useful for learning and for debugging purposes. In case you want to fix a bug or in case you want to upgrade the distribution or in case you want to switch the operating system or whatever you are better off running the steps manually. Run it with --help to see available switches.

Overview[edit]

  1. prepare-build-machine: installs build dependencies and applies a few other required settings for building from source.
  2. export-libvirt-xml: Copies files from libvirt folder to ~/whonix_binary folder and adds version numbers to them.
  3. create-debian-packages: Recursively works through packages folder, builds them one by one and adds them to a local APT repository.
  4. create-raw-image: grml-debootstraparchive.org creates virtual machine images in .raw image format. It keeps care of creating the image, the partition table, grub boot manager, minimal system debootstrap and apt installing all Whonix packages. This step requires most time in the build process.
  5. install-packages: Installs meta .deb packages whonix-shared-packages-dependencies/recommended, desktop, desktop-kde, kde-accessibility and whonix-*-packages-dependencies/recommended and whonix-workstation-default-applications. Those only include dependencies and recommended packages which are pulled from Debian's apt repository.
  6. run-chroot-scripts-post-d: Post Chroot Scripts are applied.
  7. convert-raw-to-qcow2: Only having effect when using --qocw2 switch. The .raw image gets converted to a .qcow2 image. Actually not converted, a new file will be created and the old .raw remains available until cleanup is run or manually deleted or grml-debootstrap runs again. The .raw format is more "generic". VirtualBox does not support raw (.raw) images, but .vdi and .vmdk (and others).
  8. convert-raw-to-vdi: The .raw image gets converted to a .vdi image. (Actually not converted, as explained above.)
  9. create-vbox-vm: A virtual machine (VirtualBox) will be created and the .vdi image will be attached.

Modularity[edit]

All steps could be easily replaced to add support for additional virtualizers, operating systems and so on. The numbers before the script names (20_..., 25_..., 30..., ...) are honored by derivative-maker (using run-partsarchive.org), which runs these steps in lexical order. Therefore you can easily add steps to the beginning or the end or even wretch steps in between.

File sizes[edit]

Do not get shocked by file sized. Initially the .raw is created with a size of 100 GB, but it will actually only take a fraction of that space on the harddrive. This is because .raw .vdi and .vmdk are all sparse files, which means they do not take their maximum size, but only as much data as really is inside these images.

Chroot Scripts[edit]

  • packages/kicksecure/initializer-dist/usr/libexec/initializer-dist/chroot-scripts-post.d/

Summary[edit]

Thus, given the nature of the build step orientated scripts, you can easily work on the different aspects of Whonix. For example, once you have created a clean virtual machine with the operating system only, you can make a copy, run either the gateway or the workstation copy routine or Chroot Scripts as often as you need to test your changes and if something goes wrong, go back to backup. You don't have to build everything from scratch again. [1]

Another Introduction[edit]

It is really not that difficult after all. If you like, you could read Manually Creating Whonix, which is a similar topic, which covers part of this in other words.

Debugging[edit]

Stuff you may find helpful for debugging.

Build Configuration[edit]

See Build Configuration.

Building without derivative-maker main script[edit]

most basic[edit]

Example using an apt cache (apt-cacher-ng) and building qcow2 images. 

sudo apt install apt-cacher-ng

cd derivative-maker 
sudo ./build-steps.d/*_prepare-build-machine --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/*_export-libvirt-xml --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/*_create-debian-packages --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/*_create-raw-image --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/*_install-packages --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/*_run-chroot-scripts-post-d --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/*_convert-raw-to-qcow2 --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/*_convert-raw-to-vdi --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/*_create-vbox-vm --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/*_export-vbox-vm --build --target qcow2 --flavor whonix-gateway

sudo ./build-steps.d/*_create-report --build --target qcow2 --flavor whonix-gateway

optional[edit]

For example for a qcow2 build, the following steps can be safely skipped, since these would be doing nothing anyway.

  • *_convert-raw-to-vdi
  • *_export-vbox-vm
  • *_create-report (not in use currently)

When first creating a gateway build and then building a workstation build (or vice versa) the following steps do not need to be repeated:

  • *_prepare-build-machine
  • *_create-debian-packages

Mount and inspect images[edit]

Mount and inspect images

Less Important Goodies[edit]

Unpacking .ova images[edit]

If you want for some reason to unpack an .ova, for example to get the .vdmk image, you can use tar.

tar -xvf Whonix-Gateway.ova

Will show:

Whonix-Gateway.ovf Whonix-Gateway-disk1.vmdk

Converting .vmdk images to .raw images[edit]

#qemu-img info Whonix-Gateway-disk1.vmdk

qemu-img convert Whonix-Gateway-disk1.vmdk -O raw Whonix-Gateway.raw

#qemu-img info Whonix-Gateway.raw

Build Script Features[edit]

See https://www.kicksecure.com/wiki/Dev/Derivative-Makerarchive.org

Notes[edit]

sudo ~/derivative-maker/packages/kicksecure/developer-meta-files/debug-steps/interactive-chroot-raw --target raw --flavor whonix-host-xfce --build --freedom false

Why not Replace grml-debootstrap with 'X'?[edit]

Whonix developers are not remotely close to exhausting grml-debootstrap's extensive feature-set yet. [2]

There are two sorts of VM image creation tools:

  • [A] Those that use virtualization, boot the image and perform actions.
  • [B] Those that use chroot (or maybe systemd-spawn).

[A] is incompatible with Whonix design principles due to files being created during boot, such as entropy seeds. This is a less clean method and is not suitable for redistribution. Similar to Distro Morphing vs Builds.

In either case it would be necessary to compare:

  1. The image created by grml-debootstrap versus the new tool.
  2. Create an image twice using the new tool and inspect the diff.

Related:

dm-prepare-release[edit]

dm-prepare-releasearchive.org is responsible for:

  • Creation of a unifiedarchive.org .ova image or libvirt.xz archive.
  • Creation of torrent files.
  • Creation of hash sum files.
  • Creation of digital software signatures.
  • Adding the license agreement.
  • Adding the disclaimer.
  • Redistribution.

See Also[edit]

Footnotes[edit]

  1. ↑ If something would go wrong, you would have to reinstall the whole operating system every time again. That's why we use separate steps.
  2. ↑ https://forums.whonix.org/t/replacing-grml-debootstrap-debos-build-platform/5582/3archive.org
Your support makes
all the difference!

We believe security software like Whonix needs to remain open source and independent. Would you help sustain and grow the project? Learn more about our 12 year success story and maybe DONATE!

Retrieved from "https://www.whonix.org/w/index.php?title=Dev/Source_Code_Intro&oldid=100334"