Interesting is the LLVM CFI/LTO support that can be leveraged for more security. open a Debian feature request asking if they plan to build it with these flags.

aarch64[edit]

Packages needed on x86 host:

sudo apt install qemu-system-aarch64 qemu-efi-aarch64 qemu-efi-arm

Debian built Openstack images tailored for KVM environments can be fetched from here: https://cdimage.debian.org/cdimage/openstack/

These images don't ship with a DE but were essential in coming up with a VM config that works. Here's the XML for a machine that successfully booted and reached the systemc console with minimal modifcations from defaults. Note that on an actual aach64 host, the domain type would need to be set to 'kvm' instead to run without CPU emulation. More modifications may be needed to this section, but it woun;t be possible to know without hands-on experience:

<domain type='qemu' id='58'>
  <name>debian10-aarch64</name>
  <uuid>679847c6-d4ca-4501-a4df-4d1b7ef9cfca</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="https://debian.org/debian/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='aarch64' machine='virt-3.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/debian10-aarch64_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <gic version='2'/>
  </features>
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>cortex-a57</model>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-aarch64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/debian-{{Stable project version based on Debian version short}}.9.1-20210423-openstack-arm64.qcow2'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:68:60:6c'/>
      <source network='Whonix-Internal' bridge='virbr2'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='system-serial' port='0'>
        <model name='pl011'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <video>
      <model type='virtio' heads='1' primary='yes'>
        <acceleration accel3d='no'/>
      </model>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </video>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <alias name='rng0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </rng>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-679847c6-d4ca-4501-a4df-4d1b7ef9cfca</label>
    <imagelabel>libvirt-679847c6-d4ca-4501-a4df-4d1b7ef9cfca</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+119:+126</label>
    <imagelabel>+119:+126</imagelabel>
  </seclabel>
</domain>

VirtIO Sound[edit]

Now supported in Linux 5.13. Move on from the current emulated device whenever it's available in stable. https://www.phoronix.com/news/VirtIO-Sound-Driver-Linux-5.13

Virtio-Video[edit]

Paravirtual webcam access and screen sharing capability soon to land in KVM. Worth documenting when ready.

https://www.phoronix.com/news/VirtIO-Video-2020

Emulated TPM[edit]

KVM has recently added virtualized TPM support. They can be very valuable from a security standpoint (and more trustworthy than hardware TPMs). A big thing is they can be used like virtual smartcards providing security like Qubes' split-gpg but for more types of software keys like OpenSSH, OpenSSL and more.

To boot a VM after adding the emulated TPM virtual hardware, on the host install:

sudo apt install swtpm-tools

Update:

GPG 2.3 now supports TPM2 https://www.gnupg.org/blog/20210315-using-tpm-with-gnupg-2.3.html

OpenSSH has added TPM2 support and can import existing keys as of 1.7.0-rc0^[2]

https://blog.ledger.com/ssh-with-tpm/

The upstream TPM2 project is looking at consolidating the multiple code projects out there into an upstream implementation superseding the projects below.

https://github.com/tpm2-software/tpm2-tools/issues/518

https://github.com/tpm2-software/tpm2-pkcs11

To summarize in someone else's words:

TPMs are useful as an
alternative to key cards: they provide the same security against key
theft and the same cryptographic protections.  The main difference is
that TPMs are universally present in every laptop, so they provide a
simple and ubiquitous solution to key security.  The only real down
side is that unlike key cards, TPM protected keys cannot be transferred
between laptops, you must instead keep an offline backup copy of the
key can then be transferred to the TPM of any new laptop.

The way TPM protection works is slightly different from key cards.
 Instead of moving the key inside the card, the TPM converts any given
key to a TPM specific representation (meaning it's encrypted by a
special key that only the TPM possesses).  The TPM represenation must
be stored offline somewhere and if it is lost, so is the protected key.
 The way I implemented this is to use the TPM to convert the key to
protected representation and then store it in the shadow_info of a
shadowed-private-key using a shadow type of tpm2-v1.  The TPM can
handle an arbitrary number of keys, but the price is the shadow_info
stores the keys and must be preserved.

https://lwn.net/Articles/768419/

https://blog.hansenpartnership.com/using-your-tpm-as-a-secure-key-store/

Patches for GPG support for TPMs https://lists.gnupg.org/pipermail/gnupg-devel/2018-January/033350.html has been submitted but I have yet to see what happened. Needs following up for sure so it can make it before the next freeze hopefully.

To be done:

How to secure OpenSSH keys
Suggesting Tor supports this for Onion service keys and even session key material if TPM detected.

Contents of TPM can be encrypted in the libvirt version in Bullseye.

This nifty code supports the major applications on TPM 2

https://github.com/irtimmer/tpm2-pk11

https://github.com/irtimmer/tpm2-pk11/wiki RFP: https://bugs.debian.org/941951

   * OpenSSH Client (SSH key in TPM)
   * Firefox (Private key of Client certificate in TPM)
   * GnuPG using gnupg-pkcs11-scd (PGP key in TPM)

https://phabricator.whonix.org/T931

https://packages.debian.org/bookworm/simple-tpm-pk11

simple-tpm-pk11 provides tools to create a key in your TPM (Trusted Platform Module) chip which can then be used with SSH. The package comes with a library that you can use as “PKCS11Provider” in your SSH configuration file.

swtpm binary missing in Debian Buster and hasn't been packaged yet. ^[3] Opened a RFP. ^[4]

GPG 2.3 has added support for binding keys to TPM2 chips. Needs to be tested if it works with the vTPM. ^[5]

VirtIO-FS[edit]

As of Linux 5.4 a newer and faster implementation of shared folder filesystems has been merged upstream. This replaces the 9p fs requirement with a shared memory design that has better performance. Will look into changing shared folder service in the VM to use it by Bullseye. ^[6]

https://fosdem.org/2020/schedule/event/vai_virtio_fs/attachments/slides/3666/export/events/attachments/vai_virtio_fs/slides/3666/virtio_fs_A_Shared_File_System_for_Virtual_Machines_FOSDEM.pdf

There could be inferior security of this design compared to the 9p architecture. This uses the fuse protocol and allows direct access to host memory pages for increased performance. A Rust Implementation of virtiofsd is planned.

^[7]

Image Rebasing[edit]

https://documentation.suse.com/sles/11-SP4/#redirectmsg https://web.archive.org/web/20170523161045/http://nairobi-embedded.org/manipulating_disk_images_with_qemu-img.html

Play with image rebasing. Advantages: disk space savings, one time update for all derived images.

This feature can be used to create what Qubes calls "templates".

NAT Port Forwarding[edit]

Instructions for port forwarding to GW. Useful if configuring GW as Tor relay. Relevant if /when this is supported in Whonix https://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections

Virgl3D[edit]

~~Still cannot enable on Buster because of an QEMU process restrictions which were fixed upstream, but unfortunately after the freeze. ^[8]~~

https://www.kraxel.org/blog/2016/09/using-virtio-gpu-with-libvirt-and-spice/

Quote Gerd Hoffman (libvirt dev) vhost-user virtio gpu:

This is good from the security perspective, especially if you want use virgl 3D acceleration, and it also helps with opengl performance.

QXL has been superseded by virtio-vga where most development happens (and security attention probably). 2D acceleration as a distinct mechanism is outdated and 3D mode is where all types of graphical acceleration will be handled:

This device has support for 2D acceleration. This becomes more and more useless though as modern display devices don't have dedicated 2D acceleration support any more and use the 3D engine for everything. The same happens on the software side, modern desktops are rendering with opengl or vulkan instead of using 2D acceleration.

VMs with the same or related OS have the same canvas fingerprint. So Firefox in all Debian, Ubuntu etc VMs have the same fingerprint. Tor browser blocks fingerprinting, but Iceweasel doesn't. But if some bug allowed canvas fingerprinting in Whonix Tor Browser, it would be the same as Firefox in another Debian-related VM.

Asked upstream for help testing this:

libvirt: Do different Virgil3D users running have the same CANVAS fingerprint? bancfc at openmailbox.org

Given that the 3D enabled virtio-gpu is also a virtual GPU it should show the same model for all guests and therefore I predict it will be the same as with results of virtualized graphics without this feature.

https://listman.redhat.com/archives/libvir-list/2017-January/msg00296.html

Vulkan support is being added by Google to the virtio-gpu ^[9]

Demi Marie Obenour (Qubes core developer) quote:

VirGL and Venus run the userspace driver (OpenGL and Vulkan respectively) on the host. This means that they provide a hardware-independent API to the guest, but also means that the entire userspace stack becomes guest-accessable attack surface. This attack surface is very large, and Linux graphics developers have stated on IRC that it is not a security boundary. Therefore, @marmarek has decided that Qubes OS will never use VirGL or Venus and I agree with his decision.
virtGPU native contexts, on the other hand, expose the kernel ioctl API to the guest. This API is accessible to unprivileged userspace processes, which means it is a supported security boundary. It is also much smaller than the OpenGL or Vulkan APIs, which means that its attack surface is vastly smaller. As a bonus, native contexts offer near-native performance, which should be enough even for games and other demanding tasks.
The kernel ioctl API (also known as the userspace API or uAPI) is hardware-dependent, so virtGPU native contexts are only supported on a subset of hardware. Currently, Intel, AMD, and Adreno GPUs are supported using the upstream i915, amdgpu, and freedreno drivers.
Xen supports grant-based virtio, so virtio-GPU should not be incompatible with running QEMU in a stubdomain. The virtio-GPU emulator will need to run in dom0, but its job is much simpler than that of QEMU (it is only emulating a single device) and so the attack surface should (hopefully!) be acceptable.

todo: read https://www.kraxel.org/blog/2021/05/virtio-gpu-qemu-graphics-update/

Audit Output of virsh domxml-to-native[edit]

virsh domxml-to-native qemu-argv ./usr/share/whonix-libvirt/xml/Whonix-Gateway.xml

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/bin/qemu-system-x86_64 -name Whonix-Gateway -machine ubuntu,accel=tcg,usb=off -cpu qemu64,-kvmclock,+kvm_pv_eoi,+kvm_pv_unhalt -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid d67e18a8-ea3c-4c6d-81eb-99a6324506a6 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Whonix-Gateway.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/Whonix-Gateway.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:6d:37:bc,bus=pci.0,addr=0x3 -netdev tap,id=hostnet1 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ac:89:1e,bus=pci.0,addr=0x4 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5901,addr=127.0.0.1,disable-ticketing,disable-copy-paste,disable-agent-file-xfer,seamless-migration=on -device qxl-vga,id=video0,ram_size=262144000,vram_size=262144000,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x5 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -object rng-random,id=rng0,filename=/dev/random -device virtio-rng-pci,rng=rng0,bus=pci.0,addr=0x9 -msg timestamp=on

Same but rewritten for better readability.

LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin \
QEMU_AUDIO_DRV=spice \
/usr/bin/qemu-system-x86_64 \
-name Whonix-Gateway \
-machine ubuntu,accel=tcg,usb=off \
-cpu qemu64,-kvmclock,+kvm_pv_eoi,+kvm_pv_unhalt \
-m 512 \
-realtime mlock=off  \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid d67e18a8-ea3c-4c6d-81eb-99a6324506a6 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Whonix-Gateway.monitor,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,clock=vm,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-no-hpet \
-no-shutdown \
-global PIIX4_PM.disable_s3=1 \
-global PIIX4_PM.disable_s4=1 \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 \
-drive file=/var/lib/libvirt/images/Whonix-Gateway.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-netdev tap,id=hostnet0 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:6d:37:bc,bus=pci.0,addr=0x3 \
-netdev tap,id=hostnet1 \
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:ac:89:1e,bus=pci.0,addr=0x4 \
-chardev spicevmc,id=charchannel0,name=vdagent \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \
-device usb-tablet,id=input0 \
-spice port=5901,addr=127.0.0.1,disable-ticketing,disable-copy-paste,disable-agent-file-xfer,seamless-migration=on \
-device qxl-vga,id=video0,ram_size=262144000,vram_size=262144000,bus=pci.0,addr=0x2 \
-device intel-hda,id=sound0,bus=pci.0,addr=0x5 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 \
-object rng-random,id=rng0,filename=/dev/random \
-device virtio-rng-pci,rng=rng0,bus=pci.0,addr=0x9 \
-msg timestamp=on

Does it look sense? Do we understand all of it? Is there anything we should change?

HulaHoop: Output makes sense. Needs to be updated because of recent config changes. Some upcoming.

See CI log for output of running against other xml files.

- Tested the same command on Debian+KVM+GW,WS,Kicksecure and the output is:

Whonix-Gateway[edit]


user@host:~/derivative-maker/packages/kicksecure/libvirt-dist/usr/share/libvirt-dist/xml$ sudo virsh domxml-to-native qemu-argv Whonix-Gateway.xml
    LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin USER=root HOME=/var/lib/libvirt/qemu/domain–1-Whonix-Gateway XDG_DATA_HOME=/var/lib/libvirt/qemu/domain–1-Whonix-Gateway/.local/share XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain–1-Whonix-Gateway/.cache XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain–1-Whonix-Gateway/.config /usr/bin/qemu-system-x86_64 -name guest=Whonix-Gateway,debug-threads=on -S -object ‘{“qom-type”:“secret”,“id”:“masterKey0”,“format”:“raw”,“file”:“/var/lib/libvirt/qemu/domain–1-Whonix-Gateway/master-key.aes”}’ -machine pc-i440fx-8.2,usb=off,vmport=off,dump-guest-core=off,mem-merge=off,memory-backend=pc.ram,hpet=off,acpi=on -accel kvm -cpu host,migratable=on,kvmclock=on,kvm-pv-unhalt=on,pmu=off -m size=524288k -object ‘{“qom-type”:“memory-backend-ram”,“id”:“pc.ram”,“size”:536870912}’ -overcommit mem-lock=off -smp 1,sockets=1,cores=1,threads=1 -uuid c567c4e6-f750-4ecb-ab90-321012c3d927 -device ‘{“driver”:“vmgenid”,“guid”:“99f50c12-3f81-4339-b390-c2fba5587b2c”,“id”:“vmgenid0”}’ -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain–1-Whonix-Gateway/monitor.sock,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ‘{“driver”:“piix3-usb-uhci”,“id”:“usb”,“bus”:“pci.0”,“addr”:“0x1.0x2”}’ -device ‘{“driver”:“virtio-serial-pci”,“id”:“virtio-serial0”,“bus”:“pci.0”,“addr”:“0x5”}’ -blockdev ‘{“driver”:“file”,“filename”:“/var/lib/libvirt/images/Whonix-Gateway.qcow2”,“node-name”:“libvirt-1-storage”,“auto-read-only”:true,“discard”:“unmap”}’ -blockdev ‘{“node-name”:“libvirt-1-format”,“read-only”:false,“driver”:“qcow2”,“file”:“libvirt-1-storage”}’ -device ‘{“driver”:“virtio-blk-pci”,“bus”:“pci.0”,“addr”:“0x6”,“drive”:“libvirt-1-format”,“id”:“virtio-disk0”,“bootindex”:1}’ -netdev ‘{“type”:“tap”,“fd”:“17”,“vhost”:true,“vhostfd”:“28”,“id”:“hostnet0”}’ -device ‘{“driver”:“virtio-net-pci”,“netdev”:“hostnet0”,“id”:“net0”,“mac”:“52:54:00:78:69:11”,“bus”:“pci.0”,“addr”:“0x3”}’ -netdev ‘{“type”:“tap”,“fd”:“29”,“vhost”:true,“vhostfd”:“30”,“id”:“hostnet1”}’ -device ‘{“driver”:“virtio-net-pci”,“netdev”:“hostnet1”,“id”:“net1”,“mac”:“52:54:00:f5:5c:a2”,“bus”:“pci.0”,“addr”:“0x4”}’ -chardev pty,id=charserial0 -device ‘{“driver”:“isa-serial”,“chardev”:“charserial0”,“id”:“serial0”,“index”:0}’ -chardev spicevmc,id=charchannel0,name=vdagent -device ‘{“driver”:“virtserialport”,“bus”:“virtio-serial0.0”,“nr”:1,“chardev”:“charchannel0”,“id”:“channel0”,“name”:“com.redhat.spice.0”}’ -audiodev ‘{“id”:“audio1”,“driver”:“spice”}’ -spice port=5901,addr=127.0.0.1,disable-ticketing=on,disable-agent-file-xfer=on,seamless-migration=on -device ‘{“driver”:“virtio-vga”,“id”:“video0”,“max_outputs”:1,“bus”:“pci.0”,“addr”:“0x2”}’ -object ‘{“qom-type”:“rng-random”,“id”:“objrng0”,“filename”:“/dev/urandom”}’ -device ‘{“driver”:“virtio-rng-pci”,“rng”:“objrng0”,“id”:“rng0”,“bus”:“pci.0”,“addr”:“0x7”}’ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=

Whonix-Workstation[edit]


user@host:~/derivative-maker/packages/kicksecure/libvirt-dist/usr/share/libvirt-dist/xml$ sudo virsh domxml-to-native qemu-argv Whonix-Workstation.xml
    LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin USER=root HOME=/var/lib/libvirt/qemu/domain–1-Whonix-Workstation XDG_DATA_HOME=/var/lib/libvirt/qemu/domain–1-Whonix-Workstation/.local/share XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain–1-Whonix-Workstation/.cache XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain–1-Whonix-Workstation/.config /usr/bin/qemu-system-x86_64 -name guest=Whonix-Workstation,debug-threads=on -S -object ‘{“qom-type”:“secret”,“id”:“masterKey0”,“format”:“raw”,“file”:“/var/lib/libvirt/qemu/domain–1-Whonix-Workstation/master-key.aes”}’ -machine pc-i440fx-8.2,usb=off,vmport=off,dump-guest-core=off,mem-merge=off,memory-backend=pc.ram,hpet=off,acpi=on -accel kvm -cpu host,migratable=on,kvmclock=off,kvm-pv-unhalt=on,pmu=off -m size=1572864k -object ‘{“qom-type”:“memory-backend-ram”,“id”:“pc.ram”,“size”:1610612736}’ -overcommit mem-lock=off -smp 1,sockets=1,cores=1,threads=1 -uuid aebc953d-a3ef-40f4-a080-bbddd0bc9351 -device ‘{“driver”:“vmgenid”,“guid”:“fd6de7ed-c6d3-43c0-9325-0ba2d4394069”,“id”:“vmgenid0”}’ -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain–1-Whonix-Workstation/monitor.sock,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -global kvm-pit.lost_tick_policy=delay -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ‘{“driver”:“piix3-usb-uhci”,“id”:“usb”,“bus”:“pci.0”,“addr”:“0x1.0x2”}’ -device ‘{“driver”:“virtio-serial-pci”,“id”:“virtio-serial0”,“bus”:“pci.0”,“addr”:“0x5”}’ -blockdev ‘{“driver”:“file”,“filename”:“/var/lib/libvirt/images/Whonix-Workstation.qcow2”,“node-name”:“libvirt-1-storage”,“auto-read-only”:true,“discard”:“unmap”}’ -blockdev ‘{“node-name”:“libvirt-1-format”,“read-only”:false,“driver”:“qcow2”,“file”:“libvirt-1-storage”}’ -device ‘{“driver”:“virtio-blk-pci”,“bus”:“pci.0”,“addr”:“0x6”,“drive”:“libvirt-1-format”,“id”:“virtio-disk0”,“bootindex”:1}’ -netdev ‘{“type”:“tap”,“fd”:“26”,“vhost”:true,“vhostfd”:“28”,“id”:“hostnet0”}’ -device ‘{“driver”:“virtio-net-pci”,“netdev”:“hostnet0”,“id”:“net0”,“mac”:“52:54:00:51:33:f5”,“bus”:“pci.0”,“addr”:“0x3”}’ -chardev pty,id=charserial0 -device ‘{“driver”:“isa-serial”,“chardev”:“charserial0”,“id”:“serial0”,“index”:0}’ -chardev spicevmc,id=charchannel0,name=vdagent -device ‘{“driver”:“virtserialport”,“bus”:“virtio-serial0.0”,“nr”:1,“chardev”:“charchannel0”,“id”:“channel0”,“name”:“com.redhat.spice.0”}’ -audiodev ‘{“id”:“audio1”,“driver”:“spice”}’ -spice port=5901,addr=127.0.0.1,disable-ticketing=on,disable-copy-paste=on,disable-agent-file-xfer=on,seamless-migration=on -device ‘{“driver”:“virtio-vga”,“id”:“video0”,“max_outputs”:1,“bus”:“pci.0”,“addr”:“0x2”}’ -device ‘{“driver”:“intel-hda”,“id”:“sound0”,“bus”:“pci.0”,“addr”:“0x4”}’ -device ‘{“driver”:“hda-output”,“id”:“sound0-codec0”,“bus”:“sound0.0”,“cad”:0,“audiodev”:“audio1”}’ -object ‘{“qom-type”:“rng-random”,“id”:“objrng0”,“filename”:“/dev/urandom”}’ -device ‘{“driver”:“virtio-rng-pci”,“rng”:“objrng0”,“id”:“rng0”,“bus”:“pci.0”,“addr”:“0x7”}’ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

Kicksecure[edit]


user@host:~/derivative-maker/packages/kicksecure/libvirt-dist/usr/share/libvirt-dist/xml$ sudo virsh domxml-to-native qemu-argv Kicksecure.xml
    LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin USER=root HOME=/var/lib/libvirt/qemu/domain–1-Kicksecure XDG_DATA_HOME=/var/lib/libvirt/qemu/domain–1-Kicksecure/.local/share XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain–1-Kicksecure/.cache XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain–1-Kicksecure/.config /usr/bin/qemu-system-x86_64 -name guest=Kicksecure,debug-threads=on -S -object ‘{“qom-type”:“secret”,“id”:“masterKey0”,“format”:“raw”,“file”:“/var/lib/libvirt/qemu/domain–1-Kicksecure/master-key.aes”}’ -machine pc-i440fx-8.2,usb=off,vmport=off,dump-guest-core=off,mem-merge=off,memory-backend=pc.ram,hpet=off,acpi=on -accel kvm -cpu host,migratable=on,kvmclock=off,kvm-pv-unhalt=on,pmu=off -m size=1572864k -object ‘{“qom-type”:“memory-backend-ram”,“id”:“pc.ram”,“size”:1610612736}’ -overcommit mem-lock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 51458204-52f4-4400-a335-8c491c0d2552 -device ‘{“driver”:“vmgenid”,“guid”:“b0b49359-2f5e-46d5-aa38-0b0e55751e06”,“id”:“vmgenid0”}’ -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain–1-Kicksecure/monitor.sock,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -global kvm-pit.lost_tick_policy=delay -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ‘{“driver”:“piix3-usb-uhci”,“id”:“usb”,“bus”:“pci.0”,“addr”:“0x1.0x2”}’ -device ‘{“driver”:“virtio-serial-pci”,“id”:“virtio-serial0”,“bus”:“pci.0”,“addr”:“0x5”}’ -blockdev ‘{“driver”:“file”,“filename”:“/var/lib/libvirt/images/Kicksecure.qcow2”,“node-name”:“libvirt-1-storage”,“auto-read-only”:true,“discard”:“unmap”}’ -blockdev ‘{“node-name”:“libvirt-1-format”,“read-only”:false,“driver”:“qcow2”,“file”:“libvirt-1-storage”}’ -device ‘{“driver”:“virtio-blk-pci”,“bus”:“pci.0”,“addr”:“0x6”,“drive”:“libvirt-1-format”,“id”:“virtio-disk0”,“bootindex”:1}’ -netdev ‘{“type”:“tap”,“fd”:“26”,“vhost”:true,“vhostfd”:“28”,“id”:“hostnet0”}’ -device ‘{“driver”:“virtio-net-pci”,“netdev”:“hostnet0”,“id”:“net0”,“mac”:“52:54:00:33:6a:10”,“bus”:“pci.0”,“addr”:“0x3”}’ -chardev pty,id=charserial0 -device ‘{“driver”:“isa-serial”,“chardev”:“charserial0”,“id”:“serial0”,“index”:0}’ -chardev spicevmc,id=charchannel0,name=vdagent -device ‘{“driver”:“virtserialport”,“bus”:“virtio-serial0.0”,“nr”:1,“chardev”:“charchannel0”,“id”:“channel0”,“name”:“com.redhat.spice.0”}’ -audiodev ‘{“id”:“audio1”,“driver”:“spice”}’ -spice port=5901,addr=127.0.0.1,disable-ticketing=on,disable-copy-paste=on,disable-agent-file-xfer=on,seamless-migration=on -device ‘{“driver”:“virtio-vga”,“id”:“video0”,“max_outputs”:1,“bus”:“pci.0”,“addr”:“0x2”}’ -device ‘{“driver”:“intel-hda”,“id”:“sound0”,“bus”:“pci.0”,“addr”:“0x4”}’ -device ‘{“driver”:“hda-output”,“id”:“sound0-codec0”,“bus”:“sound0.0”,“cad”:0,“audiodev”:“audio1”}’ -object ‘{“qom-type”:“rng-random”,“id”:“objrng0”,“filename”:“/dev/urandom”}’ -device ‘{“driver”:“virtio-rng-pci”,“rng”:“objrng0”,“id”:“rng0”,“bus”:“pci.0”,“addr”:“0x7”}’ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

Kernel Direct Boot[edit]

Minor importance.

Document how to do a kernel direct boot. For passing special options for debugging options such as disabling apparmor in kernel. Interesting, because even if grub was broken for some reason, VM would still be bootable.

HulaHoop: There is an example here: https://libvirt.org/formatdomain.html#direct-kernel-boot feel free to change it to suit Whonix.

Hardware side-channel research[edit]

Ongoing. Interesting developments worth watching out for at https://gruss.cc

Host to VM Communication[edit]

Name of chapter "Host to VM Communication" might be wrong. Any better idea?

VirtualBox supports VBoxManage setextradata which can be used on the host to define a pair of setting and content. The host can set arbitrary information which can be used inside the VM.

https://www.virtualbox.org/manual/ch09.html#changeacpicust

Does KVM support something similar?

This would be useful for Whonix-Host. Some setting could be set on Whonix-Host which then could be read inside Whonix KVM VMs. That would be useful to spare users from going again through setup-dist (disclaimer) and perhaps anon-connection-wizard (ACW).

This also talks about KVM: https://serverfault.com/questions/727347/solidworks-activation-license-mode-is-not-supported-in-this-virtual-environment

  <qemu:commandline>
    <qemu:arg value='type=0,vendor=LENOVO,version=FBKTB4AUS,date=07/01/2015,release=1.180'/>
  </qemu:commandline>

Maybe the same mechanism could be used.

  <qemu:commandline>
    <qemu:arg value='type=0,disclaimerdone=yes'/>
  </qemu:commandline>

What is type=0 vs type=1?

Other alternatives... Or does KVM nowadays support SLIC tables?

Something else.

Not good: network based communications. To keep this simple/reliable some of the above mechanisms would be better.

Forum discussion: https://forums.whonix.org/t/host-to-kvm-vm-communication-using-slic-setextradata/10619

In Progress[edit]

Shared Folder[edit]

Run as non-root!

groups

Check if being a member of groups:

libvirt
kvm

Create folder /mnt/shared.

sudo mkdir -p /mnt/shared

Optional. Should already be the case. Make sure folder /mnt is owned by user and group root.

sudo chown root:root /mnt

Optional. Should already be the case. Make sure folder /mnt is readable by any linux user account and linux user group ("public").

sudo chmod o+r /mnt

https://askubuntu.com/questions/862289/difference-between-executable-directory-vs-executable-files

sudo chmod o+x /mnt

Set owner of folder /mnt/shared to user libvirt and group kvm.

sudo chown --recursive libvirt:kvm /mnt/shared

Permission hardening. Prevent "public" from reading the shared folder.

sudo chmod --recursive o-rwx /mnt/shared

Allow folder /mnt/shared to be readable and writeable by user libvirt and group kvm.

sudo chmod --recursive ug+rw /mnt/shared

https://askubuntu.com/questions/862289/difference-between-executable-directory-vs-executable-files

sudo chmod ug+x /mnt/shared

Done[edit]

QCOW2 Preallocation Settings[edit]

Preallocating metadata speeds up disk use in guest at expense of time during disk creation.

https://web.archive.org/web/20220929043923/https://www.jamescoyle.net/how-to/2060-qcow2-physical-size-with-different-preallocation-settings

Other options like falloc and full cause the disk to loose sparse properties.

QCOW2 L2 Subcluster Allocation[edit]

https://blogs.igalia.com/berto/2020/12/03/subcluster-allocation-for-qcow2-images/

Problem:

Because of that qcow2 images with larger cluster sizes tend to:

grow faster, wasting more disk space and duplicating data. increase the amount of necessary I/O during cluster allocation,reducing the allocation performance.

Why reducing cluster size is not a solution:

Unfortunately, reducing the cluster size is in general not an option because it also has an impact on the amount of metadata used internally by qcow2 (reference counts, guest-to-host cluster mapping). Decreasing the cluster size increases the number of clusters and the amount of necessary metadata. This has direct negative impact on I/O performance, which can be mitigated by caching it in RAM, therefore increasing the memory requirements (the aforementioned post covers this in more detail).

The new extended L2 entries feature in QEMU 5.2+ gives us the best of all worlds:


*Sixteen times less metadata per unit of allocation, greatly reducing the amount of necessary L2 cache.
*Much faster I/O during allocating when the image has a backing file, up to 10-15 times more I/O operations per second for the same cluster size in my tests (see chart below).
*Smaller images and less duplication of data.

How to use:

In order to test this you simply need to create an image with extended_l2=on, and you also probably want to use a larger cluster size (the default is 64 KB, remember that every cluster has 32 subclusters). Here is an example:

$ qemu-img create -f qcow2 -o extended_l2=on,cluster_size=128k img.qcow2 1T

And that’s all you need to do. Once the image is created all allocations will happen at the subcluster level.

Change commit: https://github.com/adrelanos/Whonix/pull/1/commits/16f5404eb929b6748178948ceb509211c95a016c

Biometric Fingerprinting[edit]

This problem is related to all hypervisors and noted here to keep track. New info added not in mail list message:

Upstream tickets:

https://lists.nongnu.org/archive/html/qemu-discuss/2016-03/msg00023.html

https://web.archive.org/web/20161002204349/https://secure-os.org/pipermail/desktops/2016-March/000108.html

https://bugzilla.redhat.com/show_bug.cgi?id=1318813

https://marc.info/?l=linux-kernel&m=145877344732456&w=2

https://lists.freedesktop.org/archives/wayland-devel/2016-March/027607.html

https://lists.x.org/archives/xorg-devel/2016-March/049159.html

https://www.phoronix.com/news/Anti-Keystroke-FP-Wayland-Kern

https://mailman.stanford.edu/pipermail/liberationtech/2016-March/015947.html

https://web.archive.org/web/20170222094419/https://mailman.boum.org/pipermail/tails-dev/2016-March/010485.html

https://lists.torproject.org/pipermail/tor-talk/2015-July/038624.html

Testing sites listed here: https://forums.whonix.org/t/help-wanted-kvm-users-biometrics-fingerprinting-testing/2233

https://cgit.freedesktop.org/spice/spice-gtk/tree/src/channel-inputs.c

https://git.qemu.org/?p=qemu.git;a=blob;f=hw/input/ps2.c

Virtualization is useful in presenting an identical environment and set of "hardware" for each user which goes a long way in creating an anonymity set of systems. That way a system attacker, advertisers and online trackers would not be able to fingeprint a user or their hardware.

The problem: Tracking techniques have become more sophisticated with time. They advanced from simple cookies to browser/device fingerprinting (which Tor Browser focuses on defeating) to user behavior fingerprinting. The latter is about profiling how a user types on a keyboard or uses a mouse [2].

Keystroke dynamics is a super creepy way to track users based on how long they press keys (dwell time) and the time between key presses (gap time). This is extremely accurate at identifying individuals because of how unique these measurements are. Advertising networks (Google, Facebook...) that fingeprprint users on both the clearnet and Tor can deanonymize users. This technique is already actively used in the wild [6][7].

Potential Solutions:

Since input devices are all emulated its a great opportunity to stop this profiling technique.

A security researcher designed a proof of concept plugin for Chrome browser that mitigates this. Implementing something like the PoC addon in [1] known as KeyBoardPrivacy. Some random delay in milliseconds in a 50 millisecond range for dwell and gap times for the emulated keyboards is enough to skew the values to render this attack useless while not affecting performance.

The changes made to Tor Borwser to make JS timers more coarse grained but constant (250ms for keyboard events) were not enough to stop keystroke dynamics fingerprinting because a malicious script can evict the cache and allow extrapolation of true timing events within 1-5ms accuracy .[3][5] Their goal is to instead add jitter to the timers [4]. A similar solution proposed in [4] can be implemented in all QEMU-KVM timers to mitigate both attacks.

EDIT: Second solution not the way to go. High precision timers in virtualizers have bad security implications. They enable sidechannel attacks on crypto operations running on other VMs as well as aid in biometric fingerprinting. A multi hypervisor study on timer accuracy shows most have high accuracy[8] by necessity. Increasing coarseness in the hypervisor's code can lead to guest slow downs or instability. This varies by hypervisor. CPU load does NOT affect timer accuracy [in KVM] however IO does. A stress/stress-ng daemon installed on GW that runs some varying workload may introduce a skew to timer data.

EDIT 2:

Even if the user's destination does not itself surreptitiously sniff their biometrics, any one observing the traffic of any interactive JS application can generate a model for your biometric statistics. [16]

[1] https://paul.reviews/behavioral-profiling-the-password-you-cant-change/

[2] https://web.archive.org/web/20221114004700/http://jcarlosnorte.com/security/2016/03/06/advanced-tor-browser-fingerprinting.html

[3] https://www.lightbluetouchpaper.org/2015/07/30/double-bill-password-hashing-competition-keyboardprivacy/#comment-1288166

[4] https://gitlab.torproject.org/legacy/trac/-/issues/16110

[5] https://gitlab.torproject.org/legacy/trac/-/issues/1517

[6] https://web.archive.org/web/20200430120856/http://scraping.pro/no-captcha-recaptcha-challenge/

[7] https://nakedsecurity.sophos.com/2013/11/01/facebook-to-silent-track-users-cursor-movements-to-see-which-ads-we-like-best/

[8] Achieving High Resolution Timer Events in Virtualized Environment https://web.archive.org/web/20220716012044/http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4503740/pdf/pone.0130887.pdf

[9] https://archive.ph/20160528163257/http://dataprotectioncenter.com/featured/keystroke-profiling/

[10] https://arstechnica.com/information-technology/2015/07/how-the-way-you-type-can-shatter-anonymity-even-on-tor/

[11] Spoofing key-press latencies with a generative keystroke dynamics model [DOI: 10.1109/BTAS.2015.7358795] https://ieeexplore.ieee.org/document/7358795

[12] "I Am Robot: (Deep) Learning to Break Semantic Image CAPTCHAs" https://web.archive.org/web/20220224013800/http://www.cs.columbia.edu/~polakis/papers/sivakorn_eurosp16.pdf

[13] I can be You: Questioning the use of Keystroke Dynamics as Bio metrics https://web.archive.org/web/20220331095121/http://flyer.sis.smu.edu.sg/ndss13-tey.pdf

[14] How I reverse-engineered Google Docs to play back any document's keystrokes https://news.ycombinator.com/item?id=8562483

[15] An Efficient User Verification System via Mouse Movements [DOI=10.1.1.220.9410] https://web.archive.org/web/20190303210911/http://www.cs.wm.edu/~hnw/paper/ccs11.pdf

[16] Knowing the User’s Every Move – User Activity Tracking for Website Usability Evaluation and Implicit Interaction https://www.medien.ifi.lmu.de/pubdb/publications/pub/atterer2006www/atterer2006www.pdf

[17] Eliminating Fine Grained Timers in Xen https://cseweb.ucsd.edu/~bvattikonda/docs/xentimers-ccsw11.pdf

[18] https://web.archive.org/web/20220526220057/https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3835878/#B13 [19] https://www.cs.cmu.edu/~maxion/pubs/KillourhyMaxion08a.pdf (The Effect of Clock Resolution on Keystroke Dynamics) millisecond resolution is vital

[20] https://web.archive.org/web/20210412030235/https://www.comp.nus.edu.sg/~tsim/documents/keystroke-dynamics-published.pdf (Keystroke Dynamics in a General Setting) Compared fingerprinting users in a general setting in contrast to typing a password. Measured dynamics of typing common words. Another surprising finding is non-English users were even more susceptible to keystroke fingerprinting than English ones.

higher level behavior fingerprinting:

https://www.researchgate.net/profile/Roman_Yampolskiy/publication/256169398_Game_Playing_Tactic_as_a_Behavioral_Biometric_for_Human_Identification/links/569c3c1d08aea1476954742c.pdf

higher level "cognitive" behavioral biometrics are application specific. That paper uses descriptive statistics for actions that are very specific to the game and don't really apply anywhere else.

Mouse Tracking[edit]

User re-authentication via mouse movements https://web.archive.org/web/20230519163324/https://csis.pace.edu/~ctappert/it691-projects/mouse-pusara.pdf

Relatively successful tracking active readers however mouse movements are not enough to fingerprint users. That was back in 2004.

ON USING MOUSE MOVEMENTS AS A BIOMETRIC https://www.cs.sjsu.edu/faculty/pollett/papers/shivanipaper.pdf https://www.cs.sjsu.edu/faculty/pollett/masters/Semesters/Spring04/Shivani/CS298Presentation.pdf

Needs more work to achieve high accuracy. 2004.

An Efficient User Verification System via Mouse Movements https://web.archive.org/web/20190303210911/http://www.cs.wm.edu/~hnw/paper/ccs11.pdf

high accuracy achieved in limited situations - actice authentication during log-on. Does not clear EU false positive requirements however so they recommend it for combining with keystroke dynamics as extra confirmation:

In the scenario of static verification, a user is required to perform a series of mouse movements and its mouse data is verified within a certain amount of time (e.g., login time). A good example of this scenario is a click-based graphical password for user login, where five clicks are estimated to be made in no more than 25 seconds.

By contrast, in the scenario of continuous verification, a user’s mouse data is continuously collected and verified throughout the entire session. This is non-intrusive to users and meets the goal of passive monitoring. However, the frequency of user mouse actions varies significantly in different sessions. In general, the average frequency of user mouse actions will be much lower than that of the static scenario.

Even for the same user at different times, the number of mouse events per unit time varies a lot. However, to the best of our knowledge, our work is the first to achieve high accuracy with a reasonably small number of mouse events.

[1] http://www.ijicic.org/ijicic-ksi-03.pdf - User Authentication using Rhythm Click Characteristics for Non-Keyboard Devices

[2] https://web.archive.org/web/20160617233804/http://www.ibm.com/developerworks/library/os-userauth-mouse/index.html - IBM perl guide for fingerprinting mouse click-hold times

search terms:

Mouse movement authentication

Solutions[edit]

Researcher Vinnie Monaco has kindly volunteered to write tools that mitigate these attacks:

anti-keystroke defense: https://github.com/vmonaco/kloak

mouse movement defense: Upcoming

clocksource[edit]

cat /sys/devices/system/clocksource/clocksource0/available_clocksource

refined-jiffies jiffies

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

refined-jiffies

https://forums.whonix.org/t/available-clocksource-current-clocksource

Upcoming XML Changes - Stable Next[edit]

When Stretch hits stable disable:

smm which is not even supported by qemu yet.

Check if switch to virtio-input devices means virtio=serial can now be removed.

Apply Clean-filter traffic?[edit]

Advantages are its easy to set a new IP for a new WS. Has many rules to prevent MAC, IP, ARP spoofing. Can also protect the GW from DoS by using connection tracking and rate-limiting.

Sounds good but I still think a completely separate internal network is the best option to ensure separartion between multi WSs. The GW should be able to withstand DoS on its own or there will be big problems. I think its safer if libvirt knows nothing about the traffic on the internal network and does not parse anything on there.

To have a multi-WS implementation with different internal networks likely requires a big effort on the Whonix development side. Not worth it when setting a new GW is easy and costs little resources - also provides more assurance against theoretical attacks on a shared GW.

RNG[edit]

Rate limit[edit]

This is necessary to prevent malicious entropy starvation of other VMs and the host.

Test code for /dev/random throughput ^[10]

dd if=/dev/random bs=1k count=1

Re-reading the article it seems the choice made is OK. The limit sets the max a guest can use to be 25% of that of the host.

/dev/random provides high quality numbers but stops when entropy is poor: the Linux kernel keeps a entropy pool of 4 KB (see /proc/sys/kernel/random/poolsize); when the pool is empty, the kernel blocks causing a delay until it can provide the requested random numbers.

https://www.certdepot.net/rhel7-get-started-random-number-generator/

Note: Just after <rng model=’virtio’>, you can add a line like <rate bytes=’1024′ period=’1000’/> to prevent a guest from consuming more than 1 KB per second of pseudo random numbers. Otherwise, one single guest could consume all the available pseudo random numbers at the expense of all the others.

Also applied for GW since some crypto requiring operations under the control of WS processes can abuse the entropy available to it if unchecked.

Haveged[edit]

Cross-posted

   haveged relies on the RDTSC instruction, that apparently is useless in some virtualized environments. Also, the quality of random numbers output by HAVEGE is unclear, and the topic of many discussions.

https://tails.boum.org/contribute/design/random/#rngd

haveged is not useful on KVM

rdtsc or any tsc counter provides very accurate clock tick information so better to block.

blkiotune[edit]

Its possible to limit IO throughput per VM however with disk speeds varying a lot it will be impossible to find an option that makes sense for everybody. Nonetheless still interesting to have. Documented below is a IO test and how to set limits.

https://fedoraproject.org/wiki/QA:Testcase_Virtualization_IO_Throttling

Secure Clipboard[edit]

https://bugzilla.redhat.com/show_bug.cgi?id=1320263

https://gitlab.freedesktop.org/spice/spice-gtk/-/issues/97

CPU Masking[edit]

KVM cpus support a baseline of features by default. You can mask out the problematic ones and don't have to worry about the extra ones it doesn't support because it will be masked out anyhow (because it was never supported in the first place).

The only bad instructions we should filter out are a subset of whatever instructions are listed under the virtual cpu from the output of the cpu_map.xml list

cat /usr/share/libvirt/cpu_map.xml

I figured out safe defaults and will do a pull request. NB clflush was abused to carry out the rowhammer attack so its blacklisted.aes will be passed through for crypto performance - it doesn't mess with random number generation.

<cpu mode='custom' match='exact'>
   <model fallback='forbid'>qemu64</model>
   <topology sockets='1' cores='2' threads='1'/>
   <feature policy='disable' name='tsc'/>
   <feature policy='disable' name='clflush'/>
   <feature policy='optional' name='aes'/>
</cpu>

https://libvirt.org/formatdomain.html#cpu-model-and-topology

Informative link about cpu flag functionality: https://unix.stackexchange.com/questions/43539/what-do-the-flags-in-proc-cpuinfo-mean

https://www.berrange.com/posts/2010/02/15/guest-cpu-model-configuration-in-libvirt-with-qemukvm/

"Every hypervisor has its own policies for what a guest will see for its CPUs by default, Xen just passes through the host CPU, with QEMU/KVM the guest sees a generic model called “qemu32” or “qemu64”. "

cat output:

     <model name='qemu64'>
      <!-- These are supported only by TCG.  KVM supports them only if the
           host does.  So we leave them out:

           <feature name='abm'/>
           <feature name='lahf_lm'/>
           <feature name='popcnt'/>
           <feature name='sse4a'/>
      -->
      <feature name='apic'/>
      <feature name='clflush'/>
      <feature name='cmov'/>
      <feature name='cx16'/>
      <feature name='cx8'/>
      <feature name='de'/>
      <feature name='fpu'/>
      <feature name='fxsr'/>
      <feature name='lm'/>
      <feature name='mca'/>
      <feature name='mce'/>
      <feature name='mmx'/>
      <feature name='msr'/>
      <feature name='mtrr'/>
      <feature name='nx'/>
      <feature name='pae'/>
      <feature name='pat'/>
      <feature name='pge'/>
      <feature name='pni'/>
      <feature name='pse'/>
      <feature name='pse36'/>
      <feature name='sep'/>
      <feature name='sse'/>
      <feature name='sse2'/>
      <feature name='svm'/>
      <feature name='syscall'/>
      <feature name='tsc'/>
    </model>

Passed to Command Line[edit]

Minor importance.

Libvirt "only" runs the qemu-kvm command line utility with the correct command line parameters?

Is there a way to make libvirt tell with what parameters libvirt is calling qemu-kvm?

HulaHoop: Yes its possible. Conversion of commands is supported in bot directions. See documentation here: https://libvirt.org/drvqemu.html#converting-from-domain-xml-to-qemu-args

Nice and useful. Added to unit_test:
https://github.com/Whonix/whonix-libvirt/commit/4bcac51b264b9d78042899be005e4c06549224ab

Haveged test for KVM + virtio-rng Result[edit]

NB Patrick has conducted the tests without virtio-rng and they worked too.

Reults with virtio-rng:

0 failed individual tests with THRESHOLD 0.001000 on 514 individual tests
PASS: nist/test.sh
==================
All 2 tests passed
==================
make[2]: Leaving directory `/home/user/haveged-1.4'
make[1]: Leaving directory `/home/user/haveged-1.4'

Two Separate Whonix-Gateway^™[edit]

https://www.whonix.org/old-forum/index.php/topic,397.msg2973.html#new Possible and documented.

Preserve Comments in libvirt / virsh XML Files[edit]

Asked on stackexchange:
https://unix.stackexchange.com/questions/148163/is-it-possible-to-use-comments-in-virsh-libvirt-xml-files

Asked on libvirt mailinglist and here is their response: https://listman.redhat.com/archives/libvir-list/2014-August/msg00681.html

A description attribute can be used to carry human readable metadata.

Activate Watchdog?[edit]

libvirt can be told to ignore a guest's request for restart. The actual action would be a silent shutdown. Same for when a crash happens, libvirt would just leave the machine in its shutdown state instead of booting it again.

This all hinges on what you think about the attack i described earlier. Should we just advise people not to do things on their host that would leak timestamps? Like browsing for instance. ~~From what I got from the wiki, this is the only thing done on a Linux host that could leak this information.~~

Later I learned that TCP Sequence numbers could still leak time information, but its more difficult to exploit than other methods.

Patrick wrote: "the attack i described earlier" in unclear for other readers and I am also not so sure about it anymore.

Patrick wrote: Not sure exactly what you want to defend here, but I guess this info might invalid its premises. The ACPI reset command is not required. An adversary could simply use kexec. It allows to reboot a machine without ACPI reset command. kexec's purpose is to save time by not going through BIOS initialization routines. Therefore KVM ignoring ACPI reset command probably won't be a big help.

HulaHoop: Agreed. I went ahead and re-added it.

Add pvspinlock when supported in more recent version of libvirt[edit]

Was not accepting it because its a fresh feature with not support in the version of libvirt I run. Its been added as of libvirt 1.2 as per the reply. ^[11]

HulaHoop: Done.

Preconfigured Shared Folders[edit]

Patrick asks: For better usability, should we add the necessary settings for shared folders by default to whonix_gateway.xml and whonix_workstation.xml?

Patrick comments: I could then, if kvm is detected (easy using virt-what), automatically mount the shared folder in Whonix VMs. Could be implemented into the https://github.com/{{project_name_short}}/shared-folder-help shared-folder-help package.

Patrick asks: If we decide to do so, what would be good default paths for the host shared folder? /mnt/shared_gw and /mnt/shared_ws?

Patrick said: HulaHoop reported, that "virsh define" will fail if the shared folder in the xml does not exist.

See also: https://github.com/Whonix/Whonix/issues/223

Kernel Samepage Merging (KSM)[edit]

Any changes to the images required? Probably adding this init script:
https://github.com/dnaeon/ksm-init.d-debian (looks good and simple - trivial to package) Any other changes required?

Any changes to documentation or xml required to make use of it?

While not needed to enable this functionality for Whonix directly,its useful for Debian host users. Its inclusion in Whonix will definitely help when running Nested Virtualization though so please add this.

To make use of it, it needs to be enabled host-side following this guide ^[12]. It is used automatically by libvirt without further intervention, but certain parameters such as huge page sharing can be, optionally, used through the XML. ^[13]

Using it on the host is as simple as starting the two services ksm and ksmtuned following the guide above. No need to adjust the number of pages it uses anymore as newer versions automatically handle it with bigger RAM amounts as to give a bigger performance improvement compared to earlier versions that used just 2000 pages of RAM.

How to verify that KSM is working and looking at its stats. ^[14]

Notes: What appeared to be effects of KSM was actually just the Linux Guests not using all allocated RAM by default. Non Linux guests usually take over the entire assigned RAM allocated space while Linux does not. By default KSM kicks in when there is very little free free RAM on the host. Settings must be tweaked in ksmtuned.conf to change this behavior. Experimenting in progress...

KSM security implications?

3.
ATTACK ON MEMORY
DEDUPLICATION
Memory deduplication may be subject to a memory disclosure
attack from the attacker’s VM to the victim’s VM. The attack
guesses a process running on a victim’s VM or a file downloaded
by a browser.
Memory deduplication shares 4KB pages which have the
same contents. When data is written to a deduplicated page, the
page is re-created with a copy of its contents (Copy On Write).
This causes the write access time to be slower than normal,
because it includes the overhead to re-create the same page. An
attacker can measure access time to determine whether a page was
deduplicated by another VM.

^[15]

Enable Hugepages for guests? Yes. I am seeing benchmarks of 40% improvement of performance for all workloads in a guest according to the literature. ~~However using it will prevent use of memory ballooning and swapping ^[16] and is subject to lack of adequate sVirt protections. ^[17]~~ Just confirmed from Dan Walsh, SELinux policy developer that this is no longer the case. The sacrificing of one performance feature for another may not be a problem afterall. Since 2012 there has been an effort to teach KSM how to play nice with THP transparent Hugetables. ^[18]

More info ^[19]

Documentation: A kind soul posted how to interpret what ksmuned.conf's parameters mean in plain English in the first post. Its worth mirroring in our documentation: https://forum.proxmox.com/threads/ksm-with-2-6-35.5041/

No need to play around with the defaults. ksmtuned should handle memory overcommitment adjustments dynamically when the host is under pressure.

More accurate instructions on starting ksm and ksmtuned services and keeping them that way at boot. They will be summarized in the documentation too:

https://archive.ph/jwgVS https://archive.ph/TSWww https://archive.ph/6HBJc https://archive.ph/5EWIv

TO DO: packaging https://github.com/Whonix/Whonix/issues/39#issuecomment-53991872

TO DO: Document on KVM wikipage how users can enable this permanently.

Documentation and risk assessment completed.

Done.

Pass qemu commands via libvirt?[edit]

Not a good idea, has potential security and stability risks. https://listman.redhat.com/archives/libvir-list/2014-September/msg00288.html

Useful in some cases such as guest debugging via qemu's gdb while retainging libvirt's security framework. Relevant documentation added.

memory balloon[edit]

https://rwmj.wordpress.com/2010/07/17/virtio-balloon/

Any changes to documentation, xml or images required to make use of it?

HulaHoop: Just having the device added is enough to take advantage of it. At the moment this feature needs manual intervention to make use of it which limits its utility in the real world. However, towards the end of 2013 work is being done to have this feature automated and mainlined ^[20] This space will be watched for developments; as soon as its incorporated upstream and an option added in libvirt I will enable it in the configuration.

Memory ballooning is indeed another vector for side channel atacks on memory. Confirmed in a follow-up by side-channel attacks researcher Daniel Gruss:

"Not Rowhammer, but there has already been a side-channel attack on memory ballooning earlier this year: https://ieeexplore.ieee.org/document/7562068/"

Status: feature disabled for Whonix 14

Warned upstream and advised them against enabling it by default:

https://listman.redhat.com/archives/libvir-list/2016-December/msg01156.html

SPICE security implications[edit]

https://lists.freedesktop.org/archives/spice-devel/2013-December/015810.html

https://lists.freedesktop.org/archives/spice-devel/2014-January/015815.html

Protections applying to QEMU apply here too.

How to check if virtio storage access is in use or an eventually existing fallback driver?[edit]

Question[edit]

How to check if virtio storage access virtio_blk really is in use or an eventually existing fallback driver?

https://www.linux-kvm.org/page/Boot_from_virtio_block_device
https://wiki.libvirt.org/page/Virtio

Asked on unix.stackexchange[edit]

https://unix.stackexchange.com/questions/148055/kvm-how-to-check-if-virtio-storage-access-is-in-use

lsmod won't help[edit]

The following.

lsmod | grep virtio

Only shows, that the virtio kernel module is loaded. It is not hard to load arbitrary kernel modules for hardware that you don't have installed ("sudo modprobe virtio-blk"). The question remains, is this kernel module actually in use or an eventually existing fallback driver?

modinfo won't help[edit]

modinfo virtio-blk

Won't help, because it only shows general information it would also show on VBox systems (after running "sudo modprobe virtio-blk").

ls Test[edit]

/dev/sda and /dev/sda1 should not exist.

ls -la /dev/sda

ls -la /dev/sda1

/dev/vda and /dev/vda1 should exist.

ls -la /dev/vda

ls -la /dev/vda1

hwinfo Test[edit]

See which driver is loaded for each particular device.

hwinfo --short

How to check if virtio network is really in use or an eventually existing fallback driver?[edit]

How to check, that virtio_pci is really being used and not some fallback driver?

/sys/devices/virtio-pci/0/net/eth0/statistics/rx_bytes seems not to exist on Debian Wheezy.

How to enable AppArmor / SELinux for KVM on Debian?[edit]

Details on how to install, configure and enable SELinux enforcing mode: https://wiki.debian.org/SELinux/Setup#Steps_to_setup_SELinux <-- This is a start, but incomplete. When following these instructions it gets difficult at the "audit2why -al" step, because it shows a ton of policy violations which are non-trivial to fix without learning a lot about SELinux.

Why Bother? SELinux is more robust than Apparmor because its label based vs file-path based. But his comes at the expense of being difficult to write policies. However the plans I have for it in mind, it won't matter since we won't be writing SELinux profiles, but leveraging the one already present in sVirt as a protection layer around application sandboxes.

HulaHoop: This is probably a result of not having done a relabel of your filesystem after enabling SELinux. This aspect can be handled by a script from the link below. I'll need to test this at some point. Any files created before SELinux was prsent creates conflicts with policy when it is enforced. A relabel must also run if you want to isolate software that has a policy but was introduced after SELinux was enabled. Example KVM if we decide to ship it for nested purposes in the future.

https://debian-handbook.info/browse/stable/sect.selinux.html

Another guiede to test with: https://web.archive.org/web/20111211210506/http://blog.danielcosta.pt/?p=440

Note: Both MAC systems cannot be run simultaneously run. This is not supported by LSM and may also be a source of conflicts.

For Debian hosts, AppArmor makes more sense.

https://libvirt.org/drvqemu.html#apparmor-svirt-confinement

AppArmor profiles for Ubuntu could be ported to Debian: https://bazaar.launchpad.net/~ubuntu-branches/ubuntu/utopic/libvirt/utopic/files/head:/debian/apparmor/

DONE. DOCUMENTED.

Set cluster_size 512 - 2MB for better performance?[edit]

HulaHoop wrote: The qcow2 defaults seem fine to me, however the one parameter of interest we should play with is image's : 'cluster_size' 512 - 2MB the larger the size the better performance but the size is bigger. I think we should try the largest and see if the image size stays reasonable.

Patrick wrote:

When using (Whonix 7.7.8.0):

     qemu-img \
        convert \
           -p \
           -O qcow2 \
           -o preallocation=metadata \
           "$WHONIX_BINARY/$VMNAME.img" \
           "$WHONIX_BINARY/$VMNAME-$derivative-maker_whonix_version_new.qcow2"

du -h --apparent-size file.img
100 GB

du -h file.img
2,5 GB

When using (Whonix 7.7.8.0):

     qemu-img \
        convert \
           -p \
           -O qcow2 \
           -o cluster_size=2M \
           -o preallocation=metadata \
           "$WHONIX_BINARY/$VMNAME.img" \
           "$WHONIX_BINARY/$VMNAME-$derivative-maker_whonix_version_new.qcow2"

du -h --apparent-size file.img
2,9 GB

du -h file.img
2,9 GB

File size grew a bit. Not too much. We could use this.

It remains to be tested, if the images work well using these settings. (The apparent size should be shown as 100 GB, but is only 2,9 GB. Needs testing if 100 GB can be used after the image is in use.)

"qemu-img info file.img" however, shows 2,9 GB actual size and 100 GB virtual size.

git tag 7.8.0.1 and above will be using the "-o cluster_size=2M" option.

HulaHoop said: I am yet to document this in the KVM page lets do this at the end. Is there any tangible speed gains?

Exporting/Cloning of virtual machines[edit]

Important links:

Archived thread SE

I'm a bit lost with virt-manager / libvirt / KVM.
I've got a working KVM VM (Windows XP) which works nicely.
The VM is backed by a 4GB file or so (a .img).
Now I want to do something very simple: I want to duplicate my VM.
I thought "OK, no problem, let's copy the 4GB file and copy the XML" file.
But then the libvirt FAQ states in all uppercase: "you SHOULD NOT CARE WHERE THE XML IS STORED"
https://wiki.libvirt.org/page/FAQ
OK fine, I shouldn't care. But then how do I duplicate my VM?
I want to create a new VM that is a copy of that VM.

The most convenient is simply:
virt-clone --connect=qemu://example.com/system -o this-vm -n that-vm --auto-clone
Which will make a copy of this-vm, named that-vm, and takes care of duplicating storage devices. Nothing new here except detail.
More to the point, What the FAQ is saying is that the XML domain descriptions are not directly editable, you need to go through libvirt. To complete the steps taken by the virt-clone command, you could:
You cannot "clone" a running vm, stop it. suspend and destroy are also valid options for less graceful cloning
virsh shutdown this.vm
copy the storage.
cp /var/lib/libvirt/images/{this-vm,that-vm}.img
dump the xml for the original
virsh dumpxml this-vm > /tmp/that-vm.xml
hardware addresses need to be removed, libvirt will assign new addresses automatically
sed -i /uuid/d /tmp/that-vm.xml
sed -i '/mac address/d' /tmp/that-vm.xml
and actually rename the vm: (this also updates the storage path)
sed -i s/this-vm/that-vm /tmp/that-vm.xml
finally, create the new vm
virsh define /tmp/that-vm.xml
virsh start this-vm
virsh start that-vm

# dump the xml for the virtual isolated network
virsh net-dumpxml {{project_name_short}} > /tmp/whonix.xml

# hardware addresses need to be removed, libvirt will assign
# new addresses automatically
sed -i /uuid/d /tmp/whonix.xml
sed -i '/mac address/d' /whonix.xml

management tools[edit]

There are a lot management tools. Are any of these useful for Whonix use case?

Nested Virtualization[edit]

As a minor point, documenting how to use KVM with nested virtualization would be nice.

Forum thread:
https://forums.whonix.org/t/nested-virtulization-needs-64-bit-whonx-guest

This is only possible using a 64 bit kernel. Install a 64 bit kernel. Don't get confused by the package name. Works with AMD and Intel.

sudo apt update
sudo apt install linux-image-amd64

qemu i386 support[edit]

Why are our qemu xml files currently called ${VMNAME}_qemu-x86-x64.xml. Are they dependent on x86-x64? Should they work on i386 systems as well? Maybe just <pae/> must be removed? Can you test please?

Any QEMU architecture works on any host supported by QEMU.i386 tested and working but pointless when x86-64 works on i386 hosts too.

Disable Microphone Input[edit]

Figure out how to disable Microphone input to Guests. While a nice feature for VoIP, is a very dangerous setting to have by default as its unexpected.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/section-libvirt-dom-xml-sound-devices.html

https://libvirt.org/formatdomain.html

Somewhat Solved: After speaking with Cole Robinson, libvirt developer, its clear that this is a feature provided by SPICE rather than something having to do with the sound card. I found that all sound cards have this after testing yesterday. When I changed Spice to VNC this no longer happens. Before you say that we shouldn't use spice, its important to note that audio does not function without it at all.

I've asked on the mailinglists if there is some xml attribute to disable the feature. From what I've seen in the documentation this is not possible. However it can be accomplished through the Host's audio popup menu in the taskbar. The devices show up with the name "virt-manager: record" and can be set to mute as with any other device on the host.

https://libvirt.org/formatdomain.html#graphical-framebuffers

Other spice channel functionality provided:

Valid channel names include main, display, inputs, cursor, playback, record (all since 0.8.6); smartcard (since 0.8.8); and usbredir (since 0.9.12).

Mailinglist reply: https://listman.redhat.com/archives/virt-tools-list/2014-August/msg00002.html Currently not supported through libvirt but planned. Not a problem, we can just document how to disable its easy and gui based. For those running from commandlines, they should just revert to a non SPICE adapter and they'll be fine. They are not missing much because they are not using the workstation as a desktop.

Bugzilla Ticket: https://bugzilla.redhat.com/show_bug.cgi?id=1126641

Document: how to leave KVM when no X is running?[edit]

Situation... User is in terminal in a VM... No X is running ("sudo service kdm stop"). User wants to switch back to the host... How to do this?

The emulated tablet device handles this by not allowing the mouse to be captured by the guest. Its still possible though:

serverfault.com/questions/457603/any-way-to-release-focus-on-a-kvm-guest-in-virt-manager-without-havin-g-to-click

press Ctrl_L & Alt_L

Audio Retasking[edit]

SPEAKE(a)R: Turn Speakers to Microphones for Fun and Profit https://arxiv.org/ftp/arxiv/papers/1611/1611.07350.pdf

conclusion:

While virtual sound devices might be retaskable with software too the malware still cannot access the soundcard settings of the host to retask it too given that the hypervisor is not broken.

Replies from the QEMU devs. They confirm my conclusion that no risk posed to host. Also virtual soundcards optionally have half-duplex modes that make audio input impossible.

https://lists.nongnu.org/archive/html/qemu-devel/2016-11/msg04754.html2 Replies: https://lists.nongnu.org/archive/html/qemu-devel/2016-11/msg04899.html2 https://lists.nongnu.org/archive/html/qemu-devel/2016-11/msg04906.html2 https://lists.nongnu.org/archive/html/qemu-devel/2016-11/msg04904.html2

libguestfs import/export feature request[edit]

Let's together write a feature request for an import/export feature for libguestfs, similar to VirtualBox's import/export feature. Then post in libguestfs's bug tracker.

Irrelevant see virt-manager feature instead.

virt-manager import/export feature request[edit]

Let's together write a feature request for an import/export feature for virt-manager, similar to VirtualBox's import/export feature. Then post in virt-manager's bug tracker.

Update on import/export Support:

Gnome Boxes, another simplified GUI for libvirt will have an OVF import feature added. Note that OVF is not the vmware vmdk specfic format OVA. In this sense making it a true standard.

Its in the works and the Gnome contributor behind it has posted that he plans to make it into a stand-alone library so it could be used by other projects. Meaning it could be easily included in virt-manager too.

His post: https://zee-nix.blogspot.com/2014/05/berlin-dx-hackfest-boxes-rain-and.html

Initial commit: https://web.archive.org/web/20140701164007/https://gitorious.org/govf

Update 2016:

Past efforts have come to a stop. Opened new discussion: https://listman.redhat.com/archives/libvir-list/2016-March/msg00165.html

virt-v2v[edit]

Importing KVM to KVM - Not what its for: https://rwmj.wordpress.com/2015/09/18/importing-kvm-guests-to-ovirt-or-rhev/#content

How to install virt-v2v on Debian?[edit]

virt-v2v is not yet in Debian. wishlist request

Can we copy and paste virt-v2v in meanwhile?[edit]

Since virt2v2 is not yet in Debian, maybe it is a single script that can replace the [https://github.com/{{project_name_short}}/derivative-maker/blob/master/libvirt/whonix_libvirt_import whonix_libvirt_import] script?

HulaHoop: It is my opinion that we shelve the automated script idea completely and use virt-v2v when its possible/available. If virt-v2v does not work out, we should instead do the following: 1) ship multiple templates according to containment systems we want to support 2) A readme file with instructions on [a] how to copy the sparse file properly to any location a user wants if they wish [b] how to introduce the sparse file into priviledged dirs like /var/lib.. [c] how to import xml for machines [d] how to autostart the isolated network [e] how to edit xml files in the case that they want to use non-default paths

The build process would be scripted to update the version number of the name and disk img xml fields and thats it.

The absence of an easy way to import a machine is too big of an issue. We cannot compensate for a shortcoming as big as this coming from upstream. It would take tremendous effort as we have seen and some people might not be happy that it doesn't do X. So I say we leave it as described above.

How to export a VM using virt-v2v?[edit]

This is what probably must be done in the build script. We got qcow2 files and xml. What command must be used to export a VM?

How to export a VM using virt-v2v?[edit]

This is probably what users have to do to import. What command must be used to import a VM?

Closed[edit]

Looks like we've settled on not using virt-v2v for either importing Whonix VMs or for cloning VMs. Probably mainly because v2v is not in Debian and hence doesn't make things simpler.

Shared Folder[edit]

We should add instructions on how to use shared folders. https://www.linux-kvm.org/page/9p_virtio

Done: KVM#Shared_Folders

Script/automate creation and upload of raw images?[edit]

Because raw images have better performance than raw images? https://www.linux-kvm.org/page/Qcow2

True that performance is better, but I don't think it supports snapshots AFAIK. For something that supports snapshots and good performance lets stick with qcow2.

write generic import script[edit]

Maybe write generic import script as per:
https://www.whonix.org/old-forum/index.php/topic,159.msg2412.html#msg2412

A hellish solution and way too much work. Your time is spent better elsewhere. I'll use my time to get xml import validation working.

Script/automate creation of VM description files?[edit]

So not following manual instructions is required anymore, but copy/paste some commands end up with the correct VM configuration.

Once the necessary commands are documented, Patrick will be happy to automate the steps with a script.

HulaHoop created [https://github.com/{{project_name_short}}/derivative-maker/tree/master/libvirt xml files], because kvm xml files can not be created using libvirt as VirtualBox VM setting files can be created using vboxmanage. This is not a problem.

How to add a random time offset?[edit]

Having a small random offset (similar to Advanced_Security_Guide#Network_Time_Synchronization) would be a good thing.

Patrick said: Thanks to bootclockrandomization this is no longer necessary.

xml does not validate[edit]

virsh --version
0.9.12.3

virt-xml-validate whonix_gateway.xml
whonix_gateway.xml:24: element pm: Relax-NG validity error : Element domain has extra content: pm
whonix_gateway.xml fails to validate

virt-xml-validate whonix_workstation.xml
whonix_workstation.xml:24: element pm: Relax-NG validity error : Element domain has extra content: pm
whonix_workstation.xml fails to validate

virt-xml-validate whonix_network.xml
whonix_network.xml:1: element network: Relax-NG validity error : Invalid attribute connections for element network
whonix_network.xml fails to validate

kvmclock had to be under rtc for it to work. The Network xml validates on my machine without changes however. Please check if it works and post back.

The virt-xml-validate I tested with is a newer version. Try updating to a more recent one and try again, I think it may be buggy or not updated enough to interpret new schema they may have added since .9x was released.

Patrick said: HulaHoop said this is only due to different KVM versions and won't cause issues.

Patrick said: Leaving automatic validation out for now. Can still be done manually after editing the xml files.

Run virt-xml-validate during build process[edit]

We should use.

virt-xml-validate libvirt.xml

Patrick will be happy to add this command to the build process when using --kvm (or --qemu?) switch. Scripting this should be simple and the build process could fail early and closed (break).

Patrick said: Since validation is too version specific (see above), we're not doing this.

Weaker warning against KVM in whonixcheck[edit]

As soon as all blockers have been sorted out. We could recommend using KVM instructions.

Done.

Leak Tests?[edit]

HulaHoop sorted that out in [[Special:AWCforum/st/id306/]].

Hardware serials?[edit]

HulaHoop sorted that out in [[Special:AWCforum/st/id306/]].

How to Edit in nano with virsh?[edit]

su

EDITOR=nano virsh edit GuestMachineName

whonixcheck KVMClock[edit]

Whonixcheck should check if /sys/devices/system/clocksource/clocksource0/current_clocksource is kvm-clock and warn if it does.

Implemented: https://github.com/Whonix/derivative-maker/commit/2584c3bdcd9a98181c7535f963a615d257e0fefd

Is it possible to add 64, 128 or even more MB to the virtual graphics card using the command line to boost graphics performance?[edit]

Yes. Can be realised by virsh command as per the steps in here: https://serverfault.com/questions/346301/how-to-increase-video-memory-in-libvirt-kvm-gui

KVM Instructions recommend to use a no cache policy?[edit]

As per blog post it is recommended to use a no cache policy. Do our KVM instructions need changes to incorporate this?

Yes if its recommended then lets do this, Its easily done through the GUI by going to: disk → Cache policy: set to none

Script/automate creation and upload of qcow2 images?[edit]

Done.

Verifiable Builds for qcow2 images[edit]

The https://github.com/Whonix/derivative-maker/blob/master/help-steps/analyze_image requires supports raw, vdi, vmdk, qcow and qcow2 images as well as .ova appliances.

metadata preallocated images[edit]

For better performance. (As per blog post.) Done in 7.7.8.0.

qcow2 compression[edit]

When uncompressed, upload would take ~250 hours per image. Now compressing using tar. [Not using qcow2's compression feature.]

How to disable KVMClock?[edit]

In xml file add: <timer name='kvmclock' present='no'/>

What does KVMClock do? "kvmclock or KVM pvclock lets guests read the host’s wall clock time." This is bad. source

How to adjust KVM to have clock=vm on host.

See https://www.linux-kvm.org/page/KVMClock

https://libvirt.org/formatdomain.html#time-keeping

https://listman.redhat.com/archives/libvir-list/2010-March/msg01293.html

https://web.archive.org/web/20141021123003/https://doc.opensuse.org/products/draft/SLES/SLES-kvm_sd_draft/cha.qemu.running.html#cha.qemu.running.gen_opts.rtc

ls /sys/devices/system/clocksource/clocksource0/current_clocksource

To ensure that you have successfully isolated the vm clock form the host's you must run this in the vm:

sudo chmod -x /etc/init.d/sdwdate

You should verify that kvmclock is indeed disabled by running this command as root.

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

Make sure kvmclock is not listed as a response any longer.

Documented for reference but is irrelevant as of now since this problem is solved.

https://bugzilla.redhat.com/show_bug.cgi?id=783921#c6 < This link provides the solution to disabling kvmclock and verifying that it is so.

whonixcheck (since Whonix 8) would tell you if kvmclock is enabled. whonixcheck --verbose will tell you, that it is not enabled.

Fix subnet range in KVM instructions[edit]

As found out by zweeble in https://www.whonix.org/old-forum/index.php/topic,328.0.html, subnet range must be adjusted to make KVM work.

HulaHoop: I think until we can be sure that this is a problem it shouldn't be considered a blocker

Readme Commands and Warnings[edit]

Until we compile and test a list of virsh commands needed for operating on vm templates and associated disk storage, we should shot release KVM version to the public.

# virsh net-define [path]/whonix_network.xml
Internal network 'Whonix' defined from network xml file residing in the following directory path.
# virsh net-autostart {{project_name_short}} Network 'Whonix' marked to be autostarted upon boot.
# virsh net-start {{project_name_short}} Network 'Whonix' has been started

Leak[edit]

[https://www.whonix.org/old-forum/index.php/topic,243.msg1636.html New FIN ACK / RST ACK Leak Test]
[https://www.whonix.org/old-forum/index.php/topic,243.msg1636.html#msg1636 Reported by HulaHoop]
Was a [https://www.whonix.org/old-forum/index.php/topic,331.msg2269.html#msg2269 false positive] due to a bug in leak testing instructions (now fixed).

Instructions include how to set clock to UTC?[edit]

As per https://libvirt.org/formatdomain.html#time-keeping

The clock setting.

Quote:

The offset attribute takes four possible values, allowing fine grained control over how the guest clock is synchronized to the host. NB, not all hypervisors support all modes.

How to check, that it is set to utc?

It is set to utc by default. A quick look at the xml file will show: <clock offset='utc'>

How to use -rtc clock=vm ?[edit]

KVM's -rtc option as per https://web.archive.org/web/20141021123003/https://doc.opensuse.org/products/draft/SLES/SLES-kvm_sd_draft/cha.qemu.running.html#cha.qemu.running.gen_opts.rtc

This is the original qemu patchset that implements clock=vm in the timer back-ends: https://listman.redhat.com/archives/libvir-list/2010-March/msg01293.html

If you need to isolate the VM Guest clock from the host one, specify clock=vm instead of the default clock=host.

How to use -rtc clock=vm ?

How to check, that KVM -rtc is set to clock=vm ?

Does not seem to be a valid or accepted parameter by libvirt as its always sanitized from the xml file upon saving. But it is accepted by QEMU directly. However I noticed that even with kvmclock enabled, I no longer see any change in a guest's time in response to a host skew of the time. In the recent past this was not true, as any change was immediately reflected in the Guest clock immediately. kvmclock on its own can be disabled and confirmed to be so as outlined below.

What is timer pit?[edit]

What is timer pit?

HulaHoop: Whatever it was, its now gone. I overlooked this by mistake.

We are using.

    <timer name='kvmclock' present='no'/>
    <timer name='hpet' present='no'/>

But why are we using?

    <timer name='pit' tickpolicy='no'/>

Why are we not using?

    <timer name='pit' present='no'/>

virt-image[edit]

Users Expected Scenario[edit]

source

sudo apt install virt-manager

Extract Whonix-Gateway-8.3.qcow2.xz.

Import.

virt-image {{project_name_gateway_short}}-8.3.xml

Start Whonix-Gateway in virt-manager.

xml creation[edit]

How can we create an xml file, that can be imported using virt-image?

Possible help:
https://fedoraproject.org/wiki/Features/ApplianceTools#Release_Engineering

Maybe we need to learn from their fedora-aos.xml to structure Whonix xml.

Since virt-image has been deprecated, this is no option. Source:
https://blog.wikichoon.com/2014/04/deprecating-little-used-tool-virt-image1.html

How to get virtio running for storage?[edit]

Initial Instructions[edit]

Virtio-blk now works

Here are the steps, which I'll throw altogether here before the server downtime. Will move into the wiki later:

https://www.linux-kvm.org/page/Boot_from_virtio_block_device < Original steps that needed to be changed so it can apply to Debian.

Quote

   in guest os, change /boot/grub/device.map from "(hd0) /dev/sda" to "(hd0) /dev/vda"
   in guest os, change /boot/grub/grub.cfg from "root=/dev/sda1" to "root=/dev/vda1", if you are using UUID, then no need to do this step.

I changed all mention of sda1 to vda1 in the grub file. I changed the disk controller type from virtmanager to virtio which takes care of this at the libvirt xml level. Now the vm boots normally with no hangups and I feel a little faster.

A bit of advice, that I'll add so you can decide:

Quote

   <iggy> bancfc: the better option would be using LABELs and UUIDs and such in the source image
   <iggy> unless this is a one time conversion kind of thing

Using LABELs[edit]

Patrick asks: How to use LABELs?

Patrick says: Nevermind. We're now using UUIDs.

Using UUIDs[edit]

Patrick writes: In Whonix 9, fstab will look like this.

cat /etc/fstab

/dev/disk/by-uuid/26ada0c0-1165-4098-884d-aafd2220c2c6 /  auto    defaults,errors=remount-ro 0   1
proc           /proc        proc    defaults                      0   0
/dev/cdrom     /mnt/cdrom0  iso9660 ro,user,noauto                0   0
# some other examples:
# /dev/sda2       none         swap    sw,pri=0             0   0
# /dev/hda1       /Grml        ext3    dev,suid,user,noauto 0  2
# //1.2.3.4/pub   /smb/pub     cifs    user,noauto,uid=grml,gid=grml 0 0
# linux:/pub      /beer        nfs     defaults             0  0
# tmpfs           /tmp         tmpfs   size=300M            0  0
# /dev/sda5       none         swap    sw                   0  0

The relevant part here is.

/dev/disk/by-uuid/26ada0c0-1165-4098-884d-aafd2220c2c6 /  auto    defaults,errors=remount-ro 0   1

The UUID 26ada0c0-1165-4098-884d-aafd2220c2c6 is a fixed one. (Feature by grml-debootstrap.)

grep 26ada0c0-1165-4098-884d-aafd2220c2c6 /usr/sbin/grml-debootstrap
           tune2fs "$TARGET" -U 26ada0c0-1165-4098-884d-aafd2220c2c6

Maybe using UUID and even a fixed one will aid this feature.

HulaHoop: Yes it will. I found very encouraging answers to your inquiries on the forum that I'll post here:

1. activate virtio-blk in a booted image
2. check virtio-blk is really in use (https://www.whonix.org/wiki/Dev/KVM#How_to_check_if_virtio_storage_access_is_in_use_or_an_eventually_existing_fallback_driver.3F)
3. see how we can automate this using {{project_name_short}} build script / {{project_name_short}} packages

1. From my IRC chats with KVM devs, the virtio driverof concern has been included in all distros for a while now and is used by simply having libvirt select it so.

2. lsmod : lists all currently running modules

   modinfo <module name> : information about associated devices
   hwinfo --short : see which driver is loaded for each particular device

3. UUID is the answer from what I recall from different sources inluding KVM documentation. From a post towards the bottom of a thread, someone describes how to configure the kernel to use UUID to boot: forums.gentoo.org/viewtopic-p-6668919.html

/boot/grub/grub.cfg issue[edit]

Patrick comments: I think /boot/grub/grub.cfg is a unstable solution. That file says on top.

# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub

So when there is a kernel upgrade, /boot/grub/grub.cfg will be regenerated and the changes will be lost.

Patrick asks: How do we apply these settings in a way, so they do survive an "sudo update-grub" (which is automatically run on every kernel upgrade).

Patrick says: this is no longer required, since we're now using UUID's in grub.cfg.

System gets unbootable?[edit]

Patrick comments: After editing /boot/grub/grub.cfg, in worst case, maybe, the system will be no longer bootable then. You can test/simulate this case by running "sudo update-grub". Please try.

HulaHoop comments: I found a solution, one that survives kernel and grub upgrades: https://grumpymole.blogspot.com/2007/05/ubuntu-how-to-edit-grub-boot-parameters.html

2.2 Making a change survive upgrades

Depending on how frequently you upgrade your system, you might find that when a new kernel is installed, or grub is upgraded (there may be other scenarios), the /boot/grub/menu.lst file may be replaced by a newer version, or a new line might be added for the new kernel. In this case, your "permanent" changes might be as permanent as you thought.

The best way to address this is to change the "#kopt..." line in /boot/grub/menu.lst. Note that the line is not commented out, even though it starts with a "#". See the excerpt below.

    ### BEGIN AUTOMAGIC KERNELS LIST
    ## lines between the AUTOMAGIC KERNELS LIST markers will be modified
    ## by the debian update-grub script except for the default options below

    ## DO NOT UNCOMMENT THEM, Just edit them to your needs

    ## ## Start Default Options ##
    ## default kernel options
    ## default kernel options for automagic boot options
    ## If you want special options for specific kernels use kopt_x_y_z
    ## where x.y.z is kernel version. Minor versions can be omitted.
    ## e.g. kopt=root=/dev/hda1 ro
    ## kopt_2_6_8=root=/dev/hdc1 ro
    ## kopt_2_6_8_2_686=root=/dev/hdc2 ro
    # kopt=root=UUID=fccafcc7-d7cc-4594-9459-a8f0db7b9f7f ro

Add the parameter changes to the end of the line "#kopt...". These parameters will then be automatically added to any new kernel entries and should also survive an upgrade of grub

Patrick comments: Have you tested this?

Patrick comments: I am not sure editing /boot/grub/grub.cfg is a robust solution. We cannot directly ship a /boot/grub/grub.cfg in a deb, because it is an auto generated file by grub-mkconfig. (When that deb was updated, /boot/grub/grub.cfg would be outdated, system unbootable, you know?) Maybe a package could be created, that appends /boot/grub/grub.cfg, but still, any solution not touching /boot/grub/grub.cfg should be preferred. If we could edit/extend /etc/default/grub, /etc/default/grub.d or /etc/grub.d, that would be much more robust. What is the official suggestion by kvm devs to solve this as a distro?

Patrick says: this is no longer required, since we're now using UUID's in grub.cfg.

Virtio Block Latency[edit]

Not really important but is interesting:
https://www.linux-kvm.org/page/Virtio/Block/Latency

HulaHoop: Yes it could have been the situation that there is latency as the data shows, but I'm sure this was improved by now.

Well what'ya know It is a bug![edit]

HulaHoop wrote:
The reason I struggled so much to get this working is because of an ancient bug that has a workaround in the build process by naming disks to vdX instead of sdX. Please generate a test image with these changes.

https://bugs.launchpad.net/ubuntu/+source/vm-builder/+bug/517067 Please read this comment for the workarounds: https://bugs.launchpad.net/ubuntu/+source/vm-builder/+bug/517067/comments/4

> b.) linux/udev : you could suggest that the vdX devices should be symlinked or other to sdX so that your fstab finds the devices independent of how they're connected. → A suggestion for linux/udev. Not Whonix.

Patrick says: this is no longer required, since we're now using UUID's in grub.cfg.

Solved[edit]

https://www.whonix.org/old-forum/index.php/topic,401.msg3099.html#msg3099

Enable virtio by default?[edit]

Probably yes?

https://wiki.debian.org/DebianKVMGuests

Boot from virtio block device?[edit]

What is https://www.linux-kvm.org/page/Boot_from_virtio_block_device good for? Required?

Disable Microphone Feature Request[edit]

Let's together write a feature request for an option to disable sound. Maybe like this.

Adds only speaker, no microphone.

  <devices>
    <sound model='ich6'>
      <codec type='speaker'/>
    <sound/>
  </devices>

Then post it... Where?

Done:
https://bugzilla.redhat.com/show_bug.cgi?id=1126641

This is finally it. Added in Libvirt 4.4.0. Need to add it when Buster rolls around.

qemu: Add support for 'output' audio codec Support QEMU's 'hda-output' codec advertising only a line-out for ich6 and ich9 sound devices.

Debian Network NAT Issue[edit]

error: Requested operation is not valid: network 'default' is not active

See: https://www.whonix.org/old-forum/index.php/topic,690.msg6076.html#msg6076

Archived Discussions[edit]

[https://www.whonix.org/old-forum/index.php/topic,202.0/topicseen.html qcow2 with metadata preallocation file sizes]

Archived IRC Logs[edit]

* Now talking on #kvm
* Topic for #kvm is: KVM wiki website is https://www.linux-kvm.org/page/Main_Page - try status and faq pages, no really, read them! || you can also check the qemu manual at https://www.qemu.org/docs/master/ || don't paste on chan, use https://privatebin.net/ instead || libvirt/virt-manager support is #virt on irc.oftc.net
* Topic for #kvm set by aliguori at Wed Mar 30 13:38:24 2011
<bancfc> Hi How do I check if virtio storage is in use or an existing fallback driver for an emulated device?
<iggy> there's no fallback in kvm like in xen
<bancfc> so a simple grep should verify if a virtio device is used then - not merely attached I mean?
<iggy> well... if you really want to check, it is going to be /dev/vdX
<iggy> and lspci will show virtio block device
<bancfc> Thanks iggy I have a few more questions that I'd really appreciate your help with
<iggy> ask away
<bancfc> I am trying to isolate the vm clock so no changes on the host can affect it. This is currently possible with -rtc clock=vm afaik,
<bancfc> but it would be much more helpful for my project's purpose if this can be done through something like libvirt
<bancfc> is this possible with libvirt?
<iggy> that I don't know
<bancfc> the appealing part of using libvirt is the xml settings file it creates, but I think something similar can be done with qemu-kvm cfgfiles option
* iggy hates xml
<bancfc> But that means all settings would have to be imitated but directly for the qemu-kvm utility which is fine if I can get it to do the same things
<bancfc> namely the functionality of isoalted internal networking between two vms
<bancfc> is that possible with qemu-kvm directly?
<iggy> libvirt has an option to pass options directly to qemu-kvm
<bancfc> please tell me how this is done :-)
<bancfc> or where I can know this.
<bancfc> ok nevermind, I just want to know if this means that a setting in the xml file is passed directly to qemu-kvm then?
<iggy> ... You might try asking in the libvirt channel
<bancfc> I did but no one seems to reply...
<iggy> I know it is possible, I don't remember it off the top of my head
<bancfc> Thank you so far for your replies. I am actually working on a project that uses virtualization to route all of a vm's traffic through a TOR instance running on another network facing gateway vm.
<bancfc> Its known as {{project_name_short}} <bancfc> Just letting you know that I'd appreciate any help on getting the main issues with porting it to KVM solved
<bancfc> so we can prepare this for a greater good.
<bancfc> This is the bugtracker page I am working on resolving: https://www.whonix.org/wiki/Dev/KVM#How_to_get_virtio_running_for_storage.3F
<bancfc> iggy do you mind if I post the contents of this thread on our forum?
<iggy> no...

<ancfc> Is there any security drawbacks from using a raw image file?
<iggy> bancfc: security drawbacks? like what?
<bancfc> iggy: are its contents directly interpreted by the host or is that when lvm is used only?
<iggy> I'm not sure what you mean by "contents directly interpreted", but raw devices/files just means there's a 1:1 correlation between guest request and host request
<bancfc> like a virus or advanced persistent threat running in the vm
<iggy> the host doesn't "interpret" what it is reading/writing... it just becomes a read/write in the host
<bancfc> ok good just checking. iAnother question is raw snapshotting possible albeit not directly?
<bancfc> I saw something about a differencing image applied in qcow format, yes?
<iggy> only if there's something underlying the files that supports it (btrfs, lvm, zfs, etc)
<bancfc> so not at the "block" level I guess?
<bancfc> iggy: just a question or two about storage if thats ok, becuase we are trying to see which can give maximum performance
<iggy> lvm > raw files > qcow2 > everything else
<bancfc> Is qcow2 the standard? Is qed recommended instead? When is the qcow"3" changes included into the present format?
<bancfc> didn't mean to send this as you had already responded
<bancfc> but anyhow qed seems to have been designed as a speedier replacement for qcow2. But qcow2 were adding changes to become more competitive again
<bancfc> I wanted to know what you think about this and thats all. thanks
<iggy> bancfc: qed is deprecated (as all of it is performance improvements that actually ended up helping were backported to qcow2)
<iggy> the features are handled by feature flags in the qcow2 format, so there's technically no need for a "qcow3" format
<iggy> as long as you don't use the features that qed didn't support (i.e. snapshots, etc.), qcow2 should be as fast
<bancfc> iggy: thanks alot for your help
<iggy> np

* Now talking on #tor
* Topic for #tor is: Welcome to #tor, the discussion channel for Tor users and operators | https://www.torproject.org/ | https://www.torproject.org/docs/faq | https://tor.stackexchange.com/ | Developer discussion on #tor-dev | It is 'Tor', not 'TOR' | non-tor talk => #nottor
* Topic for #tor set by ChanServ!services@services.oftc.net at Sun Feb  2 13:56:09 2014
* Riastradh (~riastradh@82JAAD1W2.tor-irc.dnsbl.oftc.net) has joined #tor
<expedient> I am working on the {{project_name_short}} KVM port and ran the leaktest for this bug reported here: https://lists.torproject.org/pipermail/tor-talk/2014-March/032503.html
<expedient> It turns out that this issue applies to us too and its an upstream bug
<arma> i believe you
<arma> upstream meaning what -- the kernel? iptables?
<expedient> I am thinking iptables
<expedient> You guys can probably get to them easier than we can
<arma> is there a patch?
* drtor (~drtor@9YYAAK2T4.tor-irc.dnsbl.oftc.net) has joined #tor
<expedient> We are looking at some workaround in iptables but are not sure how well this will work
<expedient> but one thing for sure is it needs to be fixed upstream because it affects any Linux platform that uses Tor as a transproxy
<expedient> AFAIK Patrick  -Whoix leader- doesn't write kernel level code
<expedient> https://www.whonix.org/old-forum/index.php/topic,243.0.html
* realitygaps has quit (Remote host closed the connection)
<expedient> According to my findings, the leak can be seen in tcpdump running on the host, as direct communication to Google without going through any Tor node. Its a catastrophe by all means
<velope> a kernel patch is not necessarily appropriate, and upstream might well not consider it a bug
<velope> the iptables workaround i gave is in a followup post from mike in that thread
<velope> it would appear that "it needs to be fixed upstream" is just false
<velope> i only skimmed that forum page quickly, and i really don't want to wade through every bit of it
<velope> it contains my workaround based on --state INVALID
<expedient> That's fine we will apply your advice, but are you sure that there is nothing fundamentally wrong with the kernel or iptables though?
<velope> but it doesn't clearly seem to have been used
<velope> apply it and test, a lot
<velope> there is not yet any solid case made that there is something "fundamentally wrong"
<velope> using iptables to redirect for transproxying relies on connection tracking
<arma> i think the conclusion is "it is hard to use these things safely"
<arma> which is not really a bug in upstream so much as a challenge with the approach
<velope> you have to test
<velope> people don't test thoroughly
<velope> they just read a man page or get some cookbook approach from some post, and run with it
<velope> that is why something like this goes unnoticed for so long
<velope> (gee, sounds like some other notable problems ...)
<expedient> Sorry to throw another link your way, but so far this is what we do when testing:
<expedient> https://www.whonix.org/wiki/Dev/Leak_Tests#FIN_ACK_.2F_RST_ACK_-_Leak_Test
<expedient> Any advice if this is enough or anything we should add to ensure its safe after a workaround?
* quiliro has quit (Quit: Leaving.)
<velope> do violent tests with your networking connection
<expedient> You are very sharp btw. That bug was a great find
<velope> interrupt it and restore it at various moments, for various periods
* blumenkraft (~eristisk@a1.networktest.34sp.com) has joined #tor
<velope> well actually i never announced or reported the problem, so even though i fixed the problem for myself a couple years ago, mike appropriately gets credit for finding it. i just get credit for a better workaround.
* amiller has quit (Remote host closed the connection)
<velope> y'all need to understand that connection tracking was never intended to be more than a loose fit around categories of traffic, not some leakproof box
<expedient> do you know if this applies to nftables too?
<expedient> Its gradually becoming the successor to iptables and was added in 3.13
<velope> when the kernel's info on a connection is lost or dropped (timeout), it is gone forever, so subsequent traffic won't be redirected
<velope> i haven't used it and don't know. unless there's been some massively radical re-architecting of connection tracking, it is unlikely to be totally different, though there could be minor differences in behavior.
<arma> the tails approach is starting to look a bit smarter
* eristisk has quit (Ping timeout: 480 seconds)
<expedient> tails is robus but it has to secure an entire rich os stack to protect the user
<arma> approach to torifying the traffic i mean
<expedient> ok
<arma> i don't argue that wrapping stuff in vm's can make it harder for an attacker to break out
<velope> in that tails doesn't rely on transproxy redirection, yes it is better
<expedient> what specifics are you talking about for torifying maybe we can replicate that?
<arma> but torifying outgoing streams is harder to make safe than explicit proxy settings
* darkclown (~darkclown@p2011-ipbf7306marunouchi.tokyo.ocn.ne.jp) has joined #tor
* anong0 has quit (Ping timeout: 480 seconds)
<velope> right, we've always warned that transproxying is not as great as it seems
<arma> "set your proxy settings to use tor, only allow tor to use the network"
* Gentlecat has quit (Ping timeout: 480 seconds)
<arma> so, implicitly, "don't allow anything that is not configured to use tor to use the network"
<velope> and most people just ignore the warnings about transproxying, because "just stuffing everything through tor" seems just what they want, and so simple. but not really.
* mdik is now known as Guest6988
* mdik (~mdik@brln-4d0c41d4.pool.mediaWays.net) has joined #tor
<expedient>  the problem is when a box is rooted the software that is configured to use Tor can do things it shouldn't do
<expedient> but I get your point
* Guest6988 has quit (Ping timeout: 480 seconds)
<expedient> its those people who think stuffing flash through a transproxy pipe think their safe
<velope> that problem only exists for you because you insist on running tor in isolation
* huseby has quit (Ping timeout: 480 seconds)
<arma> "be sure to send unexpected traffic over tor so it can't hurt us"
<arma> vs "be sure to drop unexpected traffic so it can't hurt us"
<arma> option b sure seems wiser
<arma> then a bad guy who breaks in can, at worst, generate traffic that looks expected so it goes through tor
<sd> Useability wise, just having unique circuit per PID might be reasonable compromise.
* TheCthulhu1 has quit (Remote host closed the connection)
<arma> i think that's a totally separate topic
<expedient> arma:  an important goal we are trying to achieve is for operators of onion services to be able to have a fighting chance in case they are targeted.
<sd> Trying to educate users "don't throw * at tor, but selectively firewall everything" seems like a losing battle though.
<arma> it does? screw those users then.
<arma> better to write software that works well
<arma> and the ones who want to use the safe software will use it
<velope> right, it can only be won or lost if you consider it to be a battle. users screw themselves, every day
<arma> and we can't do much for the ones who choose to do it wrong
<velope> a better analogy might be leading a horse to water
* Rym has quit (Ping timeout: 480 seconds)
<sd> That sounds like somewhat elitist stance. "You dont deserve privacy because you're dumb" vs "Safe defaults built in."
<velope> obviously better defaults are better, if they actually exist and are actually safer
<sd> (I can see how it can go downhill from there if users are babysitted too much, though)
<velope> but defaults never cover everything, because software is flexible, and people want it to be
<expedient> The less decisions that need to be made in security, the better everyone will be
<expedient> and hence why transproxy is an idea worth working towards
<arma> so give them only the applications that you know behave well, with those applications configured to use tor correctly
<velope> that principle is correct but the conclusion about transproxying is wrong
<arma> letting them pick their own apps and hope they are safe is a recipe for surprises again and again
<expedient> we do, but we can't guarantee that nothing dangerous happens when these apps are compromised unless it *all* goes through Tor
<sd> Ultimately the problem with transproxy is just that it is difficult to identify the source app.
* sd uses assigned source port ranges, but it is far from ideal
<expedient> Kgpg, TorBrowser and xchat come by default
<velope> even if it all goes through tor, you can't guarantee that nothing dangerous happens, you only have considered the first layer of potential problems
<expedient> updated system and AppArmor are other layers we have
<velope> that's all very nice and good
<expedient> but against a well funded adversary you must throuw everything in the toolbox and the toolbox itself to protect users
<velope> actually it is just more and more desperate band-aiding
<velope> the effort would be much better directed towards thoroughly auditing apps
<expedient> Auditing firefox is out of our league
<expedient> but we have verifiable builds for {{project_name_short}} <sd> Velope, that goal is far from realistic when we're talking huge blobs of code such as web browser or libpurple.
<velope> it is the true longterm challenge that people get scared away from all the time
* mttp_ (~mttp@9YYAAK2WG.tor-irc.dnsbl.oftc.net) has joined #tor
<velope> and the "community" pays for it dearly with things like heartbleed

Footnotes[edit]

Whonix

The most watertight privacy operating system in the world.

Dark mode

Supported by Power Up Privacy

Whonix is proudly supported until 2025 by Power Up Privacy, a privacy advocacy group that seeks to supercharge privacy projects with resources so they can complete their mission of making our world a better place. (Strictly subject to our sponsorship policy.)

At Whonix we value freedom and trust, supporting Open Source and Freedom Software

By using this website, you acknowledge you have read, understood, and agree to be bound by these agreements: Terms of Service, Privacy Policy, Cookie Policy, E-Sign Consent, DMCA, Imprint

2012- 2024 ENCRYPTED SUPPORT LP

Your support makes
all the difference!

We believe security software like Whonix needs to remain open source and independent. Would you help sustain and grow the project? Learn more about our 12 year success story and maybe DONATE!

[1] ttps://stackoverflow.com/questions/26267173/converting-qemu-command-line-args-to-libvirt-domain-xml#28588704

[2] ttps://github.com/tpm2-software/tpm2-pkcs11/releases/tag/1.7.0-rc0

[3] ttps://listman.redhat.com/archives/libvirt-users/2019-October/msg00008.html

[4] ttps://bugs.debian.org/cgi-bin/bugreport.cgi?bug=941939

[5] ttps://www.phoronix.com/news/GnuPG-2.3-Released

[6] ttps://www.phoronix.com/news/VirtIO-FS-Linux-5.4-File-System

[7] ttps://blog.vmsplice.net/2020/04/virtio-fs-has-landed-in-qemu-50.html

[8] ttps://bugzilla.redhat.com/show_bug.cgi?id=1659484

[9] ttps://www.phoronix.com/news/VirtIO-GPU-Venus-Vulkan

[10] ttps://unix.stackexchange.com/questions/13771/dd-if-dev-random-is-randomly-bottlenecked-with-big-time-lags-but-i-have-no-i

[11] ttps://listman.redhat.com/archives/libvir-list/2014-August/msg01010.html

[12] ttps://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/virtualization_administration_guide/chap-ksm

[13] ttps://libvirt.org/formatdomain.html#memory-backing

[14] ttps://www.suse.com/documentation/sles11/book_kvm/data/kvm_qemu_ksm.html

[15] ttps://web.archive.org/web/20220616034225/https://staff.aist.go.jp/k.suzaki/EuroSec2011-suzaki.pdf

[16] ttps://fedoraproject.org/wiki/Features/KVM_Huge_Page_Backed_Memory#Detailed_Description

[17] ttps://bugzilla.redhat.com/show_bug.cgi?id=515741

[18] ttps://web.archive.org/web/20140429193804/http://www.linux-kvm.org/wiki/images/1/19/2012-forum-memory-mgmt.pdf

[19] ttps://fedoraproject.org/wiki/Features/KVM_Huge_Page_Backed_Memory#Benefit_to_Fedora

[20] ttps://www.linux-kvm.org/page/Projects/auto-ballooning

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]

[20]

Dev/KVM

Links[edit]

Continuous Integration[edit]

Converting QEMU command line args to libvirt domain xml[edit]

Blockers[edit]

Non-Blockers[edit]

Bullseye and Beyond[edit]

Test headless admin on Bullseye[edit]

Document new features and open relevant upstream tickets[edit]

Canokey[edit]

VirtIO Bluetooth[edit]

QEMU LLVM Support[edit]

aarch64[edit]

VirtIO Sound[edit]

Virtio-Video[edit]

Emulated TPM[edit]

VirtIO-FS[edit]

Image Rebasing[edit]

NAT Port Forwarding[edit]

Virgl3D[edit]

Audit Output of virsh domxml-to-native[edit]

Whonix-Gateway[edit]

Whonix-Workstation[edit]

Kicksecure[edit]

Kernel Direct Boot[edit]

Hardware side-channel research[edit]

Host to VM Communication[edit]

In Progress[edit]

Shared Folder[edit]

Done[edit]

QCOW2 Preallocation Settings[edit]

QCOW2 L2 Subcluster Allocation[edit]

Biometric Fingerprinting[edit]

Mouse Tracking[edit]

Solutions[edit]

clocksource[edit]

Upcoming XML Changes - Stable Next[edit]

Apply Clean-filter traffic?[edit]

RNG[edit]

Rate limit[edit]

Haveged[edit]

blkiotune[edit]

Secure Clipboard[edit]

CPU Masking[edit]

Passed to Command Line[edit]

Haveged test for KVM + virtio-rng Result[edit]

Two Separate Whonix-Gateway™[edit]

Preserve Comments in libvirt / virsh XML Files[edit]

Activate Watchdog?[edit]

Add pvspinlock when supported in more recent version of libvirt[edit]

Preconfigured Shared Folders[edit]

Kernel Samepage Merging (KSM)[edit]

Pass qemu commands via libvirt?[edit]

memory balloon[edit]

SPICE security implications[edit]

How to check if virtio storage access is in use or an eventually existing fallback driver?[edit]

Question[edit]

Asked on unix.stackexchange[edit]

lsmod won't help[edit]

modinfo won't help[edit]

ls Test[edit]

hwinfo Test[edit]

How to check if virtio network is really in use or an eventually existing fallback driver?[edit]

How to enable AppArmor / SELinux for KVM on Debian?[edit]

Set cluster_size 512 - 2MB for better performance?[edit]

Exporting/Cloning of virtual machines[edit]

management tools[edit]

Nested Virtualization[edit]

qemu i386 support[edit]

Disable Microphone Input[edit]

Document: how to leave KVM when no X is running?[edit]

Audio Retasking[edit]

libguestfs import/export feature request[edit]

virt-manager import/export feature request[edit]

virt-v2v[edit]

How to install virt-v2v on Debian?[edit]

Can we copy and paste virt-v2v in meanwhile?[edit]

How to export a VM using virt-v2v?[edit]

How to export a VM using virt-v2v?[edit]

Closed[edit]

Two Separate Whonix-Gateway^™[edit]