VIRTUAL MACHINE

VM - Bind to a VLAN-tagged interface

1. Create the tagged interface(s - here 42 and 44 for example on eno1) on the hosts network configuration /etc/network/interfaces - in this example we’ll only dhcp on the untagged/parent interface (you could dhcp on them all, but this will confuse your dhcp as the same mac is used everywhere):

   allow-hotplug eno1
   iface eno1 inet dhcp
   
   auto eno1.42
   iface eno1.42 inet manual
   
   auto eno1.44
   iface eno1.44 inet manual
   

2. Reconnect your virt-manager and the new virtual interface should be there to bridge into.

VM - Allow multicast packages

Multicast packages are generated by e.g. the avahi daemon or minidlna and is neccessary to use the avahi zeroconf service (needed for media streaming etc).

1. Show all running vms: sudo virsh list
2. Edit the xml file of the machine, which should be allowed to send out these packages: sudo virsh edit X
3. Go down to the network interface which should be allowed (e.g. NOT the [LOCALSTORAGENET]) to do that and change the following code

   <devices>
   ...
   <interface type='XXX'>
   ...
   </interface>
   ...
   </devices>
   

   to

   <devices>
   ...
   <interface type='XXX' trustGuestRxFilters='yes'>
   ...
   </interface>
   ...
   </devices>
   

4. Cold-boot the vm.

VM - Enable TRIM to save image space

This frees the unsuded space inside a vm also on disk. This may accelerates fragmentation, but on zfs it will take quite some time until that happens. The steps 1-2 are only needed for vms created before qemu v4.0 (indicated by the supported machine types (qemu-system-x86_64 -machine help), which should list pc-q35-4.1)! You can use with newer qemu versions the virtio storage directly, as it also then supports discarding (reference).

1. Change disk type to “SCSI” and set discard="unmap"
2. Change the controller type to “VirtIO SCSI” (virtio-scsi)
3. Enable the trim service inside the vm (on older versions of Debian first run sudo cp /usr/share/doc/util-linux/examples/fstrim.{service,timer} /etc/systemd/system):

   sudo systemctl enable fstrim.timer
   sudo systemctl start fstrim.timer
   sudo fstrim -av
   

   The last command just trims the ssd/disks for the first time.

4. Cold-boot the vm.

When the fstrim command exits too fast check with sudo lsblk -o MOUNTPOINT,DISC-MAX,FSTYPE for any 0B entries - in that case that disk does not support TRIM -> you have done something wrong. Also you may want to check with du -h [DISK_IMAGE] the really used space for the images (they should shrink during the first fstrim)…

• More info
• For windows vms

Watchdog & Panic Notifier

Libvirt has a watchdog feature, which can e.g. reboot a vm on crash - other than the “panic notifier” device, which just powers the vm down. How to setup the watchdog:

1. Add the Watchdog device into the vm
2. Inside the vm:

   sudo apt install watchdog
   sudo systemctl enable watchdog
   

3. Enable the device inside the service config /etc/watchdog.conf:

   watchdog-device = /dev/watchdog
   realtime        = yes
   priority        = 1
   

4. Cold-Boot the vm. If you ever wich to test the watchdog, you may crash the kernel with sync; echo c > /proc/sysrq-trigger as root!

VM - Install windows support here

SERVER

Setup

1. sudo apt install libvirt-daemon libvirt-bin qemu-kvm qemu-utils
2. sudo apt install ebtables firewalld dnsmasq

Setup (Debian Jessie+)

The first command should be: sudo apt install libvirt-daemon-system libvirt-clients qemu-kvm qemu-utils

Support guest UEFI

sudo apt install ovmf

Allow a user to control the kvm

1. sudo addgroup [USER] kvm
2. sudo addgroup [USER] libvirt

Firewalld - MAKE SURE TO FIX THAT BUG (if neccessary)

Add/Replace this to /etc/firewalld/firewalld.conf

CleanupOnExit=no

Otherwise a reboot could take up to several minutes!

Useful commands for firewalld

• Configure:
  • sudo firewall-cmd --state
  • firewall-cmd --get-active-zones
• What default zones is active? firewall-cmd --get-default-zone
• What zones is active on…? firewall-cmd --get-zone-of-interface [INTERFACE_NAME]
• List all services which are known: firewall-cmd --get-services
• Add a service for an zone: sudo firewall-cmd --permanent --add-service=[SERVICE_NAME] --zone=[ZONE_NAME]
• Add a port for an zone: sudo firewall-cmd --permanent --add-port=[PORT]/tcp --zone=[ZONE_NAME]
• Disable firewall COMPLETLY for an interface: sudo firewall-cmd --permanent --zone=trusted --change-interface=[INTERFACE_NAME]
• Disable firewall COMPLETLY for ALL interfaces (permanently only): sudo firewall-cmd --set-default-zone=trusted

More info (firewalld)

Enable automatic freezing of guests at host reboot

This are the needed files & scripts:

• Service file vmfreezer.service

  [Unit]
  Description=VMFreezer - saves / restores all running machines of libvirt from / to disk
  Requires=libvirtd.service
  #libvirt-guests.service is in after, because @shutdown this order is inverse!
  #Add here the required path (maybe to save the states on external disks) from /etc/fstab (slash must be a dash)
  #MAYBE add mnt-raid01.mount to wait for a specific mount point...
  After=network.service libvirtd.service libvirt-guests.service
  #Before=
  
  [Service]
  Type=oneshot
  # 10 minutes should be enough; increase only if really needed!
  TimeoutSec=600
  RemainAfterExit=true
  ExecStart=/root/restore.sh
  ExecStop=/root/save.sh
  
  [Install]
  WantedBy=multi-user.target
  

• Restore script to restore.sh

  #!/bin/bash
  # Restore all guests from saved state and start
  
  cd /mnt/
  echo "Working in `pwd`."
  ls -1 *.state | \
  while read GUEST; do
      echo "Restoring $GUEST..."
      virsh restore $GUEST --running
      if [ $? -eq 0 ]; then
          echo "Removing the old state $GUEST..."
          rm $GUEST
      else
          echo "Start of $GUEST failed. The state will be moved to /tmp/ - so it can manually restored... Eventually..."
          mv $GUEST /tmp/
      fi
      # Now sleep a short period of time to make sure, that e.g. dynamic memory has been populated properly...
      sleep 5
  done
  

• Restore script to save.sh

  #!/bin/bash
  # Save (store ram and shutdown) all guests
  
  cd /mnt/
  echo "Working in `pwd`."
  virsh list | `#list of running guest` \
  tail -n +3 | head -n -1 | sed 's/\ \+/\t/g' | `#strip head and tail, use tab for seperator`\
  awk '{print($2)}' | \
  while read GUEST; do
      echo "Saving $GUEST..."
      virsh save $GUEST $GUEST.state
  done
  

Make sure to modify the cd command to fit your wanted save-state-location!

Install the vmfreezer service

1. Add the vmfreezer.service file to /etc/systemd/system
2. Add the save.sh file to /root
3. Add the restore.sh file to /root
4. Set permissons for them sudo chmod 500 /root/save.sh /root/restore.sh
5. ↑ DON’T FORGET to modify the scripts to use the correct path to save and restore the vms! Also this service won’t be able to (re-)store any vm, which does not support saving anyways (e.g. the ones with shared-folders or UEFI)!
6. Enable the new service with sudo systemctl enable vmfreezer

Set static IPs for the VMs

…inside any isolated network, hosted by the host itself - just modify the respective network config with sudo virsh net-edit [LOCALSTORAGENET_NAME] and add:

<network>
...
<dhcp>
...
<host mac='52:54:00:6c:3c:01' name='vm1' ip='192.168.122.11'/>
<host mac='52:54:00:6c:3c:02' name='vm2' ip='192.168.122.12'/>
...
</dhcp>
...
</network>

Thanks Serverfault!

Shared folders

Some performance-notes out of the real world:

• NFS is fast, but is more complex to manage as it is IP authentication based (or you get Kerberos somehow working)
• Samba works, but it is complicated to setup any new user as always a corresponding system user is needed… Also special characters break things…
• 9p just works, but is always slow (small package size -> good directory listing, but bandwith limit by cpu performance; big package size -> painfully slow directoy listing, but good bandwith) - also the caching is funny. Also special characters break things…

NFS with ZFS

Make sure (on both client and server) that NFS ist installed already:

sudo apt install nfs-kernel-server

Also make sure to use for the [SEVRER_IP] a ip, which does not utilizes any bridged interface between guest and host. This won’t work!

Server: Share creation

Enable pool export:

sudo zfs set sharenfs=on [POOL_NAME]

Instead of a simple on you could also pass any NFS option to it - here some examples:

• rw/ro: Set the write mode - append e.g. [email protected] or [email protected]/24 to restrict it to specific clients/networks
• root_squash*/no_root_squash: Should the root uid remapped to an anonymous request? Needed when chown should work…
• all_squash/no_all_squash*: Should every client uid remapped to ↓?
• anonuid & anongid: Set the remapping target uid/gui (defaults to the user nobody) * -> Default!

Client: Mount it!

After exporting the dataset on the server, query the exact name on the clients by using:

showmount --exports [SERVER_IP]

Then you can mount it for testing purposes directly with:

sudo mount -t nfs [SERVER_IP]:[ZFS_EXPORT_PATH] [TARGET_PATH]

You may add it into the /etc/fstab:

[SERVER_IP]:[ZFS_EXPORT_PATH] [TARGET_PATH]  nfs      defaults,_netdev,x-systemd.automount,x-systemd.requires=network-online.target    0       0

You may also append x-systemd.idle-timeout=5m to only mount the share when needed (it gets umounted after the specified time without access). Also here an interesting NFS option:

• soft/hard: When hard NFS will retry the connection forever when it fails (freezes the application triggering it; NFS defaults to hard!)

KVM (9p)

Just add a new mapped shared folder with a new [TARGET_PATH]. To mount it, just insert following line into the guests /etc/fstab:

[TARGET_PATH]    [LOCAL_PATH]       9p      trans=virtio,version=9p2000.L,msize=262144,_netdev    0       0

IF you get emergency boot failures - insert the following into /etc/initramfs-tools/modules:

9p
9pnet
9pnet_virtio

…and update sudo update-initramfs -u!

If the listing of much files is too slow, try enabling the cache (copied from here):

cache=mode	specifies a caching policy.  By default, no caches are used.
        none = default no cache policy, metadata and data
                alike are synchronous.
        loose = no attempts are made at consistency,
                intended for exclusive, read-only mounts
        fscache = use FS-Cache for a persistent, read-only
	            cache backend.
        mmap = minimal cache that is only used for read-write
                mmap.  Northing else is cached, like cache=none

Samba (CIFS)

Install server…

sudo apt install samba

Add a virtual isolated network for loopback communication with the host and vm

• Make sure to enable DHCP, so the host will listen to the clients (instead being REALLY isolated).
• Add this interface (e.g. virbr1) to the firewall (trusted zone is okay - because the VMs should have a second interface anyway which is in the same network like the host)…
• Note that the host can contact the VMs ONLY using that networks IPs from this network!
• Because the host is always faster than the other network interfaces you REALLY SHOULD apply the following fix:
  1. Use the command sudo virsh net-edit [LOCALSTORAGENET_NAME] to open the xml-configuration-file of the virtual network.
  2. Add there the following code (if you add any other entry than the one domain=… the host will resolve the request for the client - so don’t be confused if the /etc/resolv.conf specifies then the host as dns provider)…

     <network>
     ...
     <dns>
     <forwarder domain='router.domain'/>
     <forwarder addr='1.1.1.1'/>
     </dns>
     ...
     </network>
     

     …to forward any request to either the real network dns provider or e.g. Cloudflare!

  3. Save it, restart the network and reboot any vms to apply the fix!

Setup the smb.conf to…

#THIS ALL REQUIRES samba
#This is lacated at /etc/samba/smb.conf

[global]
#Network stuff
workgroup = WORKGROUP
server string = %h
#Following: Set it to the servers local IP (the one from virbr1 / localhost)
#hosts allow = localhost 127.0.0.1 192.168.0.0/24
#hosts deny = 0.0.0.0/0
dns proxy = no
disable netbios = yes
name resolve order = bcast host

#Permissions USE sudo smbpasswd -a USER to add user, USE sudo smbpasswd -x USER to remove user
guest account = nobody
security = user
encrypt passwords = true
invalid users = root
guest ok = no

#Stuff
unix extensions = yes
unix password sync = no
usershare owner only = yes
#Log size in Kb
max log size = 50

#Server role inside the network
server role = standalone server

#Fix the permissions to allow group access!
#force user = [USER (Only if neccessary)]
force group = [FSgroup]
#Following seems to be useless with the following fixes...
#create mask = 770
#FIX permission: File: UPPER bound for the bits
create mode = 770
#FIX permission: File: LOWER bound for the bits
force create mode = 770
#FIX permission: Directory: UPPER bound for the bits
directory mode = 770
#FIX permission: Directory: LOWER bound for the bits
force directory mode = 770

#
#NOTE:
#browseable = no -> Hidden share
#

[Share1]
    path = [PATH]
    available = yes
    #Following to hide it anyways!
    browseable = no
    guest ok = no
    #Following to make read only if no user is in the write list!
    writeable = no
    valid users = [VirtUsers]
    write list = [VirtUsers]

VM - Allow a vm access to a specific share…

Nett2Know: Use sudo pdbedit -L to get current user list…

1. Add an account on the host (nologin, nohome) with sudo adduser --no-create-home --shell /usr/sbin/nologin --disabled-login [USER]
2. Add this account to the FSgroup sudo adduser [USER] [FSgroup]
3. Allow samba to map to this account (now is a good PWD neccessary) sudo smbpasswd -a [USER]
4. Add the account to the shares at the smb.conf
5. Add the share to the vm and save the credentials there (next paragraph)

Setup a vm to access and mount a specific share

Add this to fstab (it will mount on first access - this is neccessary, because some (…) systemd instances ignore the _netdev option) //[HOST_LOCALSTORAGENET_IP]/[SHARE_NAME] [TARGET_PATH] cifs noauto,x-systemd.automount,x-systemd.idle-timeout=5m,_netdev,nouser,mapchars,cache=strict,noacl,credentials=[CREDENTIAL_FILE (e.g. /root/creds)],domain=workgroup,uid=root,gid=[VM_SHARED_FOLDER_GROUP],file_mode=0770,dir_mode=0770 0 0 On cd-failures with error -13 you fucked up the password or username! Use cache=strict to fix ghosting folders (if they still appear use ‘none’ - BUT THIS WILL IMPACT PERFORMACE). When there are no ghosting folders or files you can try to use ‘loose’ to further improve performance.

Setup a vm to make shares available (needed only ONCE)…

1. Install cifs sudo apt install cifs-utils
2. Add the host localstorage interface to /etc/network/interfaces: iface [INTERFACE_NAME] inet dhcp
3. Add a group for the shares sudo addgroup [VM_SHARED_FOLDER_GROUP]
4. Add a user to this group sudo addgroup [USER (e.g. www-data)] [VM_SHARED_FOLDER_GROUP]
5. Create the authentication file (e.g. /root/creds):

   username=[USERNAME]
   password=[PASSWORD]
   

6. Set permissons for the credential file sudo chmod 500 [CREDENTIAL_FILE (e.g. /root/creds)]

CLIENT(S)

Setup management-client

sudo apt install virt-manager spice-client-gtk gir1.2-spiceclientgtk-3.0

Setup viewonly-client

sudo apt install virt-viewer

MORE INFO