Enabling console access within Libvirt (KVM) hosted Debian images

Virtualisation plays a crucial part in my own experimentation and understanding of GNU/Linux based systems. On my Libvirt VM server ("Octeron") I currently use the VM management interface virsh utlity/shell with traditional KVM virtualisation (HVM) as a way of managing/networking/snapshotting my VMs. I consistently use the virt-install utility for creating Libvirt managed VMs.

Intro

The majority of my VM environments I fire up, experiment in, and eventually tear down are Debian Linux based (no surprises there...). When I began using virsh after moving away from direct Qemu/KVM invocations I continued using VNC as it was an easy go-to graphical option that worked out of the box.

The serial console option present in both Qemu/KVM (qemu-system-x86_64) and, consequently virsh, was ultimately preferred for my console orientated experimentations but required further commands/configuration to the corresponding VM.

This guide covers three distinct methods for enabling console access depending upon the situation you find yourself in. Although this guide caters for Debian images, however it may (or may not!) be applicable for other distributions.

Packages Used

libvirt-bin: 0.9.12.3-1+deb7u1
virtinst: 0.600.1-3+deb7u2
parted: 2.3-12
gdisk: 0.8.5-1
fdisk: 0.8.5-1
util-linux: 2.20.1-5.3
genisoimage: 9:1.1.11-2
qemu-utils: 1.1.2+dfsg-6a+deb7u6
(Require kernel config: CONFIG_BLK_DEV_NBD=m)

Console Access

Libvirt uses the XML markup language for storing the configurations of VMs, the underlying hypervisor (see "supports" section here) will determine the directory where the corresponding VM's XML file is stored. The way in which the serial device is presented to both the VM server and the guest can be configured, however for simplicity this guide uses the Libvirt defaults.

Network Installation

virsh provides the functionality to perform a network based boot over a variety of protocols (HTTP, NFS, FTP) by first downloading the kernel (vmlinuz) and initrd pair and then passing them to the VM to boot from.

Below is a full virt-install example for a HTTP network based installation of Debian Wheezy (Netboot image) using only the serial console.

virt-install  
--name test_vm0 \     
--ram 1024 \        
--cpu host \        
--vcpus 2 \            
--location  ftp://ftp.debian.org/debian/dists/wheezy/main/installer-amd64/ \
--disk path=/path/to/installation/disk.qcow2,format=qcow2 \ 
--network=bridge:br1,model=virtio-net-pci \
--graphics none \
--extra-args='console=tty0 console=ttyS0,115200n8 serial'

The --extra-args line is necessary for accessing the network booted VM over a Libvirt compatible serial connection. Once virt-install generates the necessary Libvirt compatible XML the VM will begin booting and you will be automatically connected to the serial console. Serial access is provided between subsequent boots from this point onwards.

Be aware that the the network installation method requires either a locally available Debian repository mirror or an internet connection to access the globally available repositories. Also in this example I have already created the disk.qcow2 using the qemu-img utility.

Final note: From testing myself I found that Debian "Wheezy" will automatically disable (commenting out) the gettys that are normally started on ttys1..6 in /etc/inittab. Make sure to enable the getty for at least tty3 if you planned to access the VM through VNC/Spice in the future.

ISO Disk Install

A Debian Wheezy 7.7 netiso image has been downloaded (http://cdimage.debian.org/debian-cd/7.7.0/amd64/iso-cd/debian-7.7.0-amd64-netinst.iso) and used for installing a VM from scratch.

Sadly the Wheezy netiso image isn't configured to provide serial access upon boot. We will need to configure isolinux.cfg, txt.cfg, and boot.cat then finally recreating the ISO image to enable serial functionality.

This guide is fairly limited in comparison to the graphical method as no initial menu options are presented. Instead we rely on the autoselection of the standard installation option and passing a serial command to that GRUB2 option in order to access it.
(Credits: https://lists.debian.org/debian-user/2011/06/msg02544.html https://wiki.debian.org/DebianInstaller/Modify/CD)

  1. Create the mount point to use for accessing the ISO image:
    sudo mkdir /tmp/wheezy-netiso/

  2. Mount the ISO image:
    sudo mount -o loop,ro /path/to/iso/image/debian-7.7.0-amd64-netinst.iso /tmp/wheezy-netiso/

  3. Due to the read only nature of the ISO9660 format we need to create a clone of the ISO image contents and make the necessary changes:
    cp -R /tmp/wheezy-netiso/ /path/to/ISO/copy/
    We copy the entire directory because we need to include the hidden .disk directory and its contents - these are crucial for a successful ISO alteration and serial console driven installation.

  4. Unless umask rules have automatically adjusted file permissions upon copy you will need to set write permission for certain files we need to edit:
    chmod 644 /path/to/ISO/copy/isolinux/wheezy-netiso/{isolinux,txt}.cfg
    chmod 644 /path/to/ISO/copy/isolinux/wheezy-netiso/isolinux.bin

  5. With your favourite text editor configure /path/to/ISO/copy/wheezy-netiso/isolinux.cfg and alter the timeout value (1/10s units):
    timeout 100 # 10 second timeout
    Note: I'd normally use sed here however the directory containing isolinux.cfg is readonly (again, unless umask has made alterations) and inline (-i flag) edits require a temporary file to be created.

  6. With your favourite text editor configure /path/to/ISO/copy/wheezy-netiso/txt.cfg and remove the VGA command (VGA=766 - QEMU wont work with it when using graphics = none via virt-install) and append the serial console information on the last line as follows:
    append initrd=/install.amd/initrd.gz -- quiet console=ttyS0,115200n8
    Feel free to remove the quiet option as well if you so wished; doing so will "silence" the messages produced by the kernel during its uncompression at startup.

  7. Generate the new ISO image with the serial configuration in place:
    genisoimage -o /path/to/new/ISO/debian-7.7.0-amd64-netinst-serial.iso -r -J -no-emul-boot -boot-load-size 4 -boot-info-table -b isolinux/isolinux.bin -c isolinux/boot.cat /path/to/ISO/copy/wheezy-netiso
    Sheesh that's a lot of flags! For those who are curious there function is explained below:
    "-o": Output ISO location
    "-r": Generate rationalized Rock Ridge directory information ~ adds POSIX file system semantics (see here for more details)
    "-J": Generate Joliet directory information ~ Provides support for Windows-NT or Windows-95 Machines in the rare case you wish to mount them there!
    "-no-emul-boot": Boot image is 'no emulation' image ~ Creating an "El Torito" bootable CD and informing the booting system not to perform any disk emulation
    "-boot-load-size 4": Specifies the number of "virtual" (512-byte) sectors to load in no-emulation mode
    "-boot-info-table": Specifies that a 56-byte table with information of the CD-ROM layout will be patched in at offset 8 in the boot file.
    "-b": Specifies the path and filename of the boot image to be used when making an "El Torito" bootable CD for x86 PCs.
    The pathname must be relative to the source path specified to genisoimage.
    "-c": Specifies the path and filename of the boot catalog (boot.cat), which is required for an "El Torito" bootable CD.
    The pathname must be relative to the source path specified to genisoimage.

  8. Finally we can clean up the modified ISO contents and unmount the original Debian "Wheezy" installer ISO image (debian-7.7.0-amd64-netinst.iso):
    sudo rm -r /path/to/ISO/copy
    sudo umount /tmp/debian-wheezy

Below is a full virt-install example for installing a VM from a locally stored, serial console ready, Debian "Wheezy" net installer based ISO image:

virt-install  
--name test_vm1 \     
--ram 1024 \        
--cpu host \        
--vcpus 2 \            
--cdrom /path/to/new/ISO/debian-7.7.0-amd64-netinst-serial.iso \
--disk path=/path/to/installation/disk.qcow2,format=qcow2 \ 
--network=bridge:br1,model=virtio-net-pci \
--graphics none \
--boot cdrom 

Upon installation you will be dropped straight into console (as was with the net install method). If you removed the quiet flag in option 6. you should see the default installation method kernel output being displayed shortly after the 10 second mark. Finally you should be presented with the initial "Select Language" installation page.

If for whatever reason you missed the 10 second selection you can still enter console (i.e. virsh console test_vm1) and press Enter. This will automatically select the English language but you will be able to go back and change it if needs be.

Preinstalled Disk Image

For this scenario I will consider QCOW2 disk images as the VM's host disk. Another consideration I have taken into account is that multiple partitions are commonly present on Linux systems, in this case I've kept it quite simple: /boot, /home, and /.

Note: The method presented for this scenario isn't as clean as I'd like it to be (recommendations welcome in the comments). From testing I found combining QCOW2 images, chroot, and generated GRUB2 installations resulted in an unworkable grub.cfg. As far as I am aware it is down to the VM's root disk (QCOW2 based) being mounted as a Network Block Device (nbd). Ultimately the boot partition (/boot) is not being presented to GRUB2 in the correct way as to detect the underlying partition table (e.g. GPT/MBR).

First off ensure that the disk image isn't in use by an active VM, you can tell this with the lsof command:
sudo lsof /path/to/installation/disk.qcow2.
I'd also recommend temporarily backing up the VM just in case! Lets begin:

  1. To access the partitions within the QCOW2 disk we must first load the necessary nbd (Network Block Device) kernel module:
    sudo modprobe nbd max_part=3

  2. Assuming /dev/nbd0 is available mount the QCOW2 disk in a similar way we would mount an ISO image to a loopback device:
    sudo qemu-nbd -c /dev/nbd0 /path/to/installation/disk.qcow2

  3. Update the kernel about the partition table layout located on the mounted QCOW2 disk image:
    sudo partprobe /dev/nbd0

  4. List the partitions off the mounted QCOW2 Disk:
    sudo fdisk -l /dev/nbd0 # MBR
    sudo gdisk -l /dev/nbd0 # GPT
    If you have labelled your partitions (and you should!) you can use the blkid to read the label off of each partition.
    E.g. sudo blkid /dev/nbd0pX

  5. Create the mount point directory for the QCOW2 disk's root partition /:
    sudo mkdir /tmp/debian-vm/

  6. Mount the QCOW2 VM host disk's root partition to the newly created mount point directory:
    sudo mount /dev/nbd0p2 /tmp/debian-vm

  7. Mount the QCOW2 VM host disk's boot partition into the step 6.'s mounted root partition /boot directory:
    sudo mount /dev/nbd0p1 /tmp/debian-vm/boot

  8. Chroot into the mounted nbd QCOW2 VM disk:
    sudo chroot /tmp/debian-vm/
    We do this just so we do not accidentally edit host configuration files.

  9. Alter the /etc/inittab to ensure that a getty is listening on ttyS0 for the standard runlevels Debian puts the user in:
    T0:23:respawn:/sbin/getty -L ttyS0 115200 vt102

  10. Configure GRUB2 so it can be accessed over a serial abd that it appends a default serial console kernel argument for all available kernels. This configuration is performed in /etc/default/grub:
    GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200n8 serial"
    GRUB_TERMINAL=serial
    GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"
    Note that the GRUB_SERIAL_COMMAND argument may need to be appended to the file if not present. I have chosen the GRUB_CMDLINE_LINUX_DEFAULT instead of GRUB_CMDLINE_LINUX as the GRUB_CMDLINE_LINUX_DEFAULT only applies our extra serial console arguments to non-recovery kernels. You may decide to alter all boot entries, however I prefer to keep the recovery kernels untouched abiding by the mentality of "if it isn't broke don't fix it".

  11. We only need to alter the /boot/grub/grub.cfg as the QCOW2 disk image already has a bootloader installed. Carefully edit the /boot/grub/grub.cfg file (it may be in read-only mode - use chmod to enable writing) in your favourite editor to get GRUB2 accessible over the serial port ttyS0 and (for now) a single kernel to output its boot messages to the serial port ttyS0:
    ....
    serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
    terminal_input serial
    terminal_output serial
    set timeout=20
    ### END /etc/grub.d/00_header ###
    ...
    linux /vmlinuz-3.2.0-4-amd64 root=UUID=a30f22a5-b4b4-46ee-8bc3-ca39ef972be6 ro text console=tty0 console=ttyS0,115200n8 serial # 1st listed non-recovery kernel

  12. Exit the chroot:
    exit

  13. Unmount the 2 QCOW2 disk partitions:
    sudo umount -v /tmp/debian-vm/boot # Unmount /boot first
    sudo umount -v /tmp/debian-vm/ # Unmount /

  14. Disconnect the QCOW2 VM disk image from the nbd0 block device node:
    sudo qemu-nbd -d /dev/nbd0

  15. Finally we can now start the VM:
    virsh start existing-vm

  16. We now need to access the VM from console via virsh:
    virsh console existing-vm

  17. After accessing the VM and logging in we now need apply the bootup changes to all available non-recovery kernels:
    sudo update-grub

Demo

Configuring the preinstalled disk image is a fairly involved process in comparison to the other two installation options. As a result I've decided to record a terminal session demonstrating the ordering in which to perform commands and the sort of feedback you will receive.