Administering a Serial-over-LAN (RMCP+) based installation of Debian Jessie entirely through console

The recent addition of four 16GB DDR3 DIMM modules to my VM server (Octeron) has seen its RAM capacity double from 64GB to 128GB thus facilitating the creation of even bigger virtualised environments. Subsequent testing of the new RAM modules with Memtest86 had me once again venture into Octeron's enterprise grade BIOS remotely via the Java powered iKVM. It was here that I stumbled across the Serial-Over-LAN (SOL) functionality which got me considering the possibility of performing a traditional Debian GNU/Linux installation completely through a SOL session.


Unlike the iKVM interface which requires a preexisting Java Runtime Environment (JRE) and a window manager, Serial-Over-LAN can be accessed from any type* of terminal. This flexibility not only the enables the interaction from purely console based TTY terminals (typically found in server environments) but also provides the necessary foundation for access via terminal emulators across various operating systems/platforms (e.g. Terminal Emulator for Android).

Since version 1.5 of Intel's Intelligent Platform Management Interface (IPMI) SOL has been included as part of the core specification (source) and is accessed via the Remote Management Control Protocol (RMCP). The advent of IPMI 2.0 in 2004 ushered in RMCPs successor, "RMCP+", which built upon RMCP's foundation by enhancing various security and authentication mechanisms (details of which can be found here).

Examining SuperMicro's product specification for the enterprise grade H8DCL-iF motherboard used by Octeron reveals the complete implementation of the IPMI 2.0 specification. Unlike my other blog entries, this post is a purely investigatory endeavor which will probably not be too applicable for the vast majority of user environments. Nevertheless it illustrates the remarkable flexibility with today's CLI software utilities for facilitating such an operation.

As usual I have included a TL;DR section at the bottom of this page for those wanting to skip the reasoning of the commands used at each stage.


1. Network connectivity to a host with a SuperMicro motherboard that implements at least the IPMI 1.5 specification (this guide assumes IPMI 2.0 support) for SOL support. Users with active firewalls in their network environment must ensure that UDP port 623 is white-listed between their client machine and the IPMI host.

2. Administrator or Operator credentials (difference explained here) when authenticating with the IPMI host. These elevated privileges are necessary for power cycling the IPMI host in addition to attaching the virtual media (ISO image).

3. Enabling and configuring SOL as appropriate from within the BIOS menu. On Octeron I set the Serial Port Number to COM2* (default ~ see screenshot below) which translates to /dev/ttyS1 within Debian GNU/Linux. Disabling Redirection After BIOS POST was necessary otherwise: 1. The GRUB2 bootloader menu would not be visible and 2. An interactive getty session was not presented upon successful boot (verified by SSH).
Octeron's SOL configuration

4. A Samba server that the IPMI host can communicate with over the network via the SMB protocol (i.e. TCP/445, TCP/139, UDP/137, and UDP/138). The Debian Jessie installer ISO must be stored in an exported/shared directory that can be accessed by a preexisting user with (at least) read only privileges. This guide outlines the basics of setting up a simple, insecure Samba server; it should not be used in production!
Note: During installation the connection between the IPMI host and Samba server must remain intact otherwise the debian-installer will most likely fail.

Packages Used

wget: 1.13.4-3+deb7u2
genisoimage: 9:1.1.11-3
samba: 2:3.6.6-6+deb7u6
ipmitool: 1.8.14-4
curl: 7.38.0-4+deb8u3

1. Preparing the installation environment

To boot a Debian GNU/Linux installation ISO we must first satisfy the prerequisites listed above as well as enabling serial access within the Debian installer itself.
At this stage I am assuming that you have satisfied all prerequisites and are in the console of the client host.

Serial enabled Debian Installer ISO

Unfortunately the latest Debian GNU/Linux "Jessie" 8.4 installer ISO still does not have serial access enabled within its (MBR driven) isolinux bootloader. The default behaviour of the Debian installer at the isolinux bootloader phase is to wait indefinitely for a user input (i.e. selecting an installation option).
This behaviour is problematic as we cannot proceed to select an installation option via the console.

Thankfully I've already encountered and solved this issue before when configuring Debian installer ISOs for my Libvirt VM environments (here). The steps below follow the same vein as my previous workflow but accommodate for the small changes made in the Debian installation ISO since then.

1. Download the latest Debian "Jessie" 8.4 netinst ISO image:
wget --no-check-certificate -O ~/debian-8.4.0-amd64-netinst.iso

2. Setup a loopback device node (assuming /dev/loop0 for the remainder of this section) for mounting the Debian "Jessie" 8.4 netinst ISO. Once mounted we can begin copying its contents and making the necessary alterations to enable serial booting:
sudo losetup --find --show ~/debian-8.4.0-amd64-netinst.iso

3. Create a dedicated mount point for the loopback device node established in the previous step:
sudo mkdir /mnt/debian-8.4.0-amd64-netinst-orig

4. Mount the loopback device node to the dedicated mount point created in the previous step. As is with ISO9660 images the mount will automatically be in a read-only state.
sudo mount -t iso9660 /dev/loop0 /mnt/debian-8.4.0-amd64-netinst-orig

5. Create a dedicated directory for storing the modified contents of the Debian "Jessie" netinst ISO. For simplicity i've selected the current user's home directory:
mkdir ~/debian-8.4.0-amd64-netinst-serial

6. Copy the contents of the Debian "Jessie" netinst ISO to the directory created in the previous step. From this point onwards we can begin making the necessary alterations to enable serial access:
cp -v -R /mnt/debian-8.4.0-amd64-netinst-orig ~/debian-8.4.0-amd64-netinst-serial
Note: We must copy the .disk directory and its respective contents as these are required when regenerating the altered ISO image.

7. Modify the permissions of certain files so as to permit write access to them. These particular configuration files will either be directly edited or overwritten during ISO generation:
chmod 644 ~/debian-8.4.0-amd64-netinst-serial/debian-8.4.0-amd64-netinst-orig/isolinux/{isolinux,txt}.cfg
chmod 644 ~/debian-8.4.0-amd64-netinst-serial/debian-8.4.0-amd64-netinst-orig/isolinux/isolinux.bin

8. With your favourite text editor configure ~/debian-8.4.0-amd64-netinst-serial/debian-8.4.0-amd64-netinst-orig/isolinux/stdmenu.cfg and prepend the following serial console command to the top of the file:
serial 1 115200
Being a derivative of SYSLINUX, ISOLINUX is capable of displaying its bootloader menu over a specified serial port/channel at various baud rates. The configuration we apply at this stage informs ISOLINUX to use serial port/channel 1 (equates to COM2) at a baud rate of 115200 bits per second so as to present an interactive menu via the serial console.
More details about serial console configuration within SYSLINUX/ISOLINUX/PXELINUX can be found here.

9. With your favourite text editor configure ~/debian-8.4.0-amd64-netinst-serial/debian-8.4.0-amd64-netinst-orig/isolinux/txt.cfg and append the required serial configuration arguments to the Linux kernel and initrd pair booted when selecting the default "install" option in the isolinux bootloader menu:
append initrd=/install.amd/initrd.gz --- quiet console=ttyS1,115200n8
Note: Remove the VGA=766 option to ensure that the correct serial port (/dev/ttyS1 == COM2) is used instead. As we have not configured any other boot options (i.e. "Advanced options" boot menu selection) we must use this option when installing Debian "Jessie" 8.4 via SOL.

10. Generate the serial enabled Debian installer ISO image from the edited ISO contents:

genisoimage \  
  -o ~/debian-8.4.0-amd64-netinst-serial-ipmi.iso \
  -r \
  -J \
  -no-emul-boot \
  -boot-load-size 4 \
  -boot-info-table \
  -b isolinux/isolinux.bin \
  -c isolinux/ \

For brevity I've omitted an explanation of each argument/flag passed to the genisoimage binary. For those interested in what these arguments/flags imply you can find out more in my previous post here.

11. As we have now successfully generated a serial enabled Debian "Jessie" netinst ISO we can go about unmounting the loopback device node as well as the ISO editions.
sudo umount /dev/loop0
rm -r ~/debian-8.4.0-amd64-netinst-serial

12. Copy/Move the serial enabled Debian "Jessie" netinst ISO generated in step 10. to your Samba server.

Basic Samba Server Configuration

In my environment I provisioned a simple Samba server from within an unprivileged LXC container on my client machine (Lenovo laptop). I have outlined the few steps necessary for configuring a functional Samba server that provides the serial enabled Debian installer ISO to the IPMI host.
As mentioned previously, the following Samba configuration is not recommended for usage in any form of production environment.

13. Install the Samba server Debian package:
sudo apt-get install samba

14. With your favourite text editor configure the global Samba server configuration file /etc/samba/smb.conf and append the following export/share stanza:

  comment = IPMI Samba share
  path = /path/to/dir/containing/serial_iso
  read only = yes
  guest ok = no

Note: The path variable must point to a directory containing the serial enabled Debian "Jessie" netinst ISO. Moreover the exported/shared directory and the ISO image must have (at least) read-only privileges for (at least) the current user.

15. Add the owner of the Debian installer ISO and exported/shared directory to the internal Samba database. We will use this particular user for authenticating with the Samba server when accessing the ISO image from the IPMI host.
sudo smbpasswd -a username_here
Note: I would recommend not using the same password as that of your current system user as we will need to save the Samba user password in a plaintext file later on!

16. Finally we restart the Samba server to ensure that our configurations are applied:
sudo systemctl restart smbd

2. Installing Debian via SOL

If you have managed to satisfy all four prerequisites and have correctly exported/shared a serial enabled Debian "Jessie" netinst ISO then you have successfully prepared your environment.
With the necessary foundations in place we can now commence the installation of the serial enabled Debian installer ISO completely via SOL.

17. Install the ipmitool userspace utility for communicating with the IPMI host. Beyond providing the SOL shell, ipmitool facilitates "printing FRU information, LAN configuration, sensor readings, and remote chassis power control."
sudo apt-get install ipmitool

18. Download the Bash script (credits: David Wittman). This script employs curl for interacting with the IPMI host's web interface and setting the appropriate CD/ISO text boxes for "mounting" the SMB/CIFS exported/shared Debian ISO.
wget --no-check-certificate -O ~/

19. Alter the permission of the newly downloaded Bash script so as to enable executable permissions:
chmod 744 ~/

20. With your favourite text editor configure the script and edit the IPMI and Samba login credentials as appropriate for your environment:

# IPMI Credentials
# Samba Credentials
[...]--data-urlencode "user=samba_user" --data-urlencode "pwd=samba_user_password"[...]

Note: The source code for the two CGI web script files (virtual_media_share_img.cgi and uisopin.cgi) curl interacts with in the Bash script can be examined in Google Chrome via: Developer Tools -> Sources -> CGI -> [https]://ipmi-ip/cgi/url_redirect.cgi?url_name=vm_cdrom. Ensure you have navigated to the Virtual Media -> CD-ROM Image page.

21. With the correct user credentials for both IPMI and Samba in place we can now proceed to execute the script:
~/./ $IP_IPMI_HOST $IP_SAMBA_SERVER '\ipmi\debian-8.4.0-amd64-netinst-serial-ipmi.iso'

Note: Upon execution the script will perform the AJAX call "virtual_media_share_img.cgi" (located within JS function: SetSharedImageConfig()) which populates the correct input text boxes with the configuration arguments passed to the Bash script. Once completed it proceeds to pause momentarily (sleep 1) and then calls the second and final AJAX call "uisopin.cgi" (located within JS function MountSharedImage()) which, with the Samba configuration provided, attempts to mount the remote Debian installer ISO.

22. With the newly installed ipmitool utility on the client machine login to the IPMI host. If the IPMI host is "up" (i.e. it is running an OS) please power it down as we will be soon proceeding to boot via virtual media over CIFS (Samba).

ipmitool -I lanplus -H $IP_OF_IPMI -U $ADMIN_OR_OPERATOR -P $IPMI_PASSWORD power start && \  
ipmitool -I lanplus -H $IP_OF_IPMI -U $ADMIN_OR_OPERATOR -P $IPMI_PASSWORD sol activate  

Note: The reason I employ the Bash 'AND' operator (&&) is so access is gained to the SOL interface as soon as the IPMI host is powered on. Unless the BIOS boot order has been configured to select the virtual media (i.e. the remote Samba ISO mount) option first we will need to intervene and select the correct boot media option. For brevity I have omitted both an explanation of the flags presented (most are self-explanatory) as well as any advanced authentication/security mechanisms employed by the RMCP+ protocol. For those interested the details of both of these omitted aspects can be found in the man pages of ipmitool.

23. During its initial visual POST phase the SuperMicro motherboard firmware scans through all the available RAM performing what appears to be a basic check in addition to listing various peripherals it has discovered.
While it is checking the available RAM press the F11 (or F3) key from within the SOL shell to enter the "One time boot menu". After the BIOS has finished initialising all its peripherals (e.g. SAS PCI-E card) and onboard NICs you should be presented with a bootable option list:

*      Please select boot device:      *
* USB:IPMI Virtual CDROM               *
* RAID:P0-Samsung SSD 850              *
* USB:IPMI Virtual Disk                *
* Network:IBA GE Slot 0100 v1353       *
*                                      *
*                                      *
*                                      *
*                                      *
*                                      *
*                                      *
*      * and * to move selection       *
*     ENTER to select boot device      *
*      ESC to boot using defaults      *

Select the "USB:IPMI Virtual CDROM" option to begin booting the serial console enabled Debian "Jessie" netinst ISO. Once selected my SuperMicro motherboard gave off two short sharp beeps to indicate it was now using the remote virtual media as its boot media.

24. Shortly after choosing the "USB:IPMI Virtual CDROM" option you should be presented with the following ISOLINUX bootloader menu screen from within your SOL session (dependent upon your network speed from client to IPMI host):

    Debian GNU/Linux installer boot menu  
   Graphical install                     
   Advanced options                    > 
   Install with speech synthesis         


Press ENTER to boot or TAB to edit a menu entry  

As mentioned previously, in this guide we have only configured the Install option to utilise both the correct serial port (for SOL) and baud rate so go ahead and press Enter on the keyboard to commence the installation process.
Warning: Selecting any other installation option (e.g. "Graphical install") will leave you with a blank console where you cannot continue the installation process. If you accidentally enter this state restart your IPMI host and perform steps 22 to 24 carefully again!
Note: The standard "install" option simplifies the installation process by moving you between configuration menus automatically. At some point we will need to escape this behaviour, to do this press the ESC key twice.

25. Follow the menu driven installation procedure configuring the IPMI host as desired for a minimal Debian "Jessie" 8.4 environment.
Upon installing the GRUB2 bootloader ensure that you enter the installation menu by pressing ESC ESC. Once presented with the installation/configuration menu list select the Execute a shell option.

26. Select Continue when the prompt explaining the Ash shell is displayed. The reason for entering a shell session within the installation media is so we can correctly configure both the GRUB2 bootloader and systemd to provide a serial console interface on the correct port/channel (and baud rate) upon subsequent boots.

27. For GRUB2 to regenerate its /boot/grub/grub.cfg configuration file with the necessary serial configurations that will work with the preconfigured SOL session we will need to perform a series of mount operations:
mount --bind /dev /target/bind
mount --types sysfs /target/sysfs
mount --types proc /target/proc

28. Proceed to chroot into the installation path (/target) so we can begin making the necessary modifications in a familiar environment:
TERM=linux chroot /target

29. Take advantage of the root account already configured for the target installation and enter an interactive and fully fledged Bash shell:
su root

GRUB2 configuration

30. With the inbuilt nano text editor modify the GRUB2 configuration file "/etc/default/grub" so that the GRUB2 menu is displayed over the correct serial console port/channel (COM2) and at the correct baud rate:

GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS1,115200n8"  
## GRUB2 menu over serial console
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=1 --word=8 --parity=no --stop=1"  

The GRUB_CMDLINE_LINUX_DEFAULT option appends the serial console configuration onto the end of each normal (i.e. non-recovery) Linux kernel entry in the GRUB2 menu. While not 100% necessary I find it useful to see the kernel output log during boot in case of any boot time issues stopping a login shell. The GRUB_SERIAL_COMMAND and GRUB_TERMINAL options configure GRUB2 itself to use the specified serial port/channel and baud rate when presenting its bootloader menu. As per the documentation: "[...] if you want to use COM2, you must specify ‘--unit=1’ [...]".

31. Regenerate the GRUB2 configuration file /boot/grub/grub.cfg to apply the serial console alterations/additions:

systemd configuration

32. Instruct systemd to present a getty login prompt on /dev/ttyS1 at boot:
systemctl enable [email protected]

33. With the inbuilt nano text editor modify the [email protected] systemd service file for setting the correct baud rate (ExecStart option):

$ cat /etc/systemd/system/\@ttyS1
#  This file is part of systemd.
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

Description=Serial Getty on %I  
Documentation=man:agetty(8) man:systemd-getty-generator(8)  
After=dev-%i.device systemd-user-sessions.service plymouth-quit-wait.service  

# If additional gettys are spawned during boot then we should make
# sure that this is synchronized before, even though
# didn't actually pull it in.  

ExecStart=-/sbin/agetty 115200 %I $TERM  


34. We can now exit the root Bash shell, chroot environment, and ash shell to return back to the Debian installation menu:
exit # 1. Exits root Bash shell
exit # 2. Exits chroot
exit # 3. Exits ash shell

35. Select the "Finish installation" option in the installer menu and allow the IPMI host to reboot.
Note: It is important you select this option and do not perform the reboot from within the ash shell! This final installation step configures the standard user specified earlier in the installation process in addition to other cleanup operations.

36. Perform a final "clean-up" operation by unmounting the loaded Samba exported/shared Debian "Jessie" 8.4 netinst ISO from the BMC. This final step is done on the client machine but not from within the SOL session:

SESSION\_ID=$(curl -d "name=${USER}&pwd=${PASS}" "https://${IPMI\_HOST}/cgi/login.cgi" --silent --insecure -i | awk '/Set-Cookie/ && NR != 2 { print $2 }')  
curl "https://${IPMI\_HOST}/cgi/uisopout.cgi" -H "Cookie: ${SESSION_ID}" --silent --insecure --data ""  

Substituting the ${USER}, ${PASS}, and ${IPMI_HOST} with the username, password, and IP address of the IPMI host respectively.

Upon reboot the SOL shell should (assuming you have not exited the session at this point) behave as follows:

  1. Clear all text/artifacts from the Debian installation environment.
  2. Display various SuperMicro BIOS initialisation screens during bootup.
  3. Present an interactive GRUB2 menu once BIOS hands over control to the bootloader.
  4. Display the boot messages of the specified (for non-recovery kernels) Linux kernel as it initialises various hardware subsystems/peripherals.
  5. Present a getty login shell for providing access to the IPMI host in a similar fashion as a directly connected TTY.


Installation environment setup

# A. Serial enabled Debian Installation media

# Download Debian "Jessie" 8.4 ISO
[email protected]:~$ wget --no-check-certificate -O ~/debian-8.4.0-amd64-netinst.iso

# Mount the Debian "Jessie" 8.4 ISO
[email protected]:~$ sudo losetup --find --show ~/debian-8.4.0-amd64-netinst.iso  
[email protected]:~$ sudo mkdir /mnt/debian-8.4.0-amd64-netinst-orig  
[email protected]:~$ sudo mount -t iso9660 /dev/loop0 /mnt/debian-8.4.0-amd64-netinst-orig

# Copy the contents of the Debian "Jessie" 8.4 ISO for modification
[email protected]:~$ cp -v -R /mnt/debian-8.4.0-amd64-netinst-orig ~/debian-8.4.0-amd64-netinst-serial

# Enable write permissions for certain configuration files 
[email protected]:~$ chmod 644 ~/debian-8.4.0-amd64-netinst-serial/debian-8.4.0-amd64-netinst-orig/isolinux/{isolinux,txt}.cfg</code>  
[email protected]:~$ chmod 644 ~/debian-8.4.0-amd64-netinst-serial/debian-8.4.0-amd64-netinst-orig/isolinux/isolinux.bin

# Append ISOLINUX serial configuration option
[email protected]:~$ vi ~/debian-8.4.0-amd64-netinst-serial/debian-8.4.0-amd64-netinst-orig/isolinux/stdmenu.cfg  
[email protected]:~$ vi ~/debian-8.4.0-amd64-netinst-serial/debian-8.4.0-amd64-netinst-orig/isolinux/txt.cfg

# Generate the serial console enabled Debian installer ISO
[email protected]:~$ genisoimage \  
  -o ~/debian-8.4.0-amd64-netinst-serial-ipmi.iso \
  -r \
  -J \
  -no-emul-boot \
  -boot-load-size 4 \
  -boot-info-table \
  -b isolinux/isolinux.bin \
  -c isolinux/ \

# Unmount loopback ISO and cleanup
sudo umount /dev/loop0  
rm -r ~/debian-8.4.0-amd64-netinst-serial

# B. Samba Server

# Move generated installer ISO to the targeted Samba export directory
[email protected]:~$ mv ~/debian-8.4.0-amd64-netinst-serial-ipmi.iso /path/to/samba/export

# Install the Samba server package 
[email protected]:~$ sudo apt-get install samba

# Append an export entry to the Samba server configuration
[email protected]:~$ vi /etc/samba/smb.conf

# Give a system user a Samba password
[email protected]:~$ sudo smbpasswd -a username_here 

# Restart the Samba server
[email protected]:~$ sudo systemctl restart smbd  

Installing Debian via SOL

# Install the ipmitool utility on the client
[email protected]:~$ sudo apt-get install ipmitool

# Download the SuperMicro Samba mount script from GitHub
[email protected]:~$ wget --no-check-certificate -O ~/

# Edit the SuperMicro Samba mount script to include personal IPMI credentials
[email protected]:~$ vi ~/

# Invoke the SuperMicro Samba mount script with appropriate arguments
~/./ $IP_IPMI_HOST $IP_SAMBA_SERVER '\ipmi\debian-8.4.0-amd64-netinst-serial-ipmi.iso'

# Power on the IPMI host and enter a SOL session as an admin/operator
ipmitool -I lanplus -H $IP\_OF\_IPMI -U $ADMIN\_OR\_OPERATOR -P $IPMI_PASSWORD power start && \  
ipmitool -I lanplus -H $IP\_OF\_IPMI -U $ADMIN\_OR\_OPERATOR -P $IPMI_PASSWORD sol activate  

# Select the "USB:IPMI Virtual CDROM" boot option as the IPMI boots

# Select the "Install" option in the Debian installer menu

# Continue to install Debian "Jessie" 8.4 on the IPMI host as desired

# Select the "Execute a shell" option after installing the GRUB2 bootloader

# Mount the necessary pseudo-file systems for regenerating the GRUB2 config file
~ # mount --bind /dev /target/bind 
~ # mount --types sysfs /target/sysfs  
~ # mount --types proc /target/proc

# Chroot into the /target installation mount point
TERM=linux chroot /target

# Login as root for a correctly configured Bash environment
su root

# Edit the GRUB2 system configuration file
[email protected]_hostname:/# nano /etc/default/grub 

# Regenerate the GRUB2 boot configuration file
[email protected]_hostname:/# update-grub

# Instruct systemd to present a getty login prompt on /dev/ttyS1
[email protected]_hostname:/# systemctl enable [email protected]

# Adjust the baud rate of the systemd se[email protected] file

# Exit the root shell, chroot, ash shell, and return to the Debian installer menu 

# Select the "Finish installation" option in the Debian installer menu
*Finish installation*

# Unmount the Debian installer ISO from the BMC chip (from client)
SESSION_ID=$(curl -d "name=${USER}&pwd=${PASS}" "https://${IPMI_HOST}/cgi/login.cgi" --silent --insecure -i | awk '/Set-Cookie/ && NR != 2 { print $2 }')  
curl "https://${IPMI_HOST}/cgi/uisopout.cgi" -H "Cookie: ${SESSION_ID}" --silent --insecure --data ""  


Coming Soon!

Final Words

Given the flexibility of iKVM in tandem with the ubiquitous nature of Java found in today's computers, I don't believe many users will find this particular investigation effort of mine too useful! Nevertheless it illustrates the capabilities of a purely CLI environment by making it possible to bootstrap a Debian GNU/Linux Operating System on modern IPMI hardware entirely remotely.

An enjoyable aspect of this investigation was the broadening of my own understanding in respect to what particular utilities such as curl are capable of (i.e. interacting with AJAX). The creative process involved when transitioning from having a familiarity/understanding of individual utilities to concocting a plausible "recipe" (i.e. bringing these utilities together in a way you hadn't considered before) is a nice change from the traditional system administration driven focuses of greater efficiency, security, availability, resilience, etc by tweaking various subsystems.