Installing a Custom Operating System on a server using QEMU

CryptoHaz
5 min readSep 12, 2021

Many providers has a great choice of operating systems that you can install on your Virtual or Dedicated server (VPS, VDS or DS), but there are cases when you need a custom, not usual Operating System like Proxmox or maybe a latest distribution of your favorite system. In this situation probably you have two choices:

  • Upgrade a basic distribution (like Debian) to a custom, desired OS — it looks easy but your base system and local storage must be setup correctly before. In some cases you cannot use file systems that are supported by final OS but not by base OS.
  • Install ready to use OS from ISO file — the best choice. You can use all features from your OS (assuming your hardware supports it).

In this tutorial I am going to show you how to do it with the second option (ISO image) and install whatever you want. There is only one requirement — your server must support Rescue Mode.

I assume you are starting with Ubuntu OS, so the commands I will use will be for Debian like distribution.

Checking your current configuration

Before starting Rescue Mode you need to get some information about the current system. The most important thing is the current network configuration. Log into your server and type ip -c link You will receive following information and note down what is the current network interface name (like ens, enp eth etc…)

Next type ip -c addr to get the current IP address in CIDR notation (with Netmask).

You can use this calculator to separate IP and Netmask addresses from CIDR notation.

To known the Gateway address enter ip -c route

You can probably get above information from your server provider Control Panel as well.

Booting in the Rescue Mode

Go to the server Control Panel and activate Rescue Mode. If you can choose a rescue system, select Debian like distribution.

Rescue Mode is a special Linux distribution for recovery purposes that can be booted on demand over the network. It can be used to fix many problems with your server especially when it is not responding.

The Rescue Mode system will be available after a few minutes via SSH under the server IP.

Activating Rescue Mode does not affect your server file system and, moreover requires manual assembly of data partition in order to access server data.

QEMU installation and activation

To install OS from ISO image you need to mount this image in some way. You can do this using QEMU — Quick EMUlator that emulates the machine’s processor through dynamic binary translation. QEMU can run many operating system, mount drives and provide virtualization. It supports KVM (Kernel based Virtual Machine) to take advantage of hardware virtualization with near native processing speed.

After Rescue Mode activation log into your server and run following commands to install QEMU:

apt update
apt install qemu-kvm

Booting custom OS from ISO

In this step we are going to boot from your Operating System ISO image using QEMU. Enter the following command in Rescue Mode:

qemu-system-x86_64 -no-kvm -m 2048 -drive file=/dev/sda,format=raw,media=disk -vnc 127.0.0.1:1 -cdrom <my_iso_path> -boot d

Where:

  • -no-kvm — when your server provider does not allow nested virtualization. You can try -enable-kvm instead but if QEMU exits with an error, try the first one.
  • -m 2048 — use 2 GB RAM for guest system.
  • -drive file=/dev/sda,format=raw,media=disk — mount /dev/sda local partition. You have to specify your drives to mount. You can use commands lsblk or fdisk -l to list your local drives. In the case of multiple drives you have to specify all by multiple -drive ... entries.
  • -vnc 127.0.0.1:1 — start VNC Server.
  • -cdrom <my_iso_path> -boot d — mount the ISO in the CDROM and boot from it.

Installing the system

Open another terminal and setup an ssh tunnel to get graphical access to the installer.

ssh <user>@<server_ip> -L 127.0.0.1:5901:127.0.0.1:5901 -N

Then open your VNC client and connect with 127.0.0.1:5901 At this point you can use ISO installer to install the new Operating System.

In the case of disabled KVM — -no-kvm QEMU parameter, the entire installation process can be really slow and take a long time.

Configuring Network

In the most cases, after the installation is complete, your sever does not have a working network connection. So kill the current QEMU process (the one you use to boot the ISO) and boot installed OS using another QEMU command:

qemu-system-x86_64 -no-kvm -m 2048 -drive file=/dev/sda,format=raw,media=disk -vnc 127.0.0.1:1

Once again open your VNC client and connect with the server.

This is the recommended option as you can also check whether you system boots properly or not. Instead of using QEMU you have two other options:

  1. Use VNC from your server provider after exiting Rescue Mode.
  2. Mount server file system and change the network configuration in Rescue Mode.

On the most Linux distributions you need to edit the/etc/network/interfaces file to configure the network interface.

Use following basic configuration:

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto <network_interface_name>
iface <network_interface_name> inet dhcp

<network_interface_name> you got in one of the previous steps.

Kill QEMU process and type shutdown -r now to exit Rescue Mode and boot your new operating system.

If above does not work, try to specify manually a static IP and use the following configuration:

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto <network_interface_name>
iface <network_interface_name> inet dhcp
iface <network_interface_name> inet static
address <server_ip_address>
netmask <server_netmask_address>
gateway <server_gateway_address>

Conclusions

QEMU is a really powerful tool that you can use to install whatever you want on your server. It can be used for instance with VPS, VDS or DS servers. It can also save your money as you can choose a server provider that does not support custom OS installation or provides it as a paid service.

--

--

CryptoHaz

Entrepreneur, software engineering passionate, humanist