How to install custom Linux on a VPS

All VPS providers have pre-defined Linux templates. However, what if I want to install Linux in a highly customized way? Years ago, I wrote a blog post about installing Linux by cloning, but that was on a physical machine. Can I do it on a Virtual machine? Of course I can, and the process is still largely the same.

Preparing the image

First I build a new Linux system on my local Linux box as a LXC container on a LVM volume (10GB sould be enough). This step has not changed for years; I can use debbootstrap to do that easily. An LXC container only needs the userspace though; to turn it into a full Linux system bootable as a KVM image for a VPS, you would need to install the Linux kernel and Grub (the boot loader) too.

There are many things you can tweak in your containers; I've been tweaking them for years. The image need to be small; so you want to install as few softwares as possible. It is just a LVM volume on your local pc; so I can have many flavors; each tailored to a specific task/application. When you are done, you many want to trim as much as fat as possible:

apt clean
rm -f /var/lib/apt/lists/*_*

Then you can stop the container and use the LVM volume as your Linux template.

Boot the VPS for the first time

My VPS provider allow me to upload ISOs as installation medias. However, I don't want to use the distribution installer; I want to clone my own image. Just like in the case of cloning to a physical machine, I use my trusted System Rescue CD ISO. This ISO can boot the VPS into a fairly complete Linux live system, with all the necessary admin tools. And It comes with a SSH server. Please make sure to append nofirewall to the boot command line to disable the firewall; otherwise the SSH server won't work.

By default, the System Rescue CD boots with network configured with DHCP. It most likely does not work in a VPS, so you need to configure the network manually. System Rescue CD requires you to configure the network through nmcli:

nmcli conn up "Wired connection 1"
nmcli conn mod "Wired connection 1" ipv4.method manual ipv4.addr 192.168.1.20/24 ipv4.gateway 192.168.1.1
nmcli conn up "Wired connection 1"

Quite a mouthful. Please used the static ip assigned by your VPS provider. Then test your network by pinging your gateway:

ping 192.168.1.1

You don't need DNS for this one time SSH server. Then you can add a password to root by running:

passwd

Now, you can ssh in as root with the new password.

clone the image

In my VPS, the virtual disk is called /dev/vda, the virtual network adapter is called ens3. The disk need to be partitioned, just use fdisk and make a simple MBR primary partition; you don't need anything fancier such as GPT partition or LVM. Your VPS shall be dedicated anyway.

Now you can just clone the image through the network from your image host pc:

partclone.ext4 -c -s /dev/lvm/hat | ssh -C root@192.168.1.20 'partclone.ext4 -r -o /dev/vda1'

Please use your own IP and LVM volume name. It will take a while depends on the size of the file system. All the reason to make a compact and customized system image.

Your dev/vda1 partition must is larger than the file system image you are cloning; you can't clone to a smaller device anyway. After the cloning, you want to grow the filesystem to occupy all the space. Please do the following steps on the remote machine afterwards:

fsck.ext4 -f /dev/vda1
resize2fs /dev/vda1

prepare the image for reboot

The image is cloned, you need to tweak it within a chroot before the reboot:

mount /dev/sda1 /mnt
# the following 3 bind mounts is to make the chroot to be more realistic
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
chroot /mnt /bin/bash --login

Now you are in a chroot of your new image! You most likely need to tweak the hostname and network configure at the minimum.

The next step is to install grub on the virtual disk so it can boot. Please do it within the chroot:

grub-install /dev/vda

Please note grub may not be fully functional at this time, further tweaking is often needed after the reboot. Don't be afaid, you have a second chance. Now reboot!

finish grub install

After the reboot, you most likely will face a grub prompt instead of Linux login you were expecting. This is because Grub configuration is not done properly. You can manually configure grub for the first time, then make a grub config after the real linux booting.

linux (hd0,msdos1)/boot/vm...gz root=/dev/vda1
initrd (hd0,msdos1)/boot/initrd...
boot

Linux booted!

Get a root shell, you need to finalize the grub config:

update-initramfs -k all -u
grub-mkconfig -o /boot/grub/grub.cfg
grub-install /dev/vda

All the tweakings may not be necessary, if your image is already fully configured as a KVM image. only the grub-install is strictly necessary because the bootloader lives outside the filesystem image. With the new grub.cfg, grub should know how to boot next time. Reboot to test it out.

Now you have a fully customized Linux on your VPS

The process described in this blog post can also be used for disaster recovery or full system restore. For all my live VPS, I keep exactly the same images locally as Gamma environments so I can rest in peace at night, knowing I can spin up the VPSs with a moment's notice. The software and configuration is only hlf the story; Don't forget to backup your production data and test the backups on the local environment regularly.