Blogs

Making your own Amazon Machine Image

Amazon Web Services (AWS) allows you to rent virtual compute nodes using their Elastic Compute Cloud (EC2). When you launch EC2 nodes you choose an Amazon Machine Image (AMI) to run; this is a pre-installed operating system and perhaps an application, for example “Amazon Linux“, “Microsoft Windows Server 2016“, OpenVPN Access Server” or “WordPress“.  But there might be no AMI available if you need to run a particular operating system so you might want to make your own image.

This blog post describes how to make an Amazon Machine Image from the ISO install image of Linux Mint 18.2 Sonya Cinnamon (64-bit) within AWS that allows remote log in via SSH and RDP. The final versions of the scripts mentioned below are isolinux/isolinux.cfg, preseed/linuxmint.seed and preseed/linuxmint.sh.

The same general method should work for any flavour of Linux or Unix with a reasonable installer.  Amazon doesn’t let you interact with the console of your EC2 instances so the installation has to run automatically. An alternative, not described here, would be to create an image externally and import it to AWS.

1. Make a temporary EC2 instance and some volumes

Create the following three volumes. Note that instances made from the AMI will be able to grow their disks from this size, so the target size only needs to be large enough to hold the base install.

  1. Temporary root file system: 10 GB.
  2. Temporary ISO disk: 5 GB.
  3. Target image root filesystem: 10 GB.

Launch a temporary EC2 instance running Amazon Linux with the temporary root filesystem (volume 1) as /dev/xvda and temporary ISO (volume 2) as /dev/xvdf. Log in over ssh as ec2-user and install mkisofs and isohybrid:

sudo bash
yum install mkisofs syslinux

2. Extract the files from the ISO image

Download the ISO image and mount it using loopback. Copy the files to another location.

mkdir /mint
cd /mint
wget http://mirrors.usinternet.com/mint/images/linuxmint.com/stable/18.2/linuxmint-18.2-cinnamon-64bit.iso
mkdir isomnt
mkdir isonew
mount -t iso9660 -o loop linuxmint-18.2-cinnamon-64bit.iso isomnt
(cd isomnt ; tar cf - .) | (cd ../isonew ; tar xfp - )

3. Edit your image

Edit your image as required for a completely automatic, unattended install that will:

  • Install on /dev/xvdf using the whole disk (or however you want to partition it)
  • Install sshd so that you can log in remotely.
  • Install cloud-init to let AWS set stuff up when an instance is created.

  • Make the installer power off or halt at the end of installation (otherwise it will repeatedly do the install forever!)
  • Any other steps you want to add at this point, for example from step 8.

For Ubuntu or Linux Mint edit the file preseed/linuxmint.seed and isolinux/isolinux.cfg. See below for the preseed file I used.

4. Create ISO image

Create a new ISO image from the modified files.

mkisofs -o linuxmint-18.2-cinnamon-64bit-amiinstall.iso -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -J -R -V "Linux Mint 18.2 EC2 autoinstall" isonew

This ISO image will boot from an optical device but not from a hard disk, because mkisofs has not added the necessary boot sector. To do that, run isohybrid:

isohybrid linuxmint-18.2-cinnamon-64bit-amiinstall.iso

5. Write ISO image to disk

Double check using “ls -l” that /dev/xvdf is indeed a device otherwise the following command will just create a file. And make sure you dd to the temporary iso disk and not the temporary root filesystem otherwise you will have to start from scratch!

dd if=linuxmint-18.2-cinnamon-64bit-amiinstall.iso of=/dev/xvdf

6. Try booting

Turn the node off using poweroff or from the EC2 Management Console.

Now detach the volumes from the node. Reattach the temporary ISO disk (volume 2) as the boot disk /dev/xvda, and the image root filesystem (volume 3) as /dev/sdf. Turn the node back on.

You will be able to see progress, or if it has got stuck waiting for input (or been unable to boot at all) from Instance Settings -> Get Instance Screenshot in the EC2 Management Console.

If it fails then power off, detach the disks, re-attach the temporary root filesystem (volume 1) as /dev/xvda and temporary ISO disk (volume 2) as /dev/xvdf; then go back to step 2 above and fix the problem.

7. On success

When you can see from the instance screenshot that the installation has completed, power off and detach the volumes from the node. Then re-attach the image root filesystem (volume 3) as /dev/xvda and power on.

8. Post-install

Log in via ssh as installuser using the password given to the installer to perform the following tasks. It is expected that the host’s ssh key will have changed. You can alternatively do some or all of this using your installer script; if you do it all in the installer script – as in my preseed example below – then you can skip steps 7 and 8.

Install packages for AWS cloud support.

sudo apt-get install awscli cloud-init

Linux Mint installs the cloud-init networking scripts in /etc/network/interfaces.d/50-cloud-init.cfg but doesn’t change /etc/network/interfaces to include that, so we need to do that manually. (Otherwise the networking hasn’t been properly set up when it tries to fetch configuration from http://169.254.169.254/ so it gives the error “Failed to establish a new connection: [Errno 101] Network is unreachable”, and the ubuntu user is not given your ssh key: “ci-info: no authorized ssh keys fingerprints found for user ubuntu”.)

sudo sed -e "$ a source /etc/network/interfaces.d/*.cfg" -i /etc/network/interfaces

You are not going to use the console on AWS so turn off X11 to save resources. Note that update-grub is problematic if run in a chroot environment so the preseed options to do this are preferable. You can turn off the virtual console gettys too but for systemd they are generated on demand so I didn’t.)

sudo sed -e 's/GRUB_CMDLINE_LINUX_DEFAULT="quiet"/GRUB_CMDLINE_LINUX_DEFAULT=""/' -i /etc/default/grub
sudo sed -e 's/GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="text"/' -i /etc/default/grub
sudo update-grub
sudo systemctl enable multi-user.target --force
sudo systemctl set-default multi-user.target

Also disable the GUI startup so we can see the console output if necessary.

sudo sed -e 's/#GRUB_TERMINAL=console/GRUB_TERMINAL=console/' -i /etc/default/grub

For graphical remote access you probably want to install a recent version of xrdp and xorgxrdp (and/or a VNC server) from source as below, or preferably from a package.

# Remove old version if necessary
#sudo apt-get remove xrdp

# Packages needed to compile xrdp and xorgxrdp
sudo apt-get update
sudo apt-get install autoconf automake libtool libssl-dev libx11-dev libxfixes-dev libxrandr-dev nasm xserver-xorg-dev libpam0g-dev

# work around packaging bug (see https://github.com/neutrinolabs/xorgxrdp/issues/100)
sudo wget -O /usr/include/X11/fonts/fontutil.h https://raw.githubusercontent.com/phracker/MacOSX-SDKs/master/MacOSX10.5.sdk/usr/X11/include/X11/fonts/fontutil.h

# Download and build xrdp
wget https://github.com/neutrinolabs/xrdp/releases/download/v0.9.3.1/xrdp-0.9.3.1.tar.gz
tar xzf xrdp-0.9.3.1.tar.gz
cd xrdp-0.9.3.1
./bootstrap
./configure
make
sudo make install
cd ..

# Download and build xorgxrdp
wget https://github.com/neutrinolabs/xorgxrdp/releases/download/v0.2.3/xorgxrdp-0.2.3.tar.gz
tar xzf xorgxrdp-0.2.3.tar.gz
cd xorgxrdp-0.2.3
./bootstrap
./configure
make
sudo make install
cd ..

# Enable services
sudo systemctl enable xrdp xrdp-sesman

# Remove the build packages if you like
sudo apt-get remove autoconf automake libtool libssl-dev libx11-dev libxfixes-dev libxrandr-dev nasm xserver-xorg-dev libpam0g-dev
sudo rm /usr/include/X11/fonts/fontutil.h

Tidy up and remove some files so that cloud-init executes “once only” commands on the next boot.

sudo rm -rf /var/tmp/* /tmp/*
sudo rm -rf /var/lib/cloud/instances/*
sudo rm -f /var/lib/cloud/instance
sudo rm -rf /etc/ssh/ssh_host_*
sudo rm -f /etc/udev/rules.d/70-persistent-net.rules

Finally, disable (or delete) the user created by the installer, to avoid exposing this account with password login. AWS will set up the cloud-init user, ubuntu (or ec2-user for some other distributions), for login later on so you won’t use the install user again.

sudo usermod --lock --expiredate 1970-01-02 --shell /bin/nologin installuser
# sudo userdel -f installuser
sudo poweroff

9. Make the Amazon Machine Image

When you are happy, turn off the node. In the EC2 Management Console, go to Volumes, right click on the target image root filesystem (volume 3) and take a snapshot of the disk. Go to Snapshots and when the snapshot creation has finished, right click on it and select “Create Image”.

10. Finally…

Try launching instances from the EC2 Management Console using your new AMI. Put the image volume on /dev/sda1 (not /dev/sda or /dev/xvda this time!) You may want to make the volume larger than the AMI’s volume; if you do this AWS may be able to grow the root partition automatically, or you can do that manually. You can also add a swap partition or file.

If you are happy with the result, delete the three temporary volumes and the temporary EC2 node.

I have described how to do this using the EC2 Management Console but pretty much all of the interaction with AWS could be scripted, for example attaching volumes and starting nodes using the awscli commands attach-volume and start-instances.

11. Gotchas

The cloud-init networking setup isn’t seen by NetworkManager. This means you can’t see or change things in the Networking control panel app, and Evolution will start offline (unless you use “evolution --force-online“). Workaround by disabling it: “sudo systemctl stop NetworkManager.service“.

Leave a Reply

Your email address will not be published nor used for any other purpose. Required fields are marked *