In this article we will install and configure a Kognitio cluster on to a runningRead More
Making your own Amazon Machine Image
Amazon Web Services (AWS) allows you to rent virtual compute nodes using their Elastic Compute Cloud (EC2). When you launch EC2 nodes you choose an Amazon Machine Image (AMI) to run; this is a pre-installed operating system and perhaps an application, for example “Amazon Linux“, “Microsoft Windows Server 2016“, “OpenVPN Access Server” or “WordPress“. But there might be no AMI available if you need to run a particular operating system so you might want to make your own image.
This blog post describes how to make an Amazon Machine Image from the ISO install image of Linux Mint 18.2 Sonya Cinnamon (64-bit) within AWS that allows remote log in via SSH and RDP. The final versions of the scripts mentioned below are isolinux/isolinux.cfg, preseed/linuxmint.seed and preseed/linuxmint.sh.
The same general method should work for any flavour of Linux or Unix with a reasonable installer. Amazon doesn’t let you interact with the console of your EC2 instances so the installation has to run automatically. An alternative, not described here, would be to create an image externally and import it to AWS.
1. Make a temporary EC2 instance and some volumes
Create the following three volumes. Note that instances made from the AMI will be able to grow their disks from this size, so the target size only needs to be large enough to hold the base install.
- Temporary root file system: 10 GB.
- Temporary ISO disk: 5 GB.
- Target image root filesystem: 10 GB.
Launch a temporary t2.medium instance on EC2 running Amazon Linux with the temporary root filesystem (volume 1) as /dev/xvda and temporary ISO (volume 2) as /dev/xvdf. Log in over ssh as ec2-user and install mkisofs and isohybrid:
sudo bash yum install mkisofs syslinux
2. Extract the files from the ISO image
Download the ISO image and mount it using loopback. Copy the files to another location.
mkdir /mint cd /mint wget http://mirrors.usinternet.com/mint/images/linuxmint.com/stable/18.2/linuxmint-18.2-cinnamon-64bit.iso mkdir isomnt mkdir isonew mount -t iso9660 -o loop linuxmint-18.2-cinnamon-64bit.iso isomnt (cd isomnt ; tar cf - .) | (cd ../isonew ; tar xfp - )
3. Edit your image
Edit your image as required for a completely automatic, unattended install that will:
- Install on /dev/xvdf using the whole disk (or however you want to partition it)
- Install sshd so that you can log in remotely.
Install cloud-init to let AWS set stuff up when an instance is created.
- Make the installer power off or halt at the end of installation (otherwise it will repeatedly do the install forever!)
- Any other steps you want to add at this point, for example from step 8.
4. Create ISO image
Create a new ISO image from the modified files.
mkisofs -o linuxmint-18.2-cinnamon-64bit-amiinstall.iso -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -J -R -V "Linux Mint 18.2 EC2 autoinstall" isonew
5. Write ISO image to disk
Double check using “ls -l” that /dev/xvdf is indeed a device otherwise the following command will just create a file. And make sure you dd to the temporary iso disk and not the temporary root filesystem otherwise you will have to start from scratch!
dd if=linuxmint-18.2-cinnamon-64bit-amiinstall.iso of=/dev/xvdf
6. Try booting
Turn the node off using poweroff or from the EC2 Management Console.
Now detach the volumes from the node. Reattach the temporary ISO disk (volume 2) as the boot disk /dev/xvda, and the image root filesystem (volume 3) as /dev/sdf. Turn the node back on.
You will be able to see progress, or if it has got stuck waiting for input (or been unable to boot at all) from Instance Settings -> Get Instance Screenshot in the EC2 Management Console.
If it fails then power off, detach the disks, re-attach the temporary root filesystem (volume 1) as /dev/xvda and temporary ISO disk (volume 2) as /dev/xvdf; then go back to step 2 above and fix the problem.
7. On success
When you can see from the instance screenshot that the installation has completed, power off and detach the volumes from the node. Then re-attach the image root filesystem (volume 3) as /dev/xvda and power on.
Log in via ssh as installuser using the password given to the installer to perform the following tasks. It is expected that the host’s ssh key will have changed. You can alternatively do some or all of this using your installer script; if you do it all in the installer script – as in my preseed example below – then you can skip steps 7 and 8.
Install packages for AWS cloud support.
sudo apt-get install awscli cloud-init
Linux Mint installs the cloud-init networking scripts in /etc/network/interfaces.d/50-cloud-init.cfg but doesn’t change /etc/network/interfaces to include that, so we need to do that manually. (Otherwise the networking hasn’t been properly set up when it tries to fetch configuration from http://169.254.169.254/ so it gives the error “Failed to establish a new connection: [Errno 101] Network is unreachable”, and the ubuntu user is not given your ssh key: “ci-info: no authorized ssh keys fingerprints found for user ubuntu”.)
sudo sed -e "$ a source /etc/network/interfaces.d/*.cfg" -i /etc/network/interfaces
You are not going to use the console on AWS so turn off X11 to save resources. Note that update-grub is problematic if run in a chroot environment so the preseed options to do this are preferable. You can turn off the virtual console gettys too but for systemd they are generated on demand so I didn’t.)
sudo sed -e 's/GRUB_CMDLINE_LINUX_DEFAULT="quiet"/GRUB_CMDLINE_LINUX_DEFAULT=""/' -i /etc/default/grub sudo sed -e 's/GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="text"/' -i /etc/default/grub sudo update-grub sudo systemctl enable multi-user.target --force sudo systemctl set-default multi-user.target
Also disable the GUI startup so we can see the console output if necessary.
sudo sed -e 's/#GRUB_TERMINAL=console/GRUB_TERMINAL=console/' -i /etc/default/grub
For graphical remote access you probably want to install a recent version of xrdp and xorgxrdp (and/or a VNC server) from source as below, or preferably from a package.
# Remove old version if necessary #sudo apt-get remove xrdp # Packages needed to compile xrdp and xorgxrdp sudo apt-get update sudo apt-get install autoconf automake libtool libssl-dev libx11-dev libxfixes-dev libxrandr-dev nasm xserver-xorg-dev libpam0g-dev # work around packaging bug (see https://github.com/neutrinolabs/xorgxrdp/issues/100) sudo wget -O /usr/include/X11/fonts/fontutil.h https://raw.githubusercontent.com/phracker/MacOSX-SDKs/master/MacOSX10.5.sdk/usr/X11/include/X11/fonts/fontutil.h # Download and build xrdp wget https://github.com/neutrinolabs/xrdp/releases/download/v0.9.3.1/xrdp-0.9.3.1.tar.gz tar xzf xrdp-0.9.3.1.tar.gz cd xrdp-0.9.3.1 ./bootstrap ./configure make sudo make install cd .. # Download and build xorgxrdp wget https://github.com/neutrinolabs/xorgxrdp/releases/download/v0.2.3/xorgxrdp-0.2.3.tar.gz tar xzf xorgxrdp-0.2.3.tar.gz cd xorgxrdp-0.2.3 ./bootstrap ./configure make sudo make install cd .. # Enable services sudo systemctl enable xrdp xrdp-sesman # Remove the build packages if you like sudo apt-get remove autoconf automake libtool libssl-dev libx11-dev libxfixes-dev libxrandr-dev nasm xserver-xorg-dev libpam0g-dev sudo rm /usr/include/X11/fonts/fontutil.h
Tidy up and remove some files so that cloud-init executes “once only” commands on the next boot.
sudo rm -rf /var/tmp/* /tmp/* sudo rm -rf /var/lib/cloud/instances/* sudo rm -f /var/lib/cloud/instance sudo rm -rf /etc/ssh/ssh_host_* sudo rm -f /etc/udev/rules.d/70-persistent-net.rules
Finally, disable (or delete) the user created by the installer, to avoid exposing this account with password login. AWS will set up the cloud-init user, ubuntu (or ec2-user for some other distributions), for login later on so you won’t use the install user again.
sudo usermod --lock --expiredate 1970-01-02 --shell /bin/nologin installuser # sudo userdel -f installuser sudo poweroff
9. Make the Amazon Machine Image
When you are happy, turn off the node. In the EC2 Management Console, go to Volumes, right click on the target image root filesystem (volume 3) and take a snapshot of the disk. Go to Snapshots and when the snapshot creation has finished, right click on it and select “Create Image”. Change the Virtualization type from “Paravirtual” to “Hardware-assisted virtualization” and click “Create”.
Try launching instances from the EC2 Management Console using your new AMI. Put the image volume on /dev/sda1 (not /dev/sda or /dev/xvda this time!) You may want to make the volume larger than the AMI’s volume; if you do this AWS may be able to grow the root partition automatically, or you can do that manually. You can also add a swap partition or file.
If you are happy with the result, delete the three temporary volumes and the temporary EC2 node.
I have described how to do this using the EC2 Management Console but pretty much all of the interaction with AWS could be scripted, for example attaching volumes and starting nodes using the awscli commands attach-volume and start-instances.
The cloud-init networking setup isn’t seen by NetworkManager. This means you can’t see or change things in the Networking control panel app, and Evolution will start offline (unless you use “evolution --force-online“). Workaround by disabling it: “sudo systemctl stop NetworkManager.service“.