I recently decided to write some tests for one of my Ansible roles that I’ve been making a lot of use of. The plan was to use Ansible Molecule to apply the role and drive the tests. Reading the Molecule docs this seems to be a fairly straight forward task for the majority of cases, essentially you spin up a Docker container, provision your role and run your tests. Unfortunately in this case the role I wanted to test was responsible for managing Docker containers, so there was no easy way to run the role itself within a Docker container without causing some sort of Docker-in-Docker inception!
A bit more digging in to the Molecule docs confirmed that Molecule utilises various “drivers” to provision the underlying platform to which the role is applied before testing. One of these drivers, molecule-vagrant, utilises Vagrant which in turn can manage Virtualbox. Virtualbox can, of course, run a virtual machine which should provide an adequate platform on which to test my Docker role.
Building a Virtual Machine
Looking at the molecule-vagrant docs it become apparent that I would need a virtual machine image in the form of a Vagrant box. It would need (or I would need to be able to provision on to it) a mechanism of SSHing in to the operating system, for Molecule to use to apply and test the role. I had a quick look at some of the public Vagrant boxes but decided that I would rather provision my own box, so I could easily control its specification. For this task I thought I would use Packer.
Prior Packer Art
I found a handful of apparently suitable Packer templates on the interwebs, but they all seemed to suffer from some problem or another and wouldn’t build cleanly. The more I looked at the issues the more I realised they mostly seemed to stem from the fact that Ubuntu changed their automated installer when the released the 20.04 Server image. Many of the packer files seemed to be having a hard time getting things working with the new installer, Subiquity.
Subiquity + Packer Issues
The main issue I came across was that Packer was having what appeared to be SSH authentication issues when building Subiquity configured images. When I debugged this it became clear that, in these cases, Subiquity was installing and configuring the OpenSSH server quite early on in the installation. Critically this was happening before any user configuration was done in the O/S. During the build Packer periodically checks to see if SSH connectivity is available. Once the configured port comes up it tries to authenticate against the host. Because the user configuration wasn’t yet in place Packer exhausts its authentication and fails the build.
This is what Subiquity’s SSH configuration looks like:
ssh:
allow-pw: true
install-server: yes
Once I had realised what was happening it became clear that this could be worked around by using Subiquity’s early and late commands. early-commands run right at the beginning of the installation and late-commands right at the end. I used the following commands to ensure the ssh service was not started until right at the end of the build, when I knew any user configuration would be complete.
early-commands:
# If we install the SSH server using the subiquity `ssh` configuration then port 22 gets opened up to packer _before_
# the requisite configuration has been done to allow Packer to SSH on to the guest O/S. This results in a failed build
# as Packer exceeds its SSH permitted number of SSH handshake attempts.
#
# To ensure this doesn't happen we stop the SSH service until right at the end when we re-enable it
# using a late-command.
- sudo systemctl stop ssh
late-commands:
- sudo systemctl start ssh
This did the trick nicely, giving me a succeeding Packer build.
It’s worth giving a quick shout out to the authors/ contributors of these posts, without which I would have been scratching my head a little longer. Thank you folks! 
- This Hashicorp Discuss thread, which made me realise that Ubuntu has changed to using Subiquity when 20.04 Server was released.
- Efkan Isazade’s post on building 20.04, where he provides a solution to the SSH timing issue.
Share Your Work
I have uploaded my Packer template and its associated files to my packer-images GitHub repository . Please feel free to use/ fork/ tweak it if it’s useful to you. Happy to answer any questions about it either here in the comments (below) or in the repo issue tracker.
Final Thoughts
I think it’s incredible that it is possible to automate the creation and configuration of a virtual machine using entirely open source tooling and information that is freely available on the internet. Kudos to anyone who has been involved in open source tooling that makes these sorts of things possible and to those who share their experiences/ ask questions etc online.
Cheers.
Edd