Integrating coreos-installer with Ironic

CoreOS: why and how

CoreOS is a minimal operating system for running containerized workloads, developed on top of RHEL or Fedora, integrating technologies like Ignition, rpm-ostree and SELinux. It features an immutable and atomically updated root file system. In OpenShift it is used as the recommended base OS for clusters. For installation, CoreOS can be booted via PXE or as a live ISO. Then the coreos-installer utility can be used to copy the root filesystem to the machine's hard drive. Configuration is done on the first boot using Ignition.

Initially, bare metal installation worked in OpenShift in a similar way to a normal Ironic provisioning: an image (in qcow2 format) was written by the deploy ramdisk to the disk, first boot configuration was handled by cloud-init. This approach suffers from three issues:

1. The CoreOS qcow2 image is designed for OpenStack, not for bare metal.

2. By not using coreos-installer and Ignition, bare metal provisioning diverges from the standard installation procedure in CoreOS.

3. The traditional deploy ramdisk is based on pure RHEL, which adds another operating system to the picture.

The goal of my work was to find a way to integrate the (unmodified upstream) Ironic with (also unmodified) CoreOS. Fortunately, the flexibility of Ironic permits that!

Ironic and custom deploy

Which Ironic features allow non-standard provisioning processes? Two concepts come into play:

Deploy steps

The deployment process is not monolithic: it is split into several steps, and non-default steps can be enabled on request. By default the following steps are run:

1. deploy [4] initializes the deployment.

2. write_image writes the image to the target disk.

3. prepare_instance_boot prepares the bootloader and configures the boot device in the BMC.

4. tear_down_agent shuts down the deploy ramdisk.

5. switch_to_tenant_network reconfigures networking (not used in Metal3).

6. boot_instance reboots the machine into the newly deployed OS.

We're interested in replacing two steps: write_image and prepare_instance_boot with a call to coreos-installer. Here is where a custom hardware manager comes in handy.

Hardware managers

are plugins for ironic-python-agent (IPA) - the main service of the deploy ramdisk. Among many other things, they can provide custom deploy steps.

To help with this use case, I have created a custom-agent deploy interface, which is essentially a skeleton of a deploy interface that starts and stops the ramdisk but does not provide any actual logic. Nor does it require any instance_info values, i.e. image information. This is important because a CoreOS live environment already contains everything for a deployment.

Putting all bits together, we're arriving at the following deployment process:

1. Metal3 configures a node to use custom-agent and to use CoreOS as a ramdisk.

2. Ironic boots the deploy ramdisk, which in this case is a CoreOS live environment, via PXE or virtual media ISO boot.

3. Inspection and cleaning happen normally, using the built-in functionality of ironic-python-agent.

4. For deployment, Metal3 requests a new custom deploy step (which I called install_coreos) to be run between deploy and tear_down_agent.

5. The new deploy step detects the root device using root device hints and calls coreos-installer with it.

6. Once coreos-installer finishes (hopefully with success), the deployment proceeds further to tear_down_agent and other steps.

How will we get ironic-python-agent to run on CoreOS? A custom CoreOS image? Not really. But before answering this question, let us take a look at the new deploy step.

import json
import os
import subprocess

from ironic_python_agent import errors
from ironic_python_agent import hardware
from oslo_log import log
import tenacity

LOG = log.getLogger()

ROOT_MOUNT_PATH = '/mnt/coreos'


class CoreOSInstallHardwareManager(hardware.HardwareManager):

    HARDWARE_MANAGER_NAME = 'CoreOSInstallHardwareManager'
    HARDWARE_MANAGER_VERSION = '1'

    def evaluate_hardware_support(self):
        return hardware.HardwareSupport.SERVICE_PROVIDER

    def get_deploy_steps(self, node, ports):
        return [
            {
                'step': 'install_coreos',
                'priority': 0,
                'interface': 'deploy',
                'reboot_requested': False,
                'argsinfo': {},
            }
        ]

    def install_coreos(self, node, ports):

root = hardware.dispatch_to_managers('get_os_install_device',
                                     permit_refresh=True)

configdrive = node['instance_info'].get('configdrive') or {}
if isinstance(configdrive, str):
    raise errors.DeploymentError(
        "Cannot use a pre-rendered configdrive, please pass it "
        "as JSON data")

ignition = configdrive.get('user_data')

args = ['--preserve-on-error']  # We have cleaning to do this

if ignition:
    LOG.debug('Will use ignition %s', ignition)
    dest = os.path.join(ROOT_MOUNT_PATH, 'tmp', 'ironic.ign')
    with open(dest, 'wt') as fp:
        if isinstance(ignition, str):
            fp.write(ignition)
        else:
            json.dump(ignition, fp)
    args += ['--ignition-file', '/tmp/ironic.ign']

image_url = node['instance_info'].get('image_source')
if image_url:
    args += ['--image-url', image_url, '--insecure']
else:
    args += ['--offline']

command = ['chroot', ROOT_MOUNT_PATH,
           'coreos-installer', 'install', *args, root]
LOG.info('Executing CoreOS installer: %s', command)
try:
    self._run_install(command)
except subprocess.CalledProcessError as exc:
    raise errors.DeploymentError(
        f"coreos-install returned error code {exc.returncode}")
LOG.info('Successfully installed via CoreOS installer on device %s',
         root)

@tenacity.retry(
    retry=tenacity.retry_if_exception_type(subprocess.CalledProcessError),
    stop=tenacity.stop_after_attempt(3),
    reraise=True)
def _run_install(self, command):
    try:
        # NOTE(dtantsur): avoid utils.execute because it swallows output
        subprocess.run(command, check=True)
    except FileNotFoundError:
        raise errors.DeploymentError(
            "Cannot run coreos-installer, is it installed in "
            f"{ROOT_MOUNT_PATH}?")
    except subprocess.CalledProcessError as exc:
        LOG.warning("coreos-installer failed: %s", exc)
        raise

Agent in a container

CoreOS is a system that runs containers. Quite naturally, we will run ironic-python-agent as a container as well. We start with a Dockerfile; the one I've created is based on CentOS Stream 8 and packages from RDO.

Once the container is built and pushed to a suitable registry, we can proceed to the CoreOS configuration. When booted over network, CoreOS accepts parameters via the kernel command line, while the live ISO image can be customized through the special iso ignition embed command. In both cases Ignition is used to apply the configuration:

{
  "ignition": {
    "version": "3.0.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": ["<... SSH key here ...>"]
      }
    ]
  },
  "storage": {
    "files": [{
      "path": "/etc/ironic-python-agent.conf",
      "contents": {"source": "data:;base64,<... IPA config here ...>"}
    }]
  },
  "systemd": {
    "units": [
      {
        "contents": "<... service contents here ...>",
        "enabled": true,
        "name": "ironic-agent.service"
      }
    ]
  }
}

The configuration has three parts:

SSH configuration

An SSH public key for debugging. Optional.

Agent configuration

Instead of <... IPA config here ...> we need a base64-encoded configuration file. The minimum version will look roughly like this (decoded):

[DEFAULT]
api_url = http://<ironic IP>:6385
inspection_callback_url = http://<ironic IP>:5050/v1/continue

This configuration ensures that ironic-python-agent can reach back to the Ironic and Inspector services.

Systemd service

Like most modern Linux distributions, CoreOS uses systemd for launching services. We will create a service that fetches the container we prepared, mounts a few required locations (the configuration file, as well as ROOT_MOUNT_PATH), and runs ironic-python-agent:

[Unit]
Description=Ironic Agent
After=network-online.target
Wants=network-online.target

[Service]
TimeoutStartSec=0
ExecStartPre=/bin/podman pull <container registry>/ironic-agent
ExecStart=/bin/podman run --privileged --network host --mount type=bind,src=/etc/ironic-python-agent.conf,dst=/etc/ironic-python-agent/ignition.conf --mount type=bind,src=/dev,dst=/dev --mount type=bind,src=/sys,dst=/sys --mount type=bind,src=/,dst=/mnt/coreos --name ironic-agent ironic-agent

[Install]
WantedBy=multi-user.target

Insert your container registry address, then substitute the whole file into <... service contents here ...>, replacing line breaks with \n.

To simplify generating a suitable Ignition JSON, I've created a script build.py that is documented in the README:

$ ./ignition/build.py \
    --host 192.168.122.1 \
    --registry 192.168.122.1:5000 \
    --insecure-registry > /httpboot/ironic-agent.ign

The resulting JSON file can be embedded into an ISO or linked from the PXE kernel parameters, for example:

$ sudo coreos-installer iso ignition embed \
    -i /httpboot/ironic-agent.ign -f /httpboot/coreos.iso

Custom deploy in Metal3

The solution we've discussed so far is completely operational and can be testing on Bifrost. The last missing bit is Metal3 integration for the new deploy step (non-CoreOS deployments can already work). After a discussion among the Metal community, we settled down on a CustomDeploy feature that can potentially cover more than just CoreOS.

Instead of a normal BareMetalHost's image from my Metal3 overview

spec:
   # ...
   image:
     # The image URL can be in the qcow2 format or raw.
     url: http://images.example.com/images/my-os.qcow2
     # The image checksum URL: either the checksum itself or a file with
     # checksums per file name.
     checksum: http://images.example.com/images/my-os.qcow2.md5sum
     # Checksum type must be set if not md5. Supported are sha256 and sha512.
     checksumType: md5

… you will simply provide the name of the deploy step to run:

spec:
   # ...
   customDeploy: install_coreos

Finally, I have created a new script configure-coreos-ipa (currently only in the OpenShift version of ironic-image) that prepares a CoreOS live ISO.

Summary

The newly developed approach allows using coreos-installer without losing the benefits of Ironic and Metal3. It does come with certain limitations:

• The CoreOS live ISO is larger than a normal RHEL-based ramdisk image (~ 700 MiB vs ~ 400 MiB). The ironic-python-agent container is roughly 400 MiB on top of that. This leads to longer start-up times and increased memory requirements.

  On the other hand, we no longer need a large qcow2 CoreOS image, which makes the deployment itself faster.

• Ignition is not compatible with the more standard cloud-init formats, such as network data.

• The configuration is more complex and potentially error-prone.

In any case, it does its job, and you'll hopefully be able to test it in one of the upcoming OpenShift releases.

Footnotes