Bare Metal + Kubernetes = ♡

Metal3 deliverables

baremetal-operator (BMO)

The heart of the project: the BareMetalHost (BMH) custom resource definition and the controller to manage such resources. The main deployment tools can also be found in this repository.

cluster-api-provider-metal3 (CAPM3)

The cluster API provider using baremetal-operator to provision machines for the cluster.

ironic-image

The container image for Ironic with suitable configuration for Metal3.

ip-address-manager (IPAM)

An optional controller to manage IP addresses. See this blog post for details: Introducing the Metal3 IP Address Manager.

hardware-classification-controller

An optional controller to label hosts according to their physical properties, which are obtained through inspection (more about it below).

ironic-ipa-downloader

An init container to download the IPA (ironic-python-agent - our service agent that is loaded on the machines when provisioning) from the RDO project.

metal3-dev-env

Scripts and Ansible playbooks to set up a development environment using virtual machines. If you know OpenStack: metal3-dev-env is like DevStack.

ironic-client

A container image with the bare metal CLI for debugging. Of course you can just install it locally using:

$ pip install --user python-ironicclient

Why Ironic

… or how many people put it nowadays:

Why not Just Rewrite It in Go ™?

Maybe I'm just a bit too old to rewrite things that already work?

But seriously though, isn't it easier to just write bare metal management in a Kubernetes-native fashion from scratch? No. Definitely not.

When writing an image is not enough

Sometimes operators ask for the ability to use a custom installer while keeping all other benefits of Ironic and Metal3. This is also possible. By setting diskFormat to a special value live-iso, you can request Ironic to boot the provided ISO and finish the installation once it starts booting. From this point, you can use your installation procedure of choice.

spec:
   # ...
   image:
     diskFormat: live-iso
     # The image URL: an ISO in this case.
     url: http://images.example.com/images/my-installer.iso

If you already have bare metal machines provisioned through other means, you can simply add them as they are by marking them as externallyProvisioned:

spec:
   # ...
   externallyProvisioned: true
   image:
     url: http://images.example.com/images/my-os.qcow2
     checksum: http://images.example.com/images/my-os.qcow2.md5sum

How it works

When BMO starts, it accepts Ironic and Inspector endpoints and credentials (only HTTP basic authentication is supported), as well as deploy kernel/ramdisk URLs via environment variables. Ironic and Inspector are supposed to be started and managed separately, for example by using deployment templates provided in the BMO repository (e.g. OpenShift does it via a separate operator cluster-baremetal-operator).

Once started, the controller manages the BareMetalHost (BMH) CRD by synchronizing the changes between Kubernetes and Ironic. BMO expects to completely own the nodes defined as BMH. It creates them in Ironic if they are missing, updates them if they don't match the information it has, provisions if there is an image, de-provisions if the image is removed, deletes if the BMH is deleted (or detached).

Nodes are created with names looking like <NAMESPACE>~<BMH NAME>, for example on OpenShift you can see the following picture:

$ baremetal node list --fields uuid name provision_state
+--------------------------------------+---------------------------------------+--------------------+
| UUID                                 | Name                                  | Provisioning State |
+--------------------------------------+---------------------------------------+--------------------+
| 7ef3f064-2b86-4d34-8b33-fb5d127a713b | openshift-machine-api~ostest-master-0 | active             |
| 13b00cf0-562b-47ee-8d45-c0f4dba0e074 | openshift-machine-api~ostest-master-2 | active             |
| 8e13a1ad-1029-4106-aec2-ba640eb99a1e | openshift-machine-api~ostest-master-1 | active             |
+--------------------------------------+---------------------------------------+--------------------+
$ baremetal node show openshift-machine-api~ostest-master-0 --fields driver_info instance_info -f json
{
  "driver_info": {
    "deploy_iso": "http://localhost:6181/images/ironic-python-agent.iso",
    "deploy_kernel": "http://localhost:6181/images/ironic-python-agent.kernel",
    "deploy_ramdisk": "http://localhost:6181/images/ironic-python-agent.initramfs",
    "redfish_address": "http://[fd2e:6f44:5dd8:c956::1]:8000",
    "redfish_password": "******",
    "redfish_system_id": "/redfish/v1/Systems/3f32c07a-b060-4d44-b73b-894be044b347",
    "redfish_username": "admin"
  },
  "instance_info": {
    "capabilities": {},
    "image_source": "http://[fd00:1101::3]:6181/images/rhcos-48.84.202105190318-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202105190318-0-openstack.x86_64.qcow2",
    "image_os_hash_algo": "md5",
    "image_os_hash_value": "http://[fd00:1101::3]:6181/images/rhcos-48.84.202105190318-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202105190318-0-openstack.x86_64.qcow2.md5sum",
    "image_checksum": "http://[fd00:1101::3]:6181/images/rhcos-48.84.202105190318-0-openstack.x86_64.qcow2/cached-rhcos-48.84.202105190318-0-openstack.x86_64.qcow2.md5sum"
  }
}

One interesting consequence of this approach is that the Ironic's database (usually MySQL/MariaDB) becomes transient: it can be removed on the container restart, and BMO will re-create the nodes in the right states: any in-progress operations are started from scratch, while already deployed nodes are adopted.

Future plans

There is a lot of work planned in Metal3. In the near future, we want to bring BIOS and RAID support to BMO, previews of these features are already available. Another contributor is working on a network configuration proposal, which will allow in the future to manage bare metal switches while provisioning the nodes connected to them.

Looking further in the future, I'd like us to consider the possibility of dropping MySQL in favor of either SQLite or anything else that is not a full-featured database. Additionally, we need to develop our multi-master story: currently the Metal3 pod with all services is usually deployed on one master.