Container images, multi-architecture, manifests, ids, digests – what’s behind?

Many of us use container images from day to day, maybe also in many various architectures. For example on your Raspberry PI (aarch64), do you really know how it works in detail?

I have worked with container images more or less since 2015 but during an OpenShift 4 air-gapped installation and mirroring of images into a registry, I hit the wall quite hard and have to realize I didn’t know all the details very well. 

If you are not familiar with OpenShift 4 air-gapped installation, the installation is divided into two parts: first OpenShift 4 Core installation and second the cluster add-ons via OperatorHub/Operator Lifecycle Manager. The second part is optional but brings a lot of value into OpenShift. 

Today I am excited to share my learnings with you.

Let’s start off with some basics around container images

A container image is a static immutable packaging and shipping format. It contains everything you need to start a container, the actual software packages and information on how to start it at your container runtime. A container image is distributed via a container registry like quay.io or Docker Hub.

All the important format definitions for distribution and runtime of a container image are specified in the open container initiative (OCI) – https://opencontainers.org/.

The container workflow from build to run is:

Building a container 

There are various blog posts and articles about how to build containers. All you have to know for now is that you need a  Containerfile (formally known as Dockerfile) and with podman, buildah or docker you build your container from it. 

During the build process, it creates a bunch of file system layers including the container configuration (how to run, port, volume details). The file system layers contain all the software components, operating system libraries, of your software – basically all you need to run your software inside the container.

Depending on the build environment it also creates a temporary Manifest. The Manifest itself will become important later during the container image distribution. 

At this point, we have a container image that is made to run in the architecture it is built for. For example x86 on an x86 host or aarch64 on aarch64 host. At the end of the build you also get the Image ID of your container image.  

Here are the key details taken from the Image ID specification:
Each image’s ID is given by the SHA256 hash of its configuration JSON. It is represented as a hexadecimal encoding of 256 bits, e.g., sha256:a9561eb1b190625c9adb5a9513e72c4dedafc1cb2d4c5236c9a6957ec7dfd5a9. Since the configuration JSON that gets hashed references hashes of each layer in the image, this formulation of the ImageID makes images content-addressable.

After the build on the build machine, the final container image digest will not be available as the digest. The digests will be created during the push into the container registry.

The container image configuration is specified here:

• application/vnd.oci.image.config.v1+json

For FileSystem layers, their are different types / specifications available:

• application/vnd.oci.image.layer.v1.tar
• application/vnd.oci.image.layer.v1.tar+gzip
• application/vnd.oci.image.layer.v1.tar+zstd
• application/vnd.oci.image.layer.nondistributable.v1.tar
• application/vnd.oci.image.layer.nondistributable.v1.tar+gzip
• application/vnd.oci.image.layer.nondistributable.v1.tar+zstd
• application/vnd.docker.image.rootfs.diff.tar.gzip

In case you miss the first version: application/vnd.docker.container.image.v1+json: Docker Image Specification v1 Image JSON has still been widely used and officially adopted in V2 manifest and OCI Image Format Specification.

Push – image distribution and a little bit of container registry 

After an image build, you typically distribute your container image to a container image registry.  At this point the manifest comes into play. Container registries can support different manifest types:

• application/vnd.oci.image.manifest.v1+json (oci)
• application/vnd.docker.distribution.manifest.v1+json – broadly called as schema1 (v2s1)
• application/vnd.docker.distribution.manifest.v2+json – broadly called as schema2 (v2s2)

All major registries support schema2. If you want to know how quay.io – the first private registry – can grow from v1 to v2.2 (schema2)  then I recommend the following video:

Have a look at the following snippet, which shows an example of a schema2 manifest:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 2654,
    "digest": "sha256:85db140f49f9135479330babb875fe46713b7abcaeda290bf36aaf2977688569"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 42050576,
      "digest": "sha256:7697e6e7be39d9b66c05a4276d8d0674c1b617a4ceec971e2aef37c07240f139"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1819,
      "digest": "sha256:b24bc1e6fa6137aa1685aad067db2b22f3c24410f02c3994c8f71d0720f03ba0"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 158,
      "digest": "sha256:8ba23a6869850ca5c696f5cfd2505ac81c62529315c976736724599bda439704"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1307298,
      "digest": "sha256:c0cf0faa3e01881baf1e369ac39b17ef9977aaafe3a18232a5114a5cdec0bef2"
    }
  ]
}Code language: JSON / JSON with Comments (json)

The container image digest SHA256 hash (not to be confused with the SHA256 hash in the manifest layers) is calculated and created by hashing the manifest output shown above during the push. The container image digest hash will of course change if you change anything in the manifest like schemeVersion(schema1 vs schema2) or mediaType. 

IMPORTANT:  if you pull, tag, and push an image, the image digest can change in cases where the image layer compression changes and/or the manifest version has been converted. The only secure way is to use skopeo copy! 

Usually, images are pushed to a registry via a tag. You can consider a tag as a reference to a specific container image digest, which can be rewritten to another container image (digest).  If you want to run a specific version of a container image you can use the container image digest. If something changes on the container image: configuration, filesystem layer the container image digest will change too.

Here an example of the different formats:

quay.io/openshift-examples/multi-arch:x86_64
quay.io/openshift-examples/multi-arch@sha256:6e7f0459dc4c93970c8207f0640ebf0f85a5be180e0981a8a20426dd456f16c7

You may discover image tags and digests via curl as outlined in the  Open Container Initiative Distribution Specification . Here is an example:

Get the list of tags:

$ curl -s https://quay.io/v2/openshift-examples/multi-arch/tags/list | jq
{
  "name": "openshift-examples/multi-arch",
  "tags": [
    "x86_64",
    "aarch64"
  ]
}Code language: JavaScript (javascript)

Get the manifest of an tag:

$ curl -s -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
https://quay.io/v2/openshift-examples/multi-arch/manifests/x86_64 | jq -r
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 2654,
    "digest": "sha256:85db140f49f9135479330babb875fe46713b7abcaeda290bf36aaf2977688569"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 42050576,
      "digest": "sha256:7697e6e7be39d9b66c05a4276d8d0674c1b617a4ceec971e2aef37c07240f139"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1819,
      "digest": "sha256:b24bc1e6fa6137aa1685aad067db2b22f3c24410f02c3994c8f71d0720f03ba0"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 158,
      "digest": "sha256:8ba23a6869850ca5c696f5cfd2505ac81c62529315c976736724599bda439704"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1307298,
      "digest": "sha256:c0cf0faa3e01881baf1e369ac39b17ef9977aaafe3a18232a5114a5cdec0bef2"
    }
  ]
}Code language: JavaScript (javascript)

Get the digest of the manifest:

$ curl -s -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
 https://quay.io/v2/openshift-examples/multi-arch/manifests/x86_64 | sha256sum
6e7f0459dc4c93970c8207f0640ebf0f85a5be180e0981a8a20426dd456f16c7  -Code language: JavaScript (javascript)

Get the manifest via sha256:

curl -s -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
https://quay.io/v2/openshift-examples/multi-arch/manifests/sha256:6e7f0459dc4c93970c8207f0640ebf0f85a5be180e0981a8a20426dd456f16c7 | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 2654,
    "digest": "sha256:85db140f49f9135479330babb875fe46713b7abcaeda290bf36aaf2977688569"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 42050576,
      "digest": "sha256:7697e6e7be39d9b66c05a4276d8d0674c1b617a4ceec971e2aef37c07240f139"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1819,
      "digest": "sha256:b24bc1e6fa6137aa1685aad067db2b22f3c24410f02c3994c8f71d0720f03ba0"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 158,
      "digest": "sha256:8ba23a6869850ca5c696f5cfd2505ac81c62529315c976736724599bda439704"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1307298,
      "digest": "sha256:c0cf0faa3e01881baf1e369ac39b17ef9977aaafe3a18232a5114a5cdec0bef2"
    }
  ]
}Code language: JavaScript (javascript)

In the container registry, the configuration and filesystem layers are stored as a blobs.

To get the image configuration of the manifest above:

$ curl -L -s https://quay.io/v2/openshift-examples/multi-arch/blobs/sha256:85db140f49f9135479330babb875fe46713b7abcaeda290bf36aaf2977688569Code language: JavaScript (javascript)

To visualize this a bit:

Of course, if a filesystem layer (the blob at the end) still exists, it will not be pushed or pulled again. 

Pull & Run your container at your container runtime

This blog post focuses on the container image format including pushing, but not pulling and/or running a container image. For that there are a couple of other blogs listed below.

• Running containers with Podman and shareable systemd services
• Podman: Managing pods and containers in a local container runtime

More information can also be found in the runtime specification here: Open Container Initiative Runtime Specification

Multi-architecture container images 

The reason for multi-architecture (multi-arch) container images is to specify an easy way to ship your software via a container image with the same tag to different architectures and operating systems.  For example: Linux on IBM power (ppc64le), Z (s390x), ARM ( aarch64), x86, or Windows on x86. 

That brings us to multi-arch images, which are just another manifest (type also called “fat manifest”):

• application/vnd.docker.distribution.manifest.list.v2+json
• application/vnd.oci.image.index.v1+json

This manifest points to the actual container image of the specific architecture. Have a look at the following illustration and pay special attention to the sha256 in the manifest boxed. 

Here is another technical example: 

“fat manifest” aka multi-arch image: quay.io/openshift-examples/multi-arch:multi containers two architectures: amd64 and arm64

$ curl -s -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json'  https://quay.io/v2/openshift-examples/multi-arch/manifests/multi   | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 914,
      "digest": "sha256:bc27b727eaeea61242f16f5534a0a54d1f1fa17f9f5dcec52e8427e5dc49eb93",
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 914,
      "digest": "sha256:6e7f0459dc4c93970c8207f0640ebf0f85a5be180e0981a8a20426dd456f16c7",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    }
  ]
}Code language: JavaScript (javascript)

Usually, if you run the image quay.io/openshift-examples/multi-arch:multi on aarch64 you will immediately forward to aarch64 and pull the architecture matching container image. Have a look at the following example for  aarch64

$ podman run -ti --rm quay.io/openshift-examples/multi-arch:multi
hello world from architecture: aarch64Code language: JavaScript (javascript)

Another example, if you run skopeo copy on aarch64:

$ skopeo copy docker://quay.io/openshift-examples/multi-arch:multi  docker://quay.io/openshift-examples/multi-arch:skopeo-copy-on-aarch64
$ skopeo inspect docker://quay.io/openshift-examples/multi-arch:skopeo-copy-on-aarch64  | jq .Digest
"sha256:bc27b727eaeea61242f16f5534a0a54d1f1fa17f9f5dcec52e8427e5dc49eb93"Code language: JavaScript (javascript)

By default it copies only the architecture where the skopeo copy command is running. If you want to copy the fat manifest and all architectures you have to run

skopeo copy --all

How to create your own multi-architecture container images

During my discovery, I wanted to build my own multi-architecture container images to get a better understanding. 

I picked Go as a programming language and wrote a “Hello World” including printing the current architecture. I know, not the best Go example but good enough for the multi-architecture use case. You can find my Go code and the Containerfile to build the container image on Github: https://github.com/openshift-examples/multi-arch

Via Fedora Cloud I started one x86 and one aarch64 instance one AWS to build & push the different architecture container images: 

dnf install -y podman git 
git clone https://github.com/openshift-examples/multi-arch.git
cd multi-arch
podman build -t quay.io/openshift-examples/multi-arch:$(uname -m) .
podman push --format v2s2 quay.io/openshift-examples/multi-arch:$(uname -m)Code language: JavaScript (javascript)

After successfully building & pushing the x86 and aarch64 container images, we have to created the “fat manifest”:

podman manifest create mylist
podman pull quay.io/openshift-examples/multi-arch:aarch64
podman manifest add mylist quay.io/openshift-examples/multi-arch:aarch64
podman pull quay.io/openshift-examples/multi-arch:x86_64
podman manifest add mylist quay.io/openshift-examples/multi-arch:x86_64
podman manifest push mylist  docker://quay.io/openshift-examples/multi-arch:multiCode language: JavaScript (javascript)

The result is available on quay.io: https://quay.io/repository/openshift-examples/multi-arch?tab=tags

Back to my OpenShift 4 air-gapped installation and mirroring problem

Preamble: if you want to run OpenShift 4 in an air-gapped environment you have to mirror all necessary container images from Red Hat into your mirror registry as shown in the official documentation:

• Creating a mirror registry for installation in a restricted network
• Using Operator Lifecycle Manager on restricted networks

I mirrored a couple of the Operators from the Operator Lifecycle Manager/OperatorHub for usage in an air-gapped environment. Everything looked fine until I tried to install the operator and received the following error message: “ImagePullBackOff of quay.io/xxx/xxx@sha256:xyz..”  

Strange I thought I mirrored everything….

I analysed the logs of the mirroring process and it appeared that “sha256:xyz” was synced. But “sha256:xyz” is a multi-arch fat manifest and not the container image itself. The mirror process only synced my architecture container image with a different sha256 (the sha256 of the target architecture). After running a skopeo copy --all the fat manifest was available in my mirror registry. Optional you can add --filter-by-os=’.*’ to oc adm catalog mirror  or oc image mirror. Bug report: Bug 1890951

The reason for the multi-architecture images is: OpenShift is available on x86, IBM Z – Linux ONE, IBM Z and maybe at some point in time for ARM too. 

Looking forward to your feedback.