How to install OpenShift Data Foundation (ODF) 4.11 in a disconnected or air-gapped VMware cluster

Estimated reading time: 8 minutes

Motivation

Red Hat OpenShift Data Foundation (ODF), previously Red Hat OpenShift Container Storage, is a software-defined persistent storage solution for OpenShift Container Platform (OCP) workloads. In this article you’ll learn how to deploy OpenShift Data Foundation using local storage devices on a disconnected VMware cluster.

An air-gapped or disconnected network is a security measure to ensure that a computer network is physically isolated from insecure networks, like the Internet or insecure local networks. Because the network is offline or does not have full Internet access, air-gapped environments can keep critical systems and sensitive information safe from potential data theft or security breaches and reduce the risk of malicious attacks.

This article does not cover how to install OpenShift Container Platform nor how to configure it.

In this article, you’ll first learn how to mirror the required contents for your container registries and installation media as a disconnected cluster does not have an active connection to the internet. You can create this mirror registry on a host that can access both the internet and your closed network, or copy images and/ or operators to a device that you can move across network boundaries.

Once the mirroring is completed, you’ll need to install the operators to run OpenShift Data Foundation so that you can finally deploy and configure OpenShift Data Foundation.

The steps in this article can also be applied to air-gapped environments with no internet access at all.

Article overview

Prerequisites
Prepare the disconnected installation
Install the Local Storage and OpenShift Data Foundation operators
Configure and deploy the storage system

Let’s get to it!

1. Prerequisites

OpenShift version 4.10+; the steps in this article were last tested with OpenShift version 4.11
Container registry in your air-gapped environment (e.g. an OpenShift mirror registry or another Docker registry; in this article the example registry is labelled registry-mirror.openshift.airgapped:17443)
OpenShift CLI (oc) tool
Storage: ODF requires at least three infrastructure nodes with at least one locally attached empty storage device each. In this example we manually added a 200 GB disk device to each of the three compute nodes in VMware.
A Red Hat account
Podman
Access to the internet for downloads

2. Prepare the disconnected installation

The following steps outline the high-level workflow on how to use the oc-mirror utility to mirror images to a mirror registry:

Create an image set configuration file.
Mirror the image set to the mirror registry:
- Partially disconnected environment: Mirror an image set directly to the mirror registry.
- Fully disconnected environment: Mirror an image set to disk, transfer the image set to the target environment via e.g. an USB drive, then upload the image set to the target mirror registry.
Install the ImageContentSourcePolicy and CatalogSource resources that were generated by oc-mirror into the cluster.

For the sake of readability, we organized the steps based on the environment used.

Partially disconnected environment

Firstly, create the image set configuration file (ImageSetConfiguration.yaml) as in the example below:

---
apiVersion: mirror.openshift.io/v1alpha2
kind: ImageSetConfiguration
archiveSize: 4
storageConfig:
  registry:
    imageURL: registry-mirror.openshift.airgapped:17443/oc-mirror/operator
    skipTLS: false
        #  local:
        # path: /home/admin/oc_mirror_new/mirror
mirror:
  platform:
    channels:
    - name: stable-4.11
      type: ocp
    graph: true
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.11
    packages:
    - name: odf-operator
      channels:
      - name: stable-4.11
    - name: local-storage-operator
      channels:
      - name: stable
    - name: odf-csi-addons-operator
      channels:
      - name: stable-4.11
    - name: ocs-operator
      channels:
      - name: stable-4.11
    - name: mcg-operator
      channels:
      - name: stable-4.11Code language: PHP (php)

Secondly, mirror the image set directly to the mirror registry:

oc-mirror --config ImageSetConfiguration.yaml docker://registry-mirror.openshift.airgapped:17443Code language: JavaScript (javascript)

…

Finally, install the ImageContentSourcePolicy and CatalogSource resources:

Log into OpenShift with cluster-admin permissions.

oc login ...

Disable the default OperatorHub sources by adding disableAllDefaultSources: true to the OperatorHub object.

oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'Code language: JavaScript (javascript)

Finally, apply both catalogSource.yaml and imageContentSourcePolicy.yaml files generated in the previous step. This adds the mirrored operator catalog as an OperatorHub source and uses the mirror registry as a fallback for the container images.

oc apply -f catalogSource.yaml
oc apply -f imageContentSourcePolicy.yamlCode language: CSS (css)

Verify that the ImageContentSourcePolicy resources were successfully installed:

oc get imagecontentsourcepolicy --all-namespacesCode language: JavaScript (javascript)

Verify that the CatalogSource resources were successfully installed:

oc get catalogsource --all-namespacesCode language: JavaScript (javascript)

Note: If you have the Update Service Operator installed, you can also apply the configuration file set via the results-<number> folder that has been generated using the oc-mirror command:

oc apply -f ./oc-mirror-workspace/results-<number>/Code language: HTML, XML (xml)

Fully disconnected environment

Firstly, create the image set configuration file (ImageSetConfiguration.yaml) as in the example below:

---
apiVersion: mirror.openshift.io/v1alpha2
kind: ImageSetConfiguration
archiveSize: 4
storageConfig:
        #  registry:
        #    imageURL: registry-mirror.openshift.airgapped:17443/oc-mirror/operator
        #    skipTLS: false 
  local:
    path: /home/admin/oc_mirror_new/mirror
mirror:
  platform:
    channels:
    - name: stable-4.11
      type: ocp
    graph: true
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.11
    packages:
    - name: odf-operator
      channels:
      - name: stable-4.11
    - name: local-storage-operator
      channels:
      - name: stable
    - name: odf-csi-addons-operator
      channels:
      - name: stable-4.11
    - name: ocs-operator
      channels:
      - name: stable-4.11
    - name: mcg-operator
      channels:
      - name: stable-4.11Code language: PHP (php)

Secondly, mirror the image set to disk and transfer the image set to the target environment via e.g. an USB drive.

oc-mirror --config ImageSetConfiguration.yaml file://mirrorCode language: JavaScript (javascript)

…

Thirdly, upload the image set to the target mirror registry. In case you want to try this with an insecure registry, add --dest-skip-tls or --dest-use-http to oc mirror.

oc-mirror --from ./mirror docker://registry-mirror.openshift.airgapped:17443Code language: JavaScript (javascript)

…

Verify that the contents have been successfully mirrored into the registry via curl:

curl -k -u $registry_user:$registry_password -X GET https://registry-mirror.openshift.airgapped:17443/v2/_catalogCode language: PHP (php)

Finally, install the ImageContentSourcePolicy and CatalogSource resources:

Log into OpenShift with cluster-admin permissions.

oc login ...

Disable the default OperatorHub sources by adding disableAllDefaultSources: true to the OperatorHub object.

oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'Code language: JavaScript (javascript)

Finally apply both catalogSource.yaml and imageContentSourcePolicy.yaml files generated in the previous step. This adds the mirrored operator catalog as an OperatorHub source and uses the mirror registry as a fallback for the container images.

oc apply -f catalogSource.yaml
oc apply -f imageContentSourcePolicy.yamlCode language: CSS (css)

Verify that the ImageContentSourcePolicy resources were successfully installed:

oc get imagecontentsourcepolicy --all-namespacesCode language: JavaScript (javascript)

Verify that the CatalogSource resources were successfully installed:

oc get catalogsource --all-namespacesCode language: JavaScript (javascript)

Note: If you have the Update Service Operator installed, you can also apply the configuration file set via the results-<number> folder that has been generated using the oc-mirror command:

oc apply -f ./oc-mirror-workspace/results-<number>/Code language: HTML, XML (xml)

3. Install the Local Storage and OpenShift Data Foundation operators

Log into the Console using an account with cluster-admin permissions and navigate to Operators → OperatorHub:

Install the Local Storage Operator

Type local storage in the Filter by keyword... box to find the Local Storage Operator and click on it.

Click Install.

Set the following options on the Install Operator page, then click Install:

Update channel as stable
Installation mode as A specific namespace on the cluster.
Installed Namespace as Operator recommended namespace openshift-local-storage.
Update approval as Automatic.

Verify that the Local Storage Operator shows a green tick indicating successful installation.

Install the Red Hat OpenShift Data Foundation Operator

Type OpenShift Data Foundation in the Filter by keyword... box to find the OpenShift Data Foundation Operator and click on it.

Click Install.

Set the following options on the Install Operator page, then click Install:

Update Channel as stable-4.11.
Installation Mode as A specific namespace on the cluster.
Installed Namespace as Operator recommended namespace openshift-storage.
Select Approval Strategy as Automatic.
Ensure that the Enable option is selected for the Console plugin.

Verify that the OpenShift Data Foundation Operator shows a green tick indicating successful installation.

Verify that the pods are running

Firstly navigate to the Workloads → Pods section. Then verify that all pods are running in the openshift-local-storage project:

Secondly, verify that all pods are running in the openshift-storage project.

4. Configure and deploy the storage system

In the Console, under OpenShift Data Foundation select Create StorageSystem.

Select Full deployment as Deployment type and select Create a new StorageClass using local storage devices under Backing storage type.

Enter local-volume-sdb for the Volume Set Name and select the worker nodes.

Click Advanced and select the disk size, e.g. for 200 GB select Min: 200 GiB, Max: 200 GiB

Click Next and click Yes when prompted to create a LocalVolumeSet.

Note: Depending on the number of resources available, a warning may show up that a minimal cluster deployment will be performed which is fine for testing but should not be used in production.

Accept the defaults by clicking Next and finally click Create StorageSystem.

To verify that an instance of a StorageCluster was created in the back-end, run

oc get storagecluster -n openshift-storageCode language: JavaScript (javascript)

You can check the status of the storage cluster with the following:

oc get storagecluster -n openshift-storage <storageclustername> -o jsonpath='{.status.phase}{"\n"}'Code language: JavaScript (javascript)

If it says Ready, you can continue.

Miscellaneous

If you run into resource limits, you could override the ratio between requests and limits set on containers/ pods using the Cluster Resource Override Operator for testing purposes.

Make sure to check out the list of Red Hat Operators Supported in Disconnected Mode.

Conclusion

In this article we went through some of the basics of OpenShift’s operators & storage stack. You should now be able to install OpenShift Data Foundation on VMware in a disconnected or even air-gapped environment.

References

Authors

Andrea Müller-Hansen

Andrea is a technology engineer in the IBM Client Engineering team, where she pairs with clients to solve complex business challenges with innovative solutions. She has a huge appetite for learning and a passion for security.

Chris Schneider

Chris is a technology engineer in the IBM Client Engineering team. He works with clients to develop, test and deploy AI-enabled applications on container platforms like Red Hat OpenShift.

Special thanks to…

We would like to thank Cayo de Moraes for providing an air-gapped environment and Robert Bohne & Matthias Muench for their input and feedback.