Setting up an Enterprise OpenShift 3.5 Platform on macOS with VirtualBox

I am a middleware guy. I am one of those guys who love to think about solving implementation problems. I am not necessarily one of those guys who love to dig into the infrastructure part too much. Typically, required things should be there so that I can use them for my work.

But since some months I am travelling around to explain people the usage of OpenShift Container Platform. All I am talking about is the development part of it (so how to create apps, how to do release management with it etc.). But from time to time I am getting a lot of questions on the infrastructure part of OpenShift as well.

Now it was time for me to install OpenShift on my macOS based laptop inside a virtual machine. I just wanted to understand everything which is necessary to use OpenShift.

Until then I was only using either the Red Hat Container Development Kit (CDK) which is based on minishift since 3.0 or “oc cluster up” (and of course the great script set oc-cluster-wrapper done by our Evangelist team).

https://developers.redhat.com/products/cdk/overview/

https://github.com/openshift-evangelists/oc-cluster-wrapper

So this is my journey. Using VirtualBox. Installing RHEL 7.3. Using the subscription-manager to attach a valid subscription to it. Installing OpenShift 3.5 using the advanced method. And so on.

But let me start now.

Using VirtualBox and making sure you can use all the OpenShift services also from outside the virtual machine.

I am a Mac guy. I don’t really like the UI of Gnome or KDE. So my decision was easy: I want to have a server VM only. And I want to make sure that I can do everything else from my macOS host.

This sounds easier than it is. Because VirtualBox provides a set of different Network modes, where NAT and NAT Network allows the VM to use the host system as an internet gateway. But I am not able to connect inside the VM from my macOS host system, except if I’d do ssh port forwarding. But this has one BIG drawback: I can not use my host system as I would like to. The OpenShift WebUI wouldn’t work. I would not be able to access my newly created apps.

Bridged Networking was my second thought. This should work. Here the VM would be an official part of the overall network, where my host macOS is connected to as well. So everything I do with my host system would also be possible with the VM (like internet etc.). But this mode also has a BIG drawback: I am working in different environments. And this means the IP address changes every time I am in a new environment. And this is nothing OpenShift can and / or should deal with. OpenShift needs a static IP.

So the next idea was to use two network interfaces. One with NAT to use the host internet system and the other one with Host Only networking (host only means, that the Host and all VMs inside the same host-only network environment can see and access each other). And my idea was to bind OpenShift on the IP address of the host-only network, so that I can play with OpenShift from my Mac.

This has a lot of drawbacks. But the biggest one is that the pods of OpenShift are not able to talk to the outside of OpenShift as they are all bound to the internal network. Which means, I would not be able to use OpenShift’s source to image mechanism where I am just providing a GIT repository and OpenShift is building an Image out of it.

So there must be something more to do.

The idea a friend came up with is to have 2 VMs. One simple VM acts as a router. It should have two network interfaces (one NATed and the other one with Host-Only). It should have dnsmasq being configured and should be acting as DHCP and DNS server.

The other VM, where I’d like to install OpenShift on, should only have host-only networking and should use the router VM as – well – the router.

Setting up the Router VM

As I am working at Red Hat, it was clear for me that I have to use either Fedora, CentOS or RHEL for this machine. While Fedora is IMHO too new, I thought about CentOS or RHEL. At the end I decided against CentOS as my OpenShift VM should also be running RHEL.

I am giving my router VM 1 GB of RAM and 1 vCPU. As I might be using it as another Node for OpenShift, I prepared a hard disk with 30GB.

After installing RHEL 7.3 as a minimal server, I registered the system via subscription-manager, attached my Employee SKU to the system via the WebUI under access.redhat.com and enabled only the necessary repository:

$ subscription-manager register
$ subscription-manager repos --disable=*
$ subscription-manager repos --enable=rhel-7-server-rpms

My host-only network is called .ocp.lan And the hostname of the router is ‚router‘ (I was never good with names, though).

In VirtualBox I created a new host-only network called vboxnet1 and gave it the IP Address range of 192.168.56.x/24. I turned DHCP off.

$ ip a

gives me a list of all configured network interfaces:

2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:dc:4f:93 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
       valid_lft 81696sec preferred_lft 81696sec
    inet6 fe80::b230:74a5:a50f:621d/64 scope link
       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:2d:03:4f brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.10/24 brd 192.168.56.255 scope global enp0s8
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe2d:34f/64 scope link
       valid_lft forever preferred_lft forever

This shows me that enp0s3 is the NAT interface and enp0s8 the host-only network interface. This is important to know.

It’s now getting deeper into network configuration:

Let’s make sure, enp0s8 is set to static:

$ vim /etc/sysconfig/network-scripts/ifcfg-enp0s8

TYPE=Ethernet
BOOTPROTO=static
NAME=enp0s8
DEVICE=enp0s8
ONBOOT=yes
IPADDR=192.168.56.10
PREFIX=24

Make sure, ONBOOT is yes and BOOTPROTO is set to static.

Now make sure the NAT interface card is also correctly set up:

$ vim /etc/sysconfig/network-scripts/ifcfg-enp0s3

TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
NAME=enp0s3
DEVICE=enp0s3
ONBOOT=yes

Let’s make sure, our router is called accordingly:

$ vim /etc/hostname

router.ocp.lan

And then we need to define the host names and IPs for our network, using /etc/hosts

$ vim /etc/hosts

127.0.0.1 localhost
192.168.56.10 router.ocp.lan router
192.168.56.11 master.ocp.lan master
192.168.56.12 node1.ocp.lan node1
192.168.56.13 node2.ocp.lan node2

Let’s do a reboot now and make sure, you’re able to ping the internet:

$ systemctl reboot

And after the system is back up, test, if you are able to reach google.com

$ ping google.com

Then let’s install DNSmasq:

$ yum install dnsmasq

I installed the DNS and DHCP server dnsmasq.

This is the /etc/dnsmasq.conf file:

domain-needed
domain=ocp.lan
pid-file=/var/run/dnsmasq1.pid
bind-dynamic
interface=enp0s8
dhcp-range=192.168.56.101,192.168.56.199,2h
dhcp-no-override
dhcp-leasefile=/var/lib/dnsmasq/dnsmasq.leases
dhcp-lease-max=253

# Router
dhcp-option=3,192.168.56.10
dhcp-option=1,255.255.255.0

# OSCP pointing to master
address=/.shift.ocp.lan/192.168.56.11

# Static Hosts
dhcp-host=08:00:27:B9:27:D2,master,192.168.56.11,72h

Now make sure, dnsmasq is started and enabled for startup during boot:

$ systemctl enable dnsmasq
$ systemctl start dnsmasq

With

$ journalctl -f

you can check to see if dnsmasq was able to start correctly.

Setup iptables

It is easier to use IPTables as a router firewall than FirewallD. So the next thing was to turn off FirewallD and to install iptables daemon:

$ systemctl stop firewalld
$ systemctl disable firewalld
$ yum install iptables-services

Flush all iptables rules to make sure nothing is left over from firewalld

$ iptables -F

Is really everything cleaned?

$ iptables -vnL

-v means verbose, -n means numeric only and -L means LIST all rules

Now add the one and only rule, save the rule table as configuration for iptables in /etc/sysconfig/iptables and enable iptables during boot

$ iptables -t nat -A POSTROUTING -o enp0s3 -j MASQUERADE
$ iptables-save > /etc/sysconfig/iptables
$ systemctl enable iptables

-t nat means, it should do NAT

-A POSTROUTING means append this rule to postrouting chain

-o use interface enp0s3

-j MASQUERADE means jump to the output chain

If you want to configure a Linux system as a router, you need to tell the kernel that it must forward packages accordingly. This is done via

$ vim /etc/sysctl.conf

and add the follwing line in there:

net.ipv4.ip_forward = 1

For more information, please have a look here: https://www.tecmint.com/setup-linux-as-router/

That’s it. After a restart, make sure, everything still works as expected. And if so, let’s move on to do a

Full Installation of OpenShift Container Platform 3.5 on RHEL 7.3

First of all, you have to create a new VirtualBox VM with at least the following properties:

• 2 cores (minimum)
• 6-8 GB memory
• Disk 1: 30 GB disk space
• Disk 2: 30 GB disk space (for docker), DON’T use it during RHEL setup!
• ONE network adaptor based on the previously created HostOnly network

Now go to the Network Settings within the VirtualBox VM, click on Advanced settings and make a note of the MAC-Address. Should be something like 080027B927D2.

In the router VM, open /etc/dnsmasq.conf and go to the end of the file and change the line according to your mac address of the OpenShift VM:

$ vim /etc/dnsmasq.conf

# Static Hosts

dhcp-host=08:00:27:B9:27:D2,master,192.168.56.11,72h

Change the ’08:00:27…’ to your mac address and make sure that you are using ‘:’ after every two characters.

A

$ systemctl restart dnsmasq

Makes sure that dnsmasq is using your changes.

Now do your basic RHEL 7.3 installation, using minimal-server profile. Once, the system is up and running, it should automatically get the IP address from our router: 192.168.56.11

$ ip a

should have a enp0s3 interface with the IP Address properly set. Please check to see if you can ping the outside world. DNS name resolution should also work.

I have been following the blog of Keith Tenzer to install OpenShift 3.4, so I don’t want to repeat everything here. I just want to concentrate on the things which are different with our setup

https://keithtenzer.com/2017/03/13/openshift-enterprise-3-4-all-in-one-lab-environment/

Because Keith’ blog post is for OpenShift 3.4, you need to adjust some parts. Just follow the original documentation for preparing your host. Once you’re done with the prep, we are continuing to install OpenShift

https://docs.openshift.com/container-platform/3.5/install_config/install/host_preparation.html

This is the /etc/ansible/hosts file

# Create an OSEv3 group that contains the masters and nodes groups
[OSEv3:children]
masters
nodes

# Set variables common for all OSEv3 hosts
[OSEv3:vars]

# SSH user, this user should allow ssh based auth without requiring a password
ansible_ssh_user=root

# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]

os_sdn_network_plugin_name='redhat/openshift-ovs-subnet'
openshift_master_default_subdomain=shift.ocp.lan
deployment_type=openshift-enterprise
openshift_use_dnsmasq=true
openshift_hosted_router_selector='targetrouter=true'

# External NFS Host
# NFS volume must already exist with path "nfs_directory/_volume_name" on
# the storage_host. For example, the remote volume path using these
# options would be "nfs.example.com:/exports/registry"
openshift_hosted_registry_storage_kind=nfs
openshift_hosted_registry_storage_access_modes=['ReadWriteMany']
openshift_hosted_registry_storage_host=192.168.56.11
openshift_hosted_registry_storage_nfs_directory=/var/exports/openshift
openshift_hosted_registry_storage_volume_name=registry
openshift_hosted_registry_storage_volume_size=10Gi

# Metrics
openshift_hosted_metrics_deploy=true
openshift_hosted_metrics_storage_kind=nfs
openshift_hosted_metrics_storage_host=192.168.56.11
openshift_hosted_metrics_storage_access_modes=['ReadWriteOnce']
openshift_hosted_metrics_storage_nfs_directory=/var/exports/openshift
openshift_hosted_metrics_storage_volume_name=metrics
openshift_hosted_metrics_storage_volume_size=10Gi

# host group for masters
[masters]
master.ocp.lan

# host group for nodes, includes region info
[nodes]
master.ocp.lan openshift_schedulable=True openshift_node_labels="{'targetrouter': 'true'}"

This all installs the Registry, the Router and also the Metrics subsystem. — But there is still one thing we need to configure in order to have persistant volumes. NFS. As you can see in the ansible vars section above, Metrics and Registry are using a storage, which we need to configure now.

Installing NFS server and client on the same machine

I basically followed the following blog entry here:

https://linuxconfig.org/quick-nfs-server-configuration-on-redhat-7-linux

With the following exceptions:

• The nfs server is our local OpenShift installation, so the IP is 192.168.56.11
• the export folder is /var/exports/openshift

So you’re doing a

$ mkdir -p /var/exports/openshift
$ vim /etc/exports
/var/exports/openshift 192.168.56.11(no_root_squash,rw,sync)

$ service nfs-server start
$ systemctl enable nfs

To test if nfs works, you could do the following:

$ mkdir /mnt/openshift
$ mount 192.168.56.11:/var/exports/openshift /mnt/openshift
$ touch /mnt/openshift/test

you know should have a file called ‘test’ in /var/exports/openshift

Installing OpenShift: Final step

The only thing you should do now is the following:

$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml

It takes approximately 30 minutes to install it. If it doesn’t work or breaks with an error, don’t worry, you can always rerun this playbook.

For example, one of my mistakes were that I forgot to do the ssh-copy-id command as explained in Keith’s blog.

Configuring OpenShift: Create users

This creates an admin users with cluster-admin rights and a simple developer user.

$ oc login -u system:admin -n default
$ htpasswd -c /etc/origin/master/htpasswd admin
$ oadm policy add-cluster-role-to-user cluster-admin admin
$ htpasswd -c /etc/origin/master/htpasswd developer

Configuring your macOS Host

So now that your OpenShift environment is up and running, you should do some basic configuration of your macOS host. Otherwise you’re not able to access your host or – most importantly – your generated applications.

Why? Because OpenShift dynamically creates DNS names for each and every application, based on the project and the application name and the configured openshift_master_default_subdomain in the ansible hosts file.

This means, you need to find a way to let your macOS host direct any access to this subdomain to our router VM on 192.168.56.10 (which then redirects the requests to OpenShift on 192.168.56.11).

How to do that?

We already know a nice way, we already have configured it on the router VM. It’s called dnsmasq and is also available on macOS via homebrew.

https://brew.sh/index.html

$ brew install dnsmasq

Use your editor to change the configuration file:

$ vim /usr/local/etc/dnsmasq.conf
address=/shift.ocp.lan/192.168.56.11

$ sudo brew services restart dnsmasq

Make sure dnsmasq is started on boot time:

$ sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist

Now we need to find a way to let macOS know that it should use our local dnsmasq for certain IP addresses. I have found the following blog entry:

http://asciithoughts.com/posts/2014/02/23/setting-up-a-wildcard-dns-domain-on-mac-os-x/

That’s easy. So we are now going to create the following file with the following content on /etc/resolver

$ vim /etc/resolver/ocp.lan
nameserver 127.0.0.1

And that’s it. Now every OpenShift domain gets automatically resolved which means you can use your OSX host to do your coding and your two RHEL 7 based VMs are doing the work behind the scenes.

Thank you!