Culture Change in Automation: Declarative vs. Process Description

November 29, 2021

You might have guessed it: I’m an old IT guy. I grew up, when x86 was still the 8086 processor (and the TLA (Three Letter Acronym) “x86” hadn’t even been invented to describe the basis for a whole industry), so the topic of this article might seem a bit “far fetched” or reversely “state of the art”, depending on where you come from. Without any further ado, I’d like to reflect here about something that I still see as surprises for some (even many) of my customers, and therefore I believe it to be important enough to spend some writing and reading cycles here.

As you might have guessed from my last articles or my Bio, I’m currently covering two products from Red Hat as a Domain Solution Architect. Those are the Ansible Automation Platform (AAP) and the Red Hat Advanced Cluster Management for Kubernetes (RHACM).

Both these products are meant to help in managing Infrastructure. From the Bare Metal up to the whole Application Stack. And are doing automation.

Automation – The early days

When we started with automation back in the last millennium, we wrote scripts as a sequence of tasks to be executed. There also were some options to do if-then-else things, depending on the results of earlier steps in the scripts, but that basically covered it. It was very basic, and mostly a sequence of commands.

Automation – Evolution

Later we saw tools that allowed us to rely on some specific KnowHow for some specific systems, so that not everything needed to be written in scripts any longer. Also the capability to remotely execute these scripts was added. Mostly in relying on some sort of demon that needed to be run on the managed system, which received commands/requests from the automation or management tool. Still, these systems relied on some “procedural” or “sequential” approach.

Digression – Software Development

When we look at a higher level, aka the software development industry, we can see, that that also mainly started in procedural thinking, leading to software that was initially written in functions and procedures, like for example in programming languages like C. The main change appeared with object oriented programming (or even with some early AI programming languages such as Prolog), but still the objects did execute procedures or functions when they were triggered.

Change

Some entries into changing this specific behaviour appeared in the last decade, so I’m going to just describe what I saw and still see in some admin’s behaviours and the tools they use, and how that might lead to change, that would need to be more broadly seen in order to be understood and adopted. Many of these changes were triggered by concepts like “Software Defined Data Center” (SDDC) or “Infrastructure as Code” (IaC).

Let’s make it quick:

If you want to create a SDDC, you just want to write down how many servers in how many racks with how many operating systems instances etc you need. So, basically you write down a bill of material (BoM), but do not want to also create a building recipe.

This is the first “mental” approach to what I’m after.. Here in this intermediate state, these BoM things were for example referred to as “Reference Architectures” or “Building Blocks” (some even used the term “Pod”).

For the next and currently most important change I did not find a good analogy in our “real world”, so I need to quickly describe it, so that I can then get back to how we can approach it with the above mentioned tools.

Nowadays in IT in some places the idea is that you paint a picture of the final thing (or even make a photo), and some magic tool “just builds it” for you.

And, we need to be sure, that this will not lead to what we can see in this picture:

https://pmac-agpc.ca/sites/default/files/Tree.jpg

So, let’s look at that last picture in that narrative, the tire hanging from a branch of a tree.

In the old days you would write:

  • Grow a tree
  • Grow a branch
  • Tie a rope to the branch
  • Make the rope so long, that it does not reach the ground
  • Tie a tire to the rope so that the tire does not touch the ground

That is a procedural approach.

Today we simple state:

  • I want a tire hanging free above the ground from a branch of a tree.

So, how can we achieve all that in IT?

With a declarative description approach.

In the area of Systems, that might be a Playbook in Ansible, where there just is a description of the desired end state (and the so-called modules or even larger collections take care of achieving that desired end state).

In the area of Kubernetes it might be a policy in RHACM that simply states that App XYZ needs to be available on clusters ABC.

When I explain this to my customers, some finally understand what a configuration policy might be used for, or how all the ideas behind CI/CD for deploying applications might also be addressed via a policy, and so creating all procedures for a full CI/CD pipeline might be to far-fetched and complex and could much more easily be solved by a simple policy or compliance engine.

What I find surprising is that it seems a sometimes very difficult change of approach for traditional systems administrators, as they still tend to think in more procedural “tasks”. Opening their eyes to the options of a declarative target description allows them to become more efficient. Be it on the systems side via Ansible or on the platform side via policies.

What do you think?