Proof of Concept: Use Ansible to manage the Advanced Intrusion Detection Environment (AIDE)

This is the follow-up article to my Introduction to the Advanced Intrusion Detection Environment (AIDE). It presents a Proof of Concept (PoC) that shows how AIDE can be remotely controlled using an Ansible role. Knowledge of the introductory article is assumed.

Basic Ansible knowledge, such as the use of Ansible roles and the execution of playbooks, is also a prerequisite. If you are not familiar with Ansible, please refer to the official documentation.

Which tasks can be performed with Ansible?

Ensuring the aide package is installed on the target systems
The optional generation and distribution of the configuration file aide.conf
Initialization of the AIDE database
Central storage of the AIDE databases of all managed hosts on the Ansible Control Node (ACN)
Performing integrity checks
Updating the AIDE databases and storing them again on the ACN

Storing the AIDE databases and configuration files on the ACN protects them against changes on a compromised host. The files on the ACN itself are protected using Unix file-system permissions only. But if your ACN is compromised, you have a completely different problem than worrying about AIDE anyway.

My Lab Environment

My lab environment for this PoC consists of four hosts:

ansible-ctrl (RHEL 8 with the package ansible-core installed)
rhel7
rhel8
rhel9

The ACN can connect to the target systems (rhel{7,8,9}) via SSH and execute program code there with elevated rights.

The Ansible role I have developed for this PoC can be found at URL: https://github.com/Tronde/aide

Description of the Ansible role

This role is not idempotent. It calls the aide program on the target systems with various options and processes their output. To do this, the role uses the ansible.builtin.command module.

The role is controlled via Ansible Tags. If the role is executed by a playbook without specifying tags, no changes are made to the target systems.

The following code block shows an example playbook for calling the role. The tags and the variable aide_db_fetch_dir are explained below.

# SPDX-License-Identifier: MIT
---
- name: Example aide role invocation
  hosts: targets
  tasks:
    - name: Include role aide
      tags:
        - install
        - generate_config
        - init
        - check
        - update
      vars:
        aide_db_fetch_dir: files
      ansible.builtin.include_role:
        name: aideCode language: CSS (css)

install – If this tag is specified, the role ensures that the aide package is installed on the target systems
generate_config – Generates the file /etc/aide.conf using templates/aide.conf.j2; the template must be adapted to individual requirements; see next section for details
init – This initializes the AIDE database, which serves as a reference database for future checks
check – Performs an integrity check using the reference database
update – Performs an integrity check and creates a new AIDE database which will be used as a reference in the future

The variable aide_db_fetch_dir specifies the directory where the AIDE databases of the remote nodes will be stored. The default value expects the directory files to be present in the same directory where the playbook is stored.

In this directory, subdirectories are created for each host in which the AIDE database of the managed systems are stored. If a different storage location is to be used, the value of this variable must be adjusted accordingly. The AIDE databases are fetched from the managed systems using the Ansible module ansible.builtin.fetch.

The different use cases

In this section, I describe the five use cases for the PoC. All use cases were tested against RHEL 7, RHEL 8 and RHEL 9. For this article however, I will limit myself to tests against RHEL 9 only in order to improve the clarity of the output.

The playbook from the previous section is used with different tags for different use cases.

Use case 1: Installation of AIDE

In order to use AIDE, it must first be installed. This is ensured with the following playbook call:

[root@ansible-ctrl ansible]# ansible-playbook aide.yml --tags install

PLAY [Example aide role invocation] ********************************************

TASK [Gathering Facts] *********************************************************
ok: [rhel9]

TASK [Include role aide] *******************************************************

TASK [aide : Ensure required packages are installed] ***************************
changed: [rhel9]

PLAY RECAP *********************************************************************
rhel9                      : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0Code language: PHP (php)

The role is idempotent for this use case. No further changes are made to the system during a second execution:

[root@ansible-ctrl ansible]# ansible-playbook aide.yml --tags install

PLAY [Example aide role invocation] ********************************************

TASK [Gathering Facts] *********************************************************
ok: [rhel9]

TASK [Include role aide] *******************************************************

TASK [aide : Ensure required packages are installed] ***************************
ok: [rhel9]

PLAY RECAP *********************************************************************
rhel9                      : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0Code language: PHP (php)

Use case 2: Generate /etc/aide.conf

The file templates/aide.conf.j2 is delivered together with the role. It is the default configuration file from a RHEL 9 installation. In addition to the default configuration file, the path /root/.ansible* has been excluded from monitoring in order to avoid false positives.

This file must be adapted to individual requirements and should be transformed into a real Jinja2 template using variables. If you need help with the templating and Jinja2, you can find an introduction in the Ansible documentation.

The configuration file is then deployed as follows:

[root@ansible-ctrl ansible]# ansible-playbook aide.yml --tags generate_config

PLAY [Example aide role invocation] ********************************************

TASK [Gathering Facts] *********************************************************
ok: [rhel9]

TASK [Include role aide] *******************************************************

TASK [aide : Generate /etc/aide.conf] ******************************************
changed: [rhel9]

PLAY RECAP *********************************************************************
rhel9                      : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0Code language: PHP (php)

This part of the role is idempotent, too.

If this step is omitted, the default configuration file that was installed along with the aide package is used in all subsequent use cases.

Use case 3: Initialization of the AIDE database

In order to perform integrity checks, the AIDE database must first be initialized. This is done with the following call:

[root@ansible-ctrl ansible]# ansible-playbook aide.yml --tags init

PLAY [Example aide role invocation] ********************************************

TASK [Gathering Facts] *********************************************************
ok: [rhel9]

TASK [Include role aide] *******************************************************

TASK [aide : Initialize AIDE database] *****************************************
changed: [rhel9]

TASK [aide : Fetch AIDE database] **********************************************
changed: [rhel9]

TASK [aide : Remove remote AIDE database file] *********************************
changed: [rhel9]

PLAY RECAP *********************************************************************
rhel9                      : ok=4    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0Code language: PHP (php)

Once the AIDE database has been initialized, it is copied to the ACN and removed from the managed systems. The reason for this is that the ACN is a very well-secured system and the databases are best protected against compromise here.

If the default value of the variable aide_db_fetch_dir is used, the AIDE database can now be found in the path files/rhel9/var/lib/aide/aide.db.new.gz. Here, rhel9 in the path corresponds to the inventory_hostname of the respective remote system.

This part of the role is not idempotent. If the playbook is executed again, a new AIDE database is created, downloaded to the ACN and deleted from the remote system.

Use case 4: Integrity Check

The following code block shows the playbook call for the integrity check. Here, the AIDE database is first copied to the remote system and then an AIDE check is performed. As no changes were detected in the following example, the task “[aide : Check against AIDE reference database]” has the status “ok”.

[root@ansible-ctrl ansible]# ansible-playbook aide.yml --tags check

PLAY [Example aide role invocation] ********************************************

TASK [Gathering Facts] *********************************************************
ok: [rhel9]

TASK [Include role aide] *******************************************************

TASK [aide : Copy AIDE reference database to remote] ***************************
changed: [rhel9]

TASK [aide : Check against AIDE reference database] ****************************
ok: [rhel9]

PLAY RECAP *********************************************************************
rhel9                      : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0Code language: PHP (php)

This part of the role is not idempotent. A new integrity check is performed each time you call the playbook with this tag.

I have manipulated the /etc/hosts file on the target system to also show the case when a change was detected.

At the beginning of the following output you can see that the task “[aide : Copy AIDE reference database to remote]” has the status “ok”. Ansible has recognized that the AIDE database already exists in an unchanged state on the remote system and has therefore not transferred it again. However, the task “[aide : Check against AIDE reference database]” now fails (status: “fatal”) because changes have been detected. The admittedly somewhat cluttered output contains the message that the /etc/hosts file has been changed.

[root@ansible-ctrl ansible]# ansible-playbook aide.yml --tags check

PLAY [Example aide role invocation] ********************************************

TASK [Gathering Facts] *********************************************************
ok: [rhel9]

TASK [Include role aide] *******************************************************

TASK [aide : Copy AIDE reference database to remote] ***************************
ok: [rhel9]

TASK [aide : Check against AIDE reference database] ****************************
fatal: [rhel9]: FAILED! => {"changed": true, "cmd": ["aide", "--check"], "delta": "0:00:27.177397", "end": "2024-03-29 05:16:50.682795", "msg": "non-zero return code", "rc": 4, "start": "2024-03-29 05:16:23.505398", "stderr": "", "stderr_lines": [], "stdout": "Start timestamp: 2024-03-29 05:16:23 -0400 (AIDE 0.16)\nAIDE found differences between database and filesystem!!\n\nSummary:\n  Total number of entries:\t45541\n  Added entries:\t\t0\n  Removed entries:\t\t0\n  Changed entries:\t\t1\n\n---------------------------------------------------\nChanged entries:\n---------------------------------------------------\n\nf   ...    .C... : /etc/hosts\n\n---------------------------------------------------\nDetailed information about changes:\n---------------------------------------------------\n\nFile: /etc/hosts\n  SHA512   : YobgpcvAMPey0QX1lK4K+5EFySF1xrB/ | 7nIivvNa5ozfhOqSFLmPIiu6g04Wbx1n\n             9FRzTCPNC93+13Y5/lm2inC4x4rydlf2 | iGNf0/QTgFjaMGug8HywxTiO2PREZRNS\n             EcvonCf3pHuXj6lEmAjBnw==         | 3qNEi4Qm6an5inSY72sjfA==\n\n\n---------------------------------------------------\nThe attributes of the (uncompressed) database(s):\n---------------------------------------------------\n\n/var/lib/aide/aide.db.gz\n  MD5      : gMgRyMOExVAdOAvdgt4QDA==\n  SHA1     : w7tmPKNvRYggY/JZ5wv+7ZdcSZM=\n  RMD160   : CO0pK5tfg66MaO17YB8eaRuyyMw=\n  TIGER    : n8UbZJNt9gL672+pR9IPjoyhpAsUJ46O\n  SHA256   : k8UHnv2CK4zYrfZN+bDp6SCcLkx21px6\n             GNZlwySPKcY=\n  SHA512   : DFw5wlBoJQOBCrs0ulvVxaMvoQk/oBEQ\n             TkOmhfHAdevUWNAgCJ0KH0q26LsynEMj\n             MWQpsGf7v12iACc4SP9ANA==\n\n\nEnd timestamp: 2024-03-29 05:16:50 -0400 (run time: 0m 27s)", "stdout_lines": ["Start timestamp: 2024-03-29 05:16:23 -0400 (AIDE 0.16)", "AIDE found differences between database and filesystem!!", "", "Summary:", "  Total number of entries:\t45541", "  Added entries:\t\t0", "  Removed entries:\t\t0", "  Changed entries:\t\t1", "", "---------------------------------------------------", "Changed entries:", "---------------------------------------------------", "", "f   ...    .C... : /etc/hosts", "", "---------------------------------------------------", "Detailed information about changes:", "---------------------------------------------------", "", "File: /etc/hosts", "  SHA512   : YobgpcvAMPey0QX1lK4K+5EFySF1xrB/ | 7nIivvNa5ozfhOqSFLmPIiu6g04Wbx1n", "             9FRzTCPNC93+13Y5/lm2inC4x4rydlf2 | iGNf0/QTgFjaMGug8HywxTiO2PREZRNS", "             EcvonCf3pHuXj6lEmAjBnw==         | 3qNEi4Qm6an5inSY72sjfA==", "", "", "---------------------------------------------------", "The attributes of the (uncompressed) database(s):", "---------------------------------------------------", "", "/var/lib/aide/aide.db.gz", "  MD5      : gMgRyMOExVAdOAvdgt4QDA==", "  SHA1     : w7tmPKNvRYggY/JZ5wv+7ZdcSZM=", "  RMD160   : CO0pK5tfg66MaO17YB8eaRuyyMw=", "  TIGER    : n8UbZJNt9gL672+pR9IPjoyhpAsUJ46O", "  SHA256   : k8UHnv2CK4zYrfZN+bDp6SCcLkx21px6", "             GNZlwySPKcY=", "  SHA512   : DFw5wlBoJQOBCrs0ulvVxaMvoQk/oBEQ", "             TkOmhfHAdevUWNAgCJ0KH0q26LsynEMj", "             MWQpsGf7v12iACc4SP9ANA==", "", "", "End timestamp: 2024-03-29 05:16:50 -0400 (run time: 0m 27s)"]}

PLAY RECAP *********************************************************************
rhel9                      : ok=2    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0Code language: PHP (php)

At this point it was shown that both unchanged systems and systems with changes are recognized and reported. Of course, nobody has to observe the standard output. Instead, logging can be configured for Ansible outputs.

Use case 5: Update the AIDE database

This use case assumes that changes made are legitimate and should be included in the AIDE reference database. This is done as follows:

[root@ansible-ctrl ansible]# ansible-playbook aide.yml --tags update

PLAY [Example aide role invocation] ********************************************

TASK [Gathering Facts] *********************************************************
ok: [rhel9]

TASK [Include role aide] *******************************************************

TASK [aide : Update AIDE database] *********************************************
changed: [rhel9]

TASK [aide : Fetch AIDE database] **********************************************
changed: [rhel9]

TASK [aide : Remove remote AIDE database file] *********************************
changed: [rhel9]

PLAY RECAP *********************************************************************
rhel9                      : ok=4    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0Code language: PHP (php)

Once the reference database has been updated, it is copied back to the ACN and removed from the target system.

The following example shows that the AIDE check is now completed on the target system without errors:

[root@ansible-ctrl ansible]# ansible-playbook aide.yml --tags check

PLAY [Example aide role invocation] ********************************************

TASK [Gathering Facts] *********************************************************
ok: [rhel9]

TASK [Include role aide] *******************************************************

TASK [aide : Copy AIDE reference database to remote] ***************************
changed: [rhel9]

TASK [aide : Check against AIDE reference database] ****************************
ok: [rhel9]

PLAY RECAP *********************************************************************
rhel9                      : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0Code language: PHP (php)

Ansible has recognized that the AIDE database on the remote host does not match the current reference database and has therefore transferred the latter to the target system. The check ends with the status “ok”. The system corresponds to the desired state.

Summary

The above proof of concept has shown that AIDE can be used and managed remotely with an Ansible role. The AIDE database and configuration file are stored separately from the managed systems and are therefore protected against changes if the remote systems are compromised. If necessary, when Ansible detects discrepancies between the actual and desired state, these files are transferred to the remote systems.

The greatest amount of work is involved in creating one or more AIDE configuration files that optimally match your own environment and, if possible, avoids generating false positives. However, this effort is also required if AIDE is used without Ansible.

This PoC has not taken into account how to monitor and analyze the Ansible log files. It is of course useless to you only log the output of the playbooks without analyzing those logs in order to trigger corresponding alarms in your monitoring or auditing systems. This last aspect is left to the users to practice on their own ;-)