Advanced JBoss EAP Logging with rsyslog: Using TCP and Custom Handlers

This blog post provides a detailed look at using TCP for log forwarding, leveraging custom handlers, and managing container-specific logs. Everything is also documented in this Github repository: https://github.com/marcoklaassen/jboss-eap-rsyslog.

Centralized logging is essential for monitoring and troubleshooting in distributed enterprise systems. JBoss EAP, a robust application server platform, can integrate seamlessly with rsyslog for centralized log collection. While the default setup often uses UDP for log forwarding, leveraging TCP provides enhanced reliability in log transmission, especially in high-availability environments. Furthermore, using a custom logging handler instead of the traditional SyslogHandler offers greater flexibility and control over log management.

In this guide, we’ll walk through configuring JBoss EAP to forward logs over TCP, use a custom logging handler, and split logs from different containers into separate log files, ensuring streamlined log management across the board.

Why TCP for Logging?

Traditionally, syslog protocols use UDP (User Datagram Protocol) for log forwarding, which is lightweight and fast but doesn’t guarantee message delivery. In enterprise systems where logs are critical for compliance, debugging, or auditing, using TCP (Transmission Control Protocol) is recommended. TCP provides reliable data transmission by ensuring that logs are delivered in sequence and acknowledged by the receiving server.

Benefits of Using TCP for Syslog

• Guaranteed Delivery: TCP ensures that logs are delivered reliably, even in the event of network congestion.
• Connection-oriented: Unlike UDP, which is connectionless, TCP maintains a persistent connection, offering better control over the logging stream.
• Error Recovery: If a packet is lost during transmission, TCP automatically retransmits it, making it more reliable in complex network environments.

Step 1: Configuring rsyslog for TCP

Before configuring JBoss EAP, we need to ensure that the rsyslog server is configured to accept logs over TCP.

1. Open your rsyslog.conf configuration file:

vi /etc/rsyslog.conf

2. Enable the TCP listener by adding the following configuration:

# Load TCP module
module(load="imtcp")

# Enable TCP syslog reception on port 2514
input(type="imtcp" port="2514")Code language: PHP (php)

3. Save the file and restart the rsyslog service to apply changes:

systemctl restart rsyslog

The server is now ready to receive logs over TCP on port 2514.

Step 2: Configuring JBoss EAP to Use TCP for Syslog

JBoss EAP provides built-in support for sending logs to rsyslog but to enhance logging control and flexibility, we’ll use a custom logging handler instead of the default SyslogHandler. This approach was introduced following bug reports and feature requests in the JBoss community (e.g., Bugzilla 1011882 and WFCORE-109).

Using the Custom Syslog Handler

To leverage more control over your log configuration, we will replace the default SyslogHandler with a custom handler. This custom handler allows more flexibility in how logs are transmitted.

Here’s how to set it up:

• Execute the JBoss CLI command to add the new handler

# configure custom syslog-handler with TCP (in the custom handler you are able to set the protocol field to TCP)
/opt/eap/bin/jboss-cli.sh '/subsystem=logging/custom-handler=tcpsyslog:add( \
    module=org.jboss.logmanager, class=org.jboss.logmanager.handlers.SyslogHandler, \
    properties={ \
        appName="JBOSS_SAMPLE_APPLICATION", \
        facility="LOCAL_USE_6", \
        serverHostname="rsyslog-server-tcp.jboss-eap-test.svc.cluster.local", \
        hostname="'$(hostname)'", \
        port="2514", \
        syslogType="RFC3164", \
        protocol="TCP", \
        messageDelimiter="-", \
        useMessageDelimiter="true" \
    }, \
    formatter="%-5p [%c] (%t) %s%E%n")' --connect

/opt/eap/bin/jboss-cli.sh '/subsystem=logging/root-logger=ROOT:add-handler(name=tcpsyslog)' --connectCode language: PHP (php)

• appName: The application name to be tagged in the logs for easy identification.
• facility: Specifies the syslog facility used.
• serverHostname: The hostname of your centralized syslog server.
• hostname: the hostname of the JBoss EAP container / server.
• port: The port on which the rsyslog server is listening for TCP traffic.
• syslogType: set format of the message in this case to the original BSD format (RFC3164).
• protocol: Set to TCP to enable the reliable transmission of logs.

At this point, JBoss EAP logs will be forwarded to the rsyslog server over TCP, ensuring reliable log transmission.

Step 3: Splitting Logs from Different Containers

In containerized environments, separating logs for each container is crucial to maintain clarity and traceability.^* To achieve this, we can configure rsyslog to create individual log files for each container, based on identifiers like the hostname or application name.

1. Open the rsyslog.conf file on your centralized logging server.
2. Add the following rule to split logs based on the hostname field, which we defined earlier in JBoss EAP (hostname of the container in this case):

# log every host in a separate file
$template LogInFileByHost,"/var/log/container-%HOSTNAME%.log"
*.* -?LogInFileByHostCode language: PHP (php)

This configuration will create separate log files for each container, ensuring that logs are organized and easy to query.

3. Save the file and restart rsyslog:

systemctl restart rsyslog

Now, each container’s logs will be stored in separate files, allowing for clear differentiation and simplified analysis:

ls -al /var/log/
-rw-r--r--. 1 1000730000 root   285 Oct  8 14:40 container-10-128-1-243.jboss-rsyslog-0.jboss-eap-test.svc.cluster.local.log
-rw-r--r--. 1 1000730000 root  1140 Oct  8 14:40 container-10-128-1-244.jboss-rsyslog-1.jboss-eap-test.svc.cluster.local.logCode language: JavaScript (javascript)

* Please note that logging in cloud-native environments should not be configured at the container level according to best practice. This is usually better done via a provided cluster-level service. This article deals with a very specific use case that may make sense. However, if there is no specific reason to configure logging specifically in the JBoss container, the cloud-native logging service of the Kubernetes cluster should be used, e.g. in OpenShift: https://docs.openshift.com/container-platform/4.17/observability/overview/index.html#cluster-logging-index_observability-overview

Conclusion

Incorporating TCP for log transmission in JBoss EAP, combined with the flexibility of a custom logging handler, provides a powerful solution for centralized logging. This setup ensures reliable delivery of logs and gives you enhanced control over the logging process. By further configuring rsyslog to split logs from different containers into separate files, you can improve your log organization, making it easier to monitor and troubleshoot distributed systems.

For more details or to contribute to the project, visit the GitHub repository: jboss-eap-rsyslog.