Syslog-ng

Syslog-ng is a free and open-source implementation of the Syslog protocol, an enhanced log daemon, supporting a wide range of input and output methods: Syslog, unstructured text, queueing, SQL & NoSQL

Installation of Syslog-ng

To install Syslog-ng on your system, please refer to the official git repo given below

GitHub - syslog-ng/syslog-ng: syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.GitHub

Configuration of Syslog-ng

Apica Ascent supports data ingestion from Syslog-ng. The following instructions describe the steps for configuring log forwarding from Syslog-ng to Apica Ascent by modifying the Syslog-ng configuration file.

Syslog-ng supports both TCP and UDP protocols, Apica Ascent only supports ingestion of logs over TCP protocol to ensure packets are not lost or dropped. TCP relies on acknowledgments from the receiver to make sure the packet is delivered.

Apica Ascent hosts the syslog protocol at port 514. The ports are configurable and can be changed if needed. Additional port for TLS is available at 7514

INPUT

The below configuration typically sends all system and internal logs, such as sshd, systemd, NetworkManager

@version: 3.36
@include "scl.conf"

options {
    time-reap(30);
    mark-freq(10);
    keep-hostname(yes);
    chain-hostnames(no);
};

source s_local {
       system();
       internal();
};

The below configuration accepts system logs from /dev/log (from applications or forwarded by systemd) and writes everything to a single file.

@version: 3.36
@include "scl.conf"

log {
	source { system(); };
	destination { file("/var/log/syslog"); };
};

OUTPUT

Apica Ascent supports multiple way to ingest data via standard interfaces. With syslog-ng you can use the built-in syslog, http output destinations to push data to Apica Ascent. We however recommend using Apica Ascent's python destination for syslog-ng driver as it provides most capabilities and control over pushing you data at scale.

Python syslog-ng destination for Apica Ascent

The syslog-ng python destination driver for Apica Ascent is avalable as a python package and can be installed via pip. To enable the python destination support, first install the python destination support for syslog-ng. Below is an example of how to do this on an ubuntu system.

apt-get install syslog-ng-mod-python

You can refer to the syslog-ng website for other operating systems. You can now proceed to install the Apica Ascent driver next. This assumes you have python3 installed. More details on the Apica Ascent driver can be found at https://pypi.org/project/logiqaidstsyslogng/

pip install --upgrade logiqaidstsyslogng

or

pip3 install --upgrade logiqaidstsyslogng

Checking Python2/3 linkage for syslog-ng

Note you may need to use pip3 vs pip to install depending on what the syslog-ng python linkage is. A quick way to test your python linkage is to create a dummy syslog-ng config file with a python destination. See below

@version: 3.38

source s_dummy {
};

python {
import sys
class VersionTest(object):
    def init(self, options):
        print(sys.version)
        return True
    def send(self, msg):
        return True
};

destination d_test {
    python(
        class("VersionTest")
    );
};

log { source(s_dummy); destination(d_test); };

Run syslog-ng with the above dummy config, and you should see the version of python that it loads

#syslog-ng -f ./dummy-syslog-ng.conf --foreground
3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0]

# syslog-ng -f ./dummy-syslog-ng.conf --foreground
2.7.17 (default, Jul  1 2022, 15:56:32)
[GCC 7.5.0]

Creating a logiq.conf for the driver

A logiq.conf is required that provides details of the LOGIA.AI instance and how to connect to it. Please note down the path where you save the config as we will need it to configure the destination in syslog-ng.conf. We recommend you store it in /etc/syslog-ng/logiq.conf along with rest of the syslog-ng configruation files.

logiq.conf

[logiq]
# Host or IP for LOGIQ.AI
host = my-logiq-cluster.example.com
# Protocol https
protocol = https
# API Ingest token
key = <JWT Token>
# destination reachability timeout in seconds
timeout = 5

Creating a syslog-ng destination for Apica Ascent

You are now ready to update your syslog-ng conf to add a Apica Ascent python destination

syslog-ng.conf

destination d_logiq {
    python(
        batch-timeout(500)
        batch-lines(400)
        class("logiqaidstsyslogng.LogDestination")
        value-pairs(
          key(ISODATE)
          scope(rfc5424 nv-pairs)
        )
        options(config "/etc/syslog-ng/logiq.conf")
        options(workers-batch 8)
        options(worker-batch-lines 25)
        options(loglevel WARN)
    );
};

# Connect the source to the logiq destination
log { source(s_local); destination(d_logiq); };

Apica Ascent only works with RFC3339/ISO8601 time stamps so the python destination must add the ISODATE key which will be sent as timestamp.

The scope for value-pairs also needs to contain rfc5424 so we can pull out all the standard syslog fields such as host, application, pid, sdata etc.

The python destination provides some useful options as described below

Name	Values	Default	Notes
batch-timeout	e.g. 500 (milliseconds)	NA	Refer to syslog-ng documentation. This allows for batching logs
batch-lines	e.g. 400	NA	Refer to syslog-ng documentation. This limits the max batch size allowed before python plugin calls a flush
class	logiqaidstsyslogng.LogDestination
	NA	Specifies the Apica Ascent python driver class to load

The Apica Ascent driver provides various options as described below

Option Name	Values	Default	Notes
config	e.g. /etc/syslog-ng/logiq.conf	None	Location of logiq.conf file
workers-batch	e.g. 8/16	1	Number of workers, more workers allow more parallelism when pushing to a loadbalanced LOGIQ.AI endpoint
worker-batch-lines	e.g. 25	25	How many maximum log events are batched per worker
loglevel	e.g. INFO/WARN/ERROR/DEBUG	INFO	Debug level. Logs are sent to /var/log/logiqaidstsyslogns
debug	true/false	false	Prints additional debug including log event to the log file
namespace-key	e.g. HOST	Not set	Which key from log event maps to namespace
application-key	e.g. PROGRAM	Not set	Which key from log event maps to application
cluster-key	e.g. HOST	Not set	Which key from log event maps to cluster/group identifier

Organizing data in Apica Ascent

Data in the Apica Ascent gets organized as flows. A flow consists of a Namespace, an application name and one or more subflows or ProcId's. This allows mapping most legacy and cloud native environments in Apica with minimal configuration

Please set namespace, application and cluster_id mappings in syslogn-ng.conf. If this is not configured, default mappings will be used.

Additional output methods

Additional methods to push data via syslog-ng are documented below but are not recommended for production and scale use.

OUTPUT (non-TLS)

The below configuration describes how logs are sent to Apica Ascent over non-TLS protocol using syslog() driver

destination d_network {
    syslog("<logiq-endpoint>" 
    transport("tcp") 
    port(514));
};


log {
    source(s_local); 
    destination(d_network);
};

OUTPUT (TLS)

The below configuration forwards logs over TLS to Apica Ascent over non-TLS protocol using syslog() driver

destination d_syslog_tls {
    syslog("<logiq-endpoint>"
        transport("tls")
        port(7514)
        tls(peer-verify(required-trusted)
            ca-dir('/opt/syslog-ng/etc/syslog-ng/keys/ca.d/')
            key-file('/opt/syslog-ng/etc/syslog-ng/keys/client_key.pem')
            cert-file('/opt/syslog-ng/etc/syslog-ng/keys/client_certificate.pem')
        )
    );
};

Example (non-TLS)

The below configuration shows the full configuration of how Syslog-ng forwards logs over non-TLS protocol using syslog() driver

@version: 3.36
@include "scl.conf"

options {
    time-reap(30);
    mark-freq(10);
    keep-hostname(yes);
    chain-hostnames(no);
};

source s_local {
       system();
       internal();
};

destination d_network {
    syslog("<logiq-endpoint>" 
    transport("tcp") 
    port(514));
};


log {
    source(s_local); 
    destination(d_network);
};

OUTPUT http/https

NOTE: The payload in ${MSG} and other fields that come from syslog-ng variable need to be escaped properly, otherwise they will be rejeced with a 400 error.

Below example shows how to construct a body

With http/https publish, the following are mandatory

• Ingest token

• namespace

• @timestamp

• application_name

• cluster_id

• hostname

In addition to the fields above, arbitrary json attributes can be provided. Nested JSON will automatically get flattened.

Example config below. Please modify relevant fields such as cluster_id, namespace, application_name before sending data

destination d_logiq {
    http(url("https://<logiq dns or ip>/v1/json")
        method("POST")
        peer-verify(no)
        headers("Content-Type: application/json")
        headers("Authorization: Bearer <ingest token goes here>")
        body("{ \"@timestamp\": \"${ISODATE}\",
                \"hostname\": \"${HOST}\",
                \"namespace\": \"syslogng-test-logs\",
                \"message\": \"${MSG}\",
                \"application_name\": \"${PROGRAM}\",
                \"proc_id\": \"${PID}\",
                \"cluster_id\": \"logiq-cluster-1\",
                \"severity\": \"${LEVEL}\",
                \"sourcetype\": \"mysourcetype2\"
              }")
    );
};

For more information please refer Syslog-ng documentation