Aviatrix Controller and Gateway Logging

In the CoPilot > Settings > Configuration > Logging Services page, you can configure the forwarding of logs from the Aviatrix platform to the log servers of well known log management systems.

Overview of Controller and Gateway Logging

The Aviatrix Controller and all of its managed gateways can be configured to forward their logs to well known log management systems. The Controller and all the managed gateways will forward the logs directly to the logging server. As such, the Controller and each managed gateway needs network connectivity to the logging server.

Aviatrix supports using Remote Syslog (rsyslog) for forwarding log messages. Remote Syslog as the log forwarder is both efficient and the industry standard. Most log collectors support rsyslog as a log forwarder.

Log data collected from Aviatrix Controller and all the managed gateways can be forwarded by Remote Syslog to your log server, such as to:

  • Datadog

  • Netflow

  • AWS CloudWatch

In addition to standard information on syslog, Aviatrix logs also provide informational insights into user VPN connections, VPN user TCP sessions, security rule violation statistics, gateway stats and FQDN filter violations.

The chosen log management system can sift through the Aviatrix logs to get meaningful trend charts that can help monitor the network connectivity and user VPN sessions. See Aviatrix Log Formats for a list of useful Aviatrix logs which can be parsed on the log management system to display relevant analytics of data collected from Aviatrix Controller and gateways.

The process the Gateways and Controller use for exporting their log files is as follows:

The process the Gateways and Controller use for exporting their log files is different in Controller software versions earlier than 7.0.1726. In addition, the log file prefix previously included the log filename and log timestamp. For details, see Field Notice 42.

Aviatrix gateways and Controller stream the log lines being written to the syslog and auth.log files. When you use the default rsyslog server configuration shown in Remote Syslog Configuration on Controller, the logs streamed from the Controller and gateways have multiple files. Each file is named with the application that generated the log. For example, all logs generated by the avx-gw-state-sync application are re-directed to a file named avx-gw-state-sync" on the log server.

The log format is shown below. Your syslog collectors and any related automation for ingesting logs must accept the log format.

Format: [Timestamp] GW-[Name of Gateway]-[Public IP of Gateway] [Name of Application generating log][Application Process ID]: [Log message]

Example of log format:

Mar 23 19:17:50 GW-UdpGateway-50.17.41.173 avx-gw-state-sync[11249]: warn#011gateway_launcher/gateway_launcher.go:212#011daemon exited

Prefix of log: [Timestamp] GW-[Name of Gateway]-[Public IP of Gateway]

Example prefix: Mar 23 19:17:50 GW-gg-aws-usw2-s127-35.162.124.66

Enabling Aviatrix Controller Logging

To enable logging at the Aviatrix Controller, you can use the Controller UI or the CoPilot UI.

In Controller UI, go to Controller > Settings > Logging page. After setting the logging configuration in this page, you must click Enable for the configuration to be saved. After logging is enabled, both the Controller and all gateways will forward logs directly to the logging server.

In CoPilot UI, go to CoPilot > Settings > Configuration > Logging Services page.

Configuring the Aviatrix Datadog Agent

If you use Datadog, the Aviatrix platform offers a Datadog agent for sending system metrics from Aviatrix gateways to your configured Datadog instance.

Before configuring the Aviatrix Datadog agent, collect the following information:

  • Datadog account

  • Datadog API key

You can configure the Aviatrix Datadog agent in CoPilot > Settings > Configuration > Logging Services > Datadog Agent.

Configuring the Aviatrix CloudWatch Agent

If you use CloudWatch, the Aviatrix platform offers a CloudWatch agent for sending syslog from Aviatrix Controller and Aviatrix gateways to your configured AWS CloudWatch instance.

Before configuring the Aviatrix CloudWatch agent, collect the following information:

  • CloudWatch role ARN

  • AWS Cloud Type

  • Log group name

You can configure the Aviatrix CloudWatch agent in CoPilot > Settings > Configuration > Logging Services > CloudWatch Agent.

Configuring the Aviatrix Netflow Agent

If you want to forward NetFlow data collected on Aviatrix gateways, the Aviatrix platform offers a Netflow agent for sending Netflow data from Aviatrix gateways to your designated service point.

Before configuring the Aviatrix Netflow agent, collect the following information:

  • IP address of the destination NetFlow service

  • Port number of the destination NetFlow service

You can configure the Aviatrix Netflow agent in CoPilot > Settings > Configuration > Logging Services > NetFlow Agent.

If you want to use the FlowIQ feature in Aviatrix CoPilot, you must configure the Aviatrix Netflow agent to forward NetFlow data to CoPilot.

Configuring the Aviatrix Remote Syslog Forwarder

If you want to forward syslog data collected on Aviatrix gateways, the Aviatrix platform offers a remote syslog forwarder for sending syslog data from Aviatrix gateways to your designated remote syslog server.

Before configuring the Aviatrix Remote Syslog forwarder, collect the following information:

  • FQDN or IP address of the remote syslog server

  • Port number of the listening port of the remote syslog server

  • Certificate Authority (CA) certificate

  • Public certificate signed by the same CA

  • Private key of the Controller that pairs with the public certificate

You can configure the Aviatrix Remote Syslog forwarder in CoPilot > Settings > Configuration > Logging Services > Remote Syslog. If using the Controller UI, see Remote Syslog Configuration on Controller (Controller UI) below.

If you use Aviatrix CoPilot, you must configure the Aviatrix Remote Syslog forwarder to forward syslog data to CoPilot. Several CoPilot features, including the CoPilot > Administration > Audit feature, rely on syslog data.

Remote Syslog Configuration on Controller (Controller UI)

On the Aviatrix Controller, go to Controller > Settings > Logging > Remote Syslog tile:

  1. In Profile Index, select a profile to edit and then click the Status button to expand the configuration pane.

  2. Server: FQDN or IP address of the remote syslog server

  3. Port: Listening port of the remote syslog server (6514 by default)

  4. CA Certificate: Certificate Authority (CA) certificate

  5. Server Public Certificate: Public certificate of the Controller signed by the same CA

  6. Server Private Key: Private key of the Controller that pairs with the public certificate

  7. Protocol: TCP or UDP (TCP by default)

  8. Optional Custom Template: Useful when forwarding to 3rd party servers like Datadog. See topic Using Rsyslog with Datadog below for an example.

A total of 10 profiles from index 0 to 9 are supported for remote syslog, while index 9 is reserved for CoPilot.

Newly deployed gateways will be added to a profile if it is the only profile enabled in the index range of 0 to 8.

If more than one profile is enabled in the range of 0 to 8, the newly deployed gateway will not be added to any profile in the range of 0 to 8. You can use the advanced options in the logging "Edit Options" window to edit the exclude and include list.

Newly deployed gateways will always be added to profile 9 which is reserved for CoPilot to monitor.

Using Rsyslog with Datadog

If you use Datadog, you can configure the Aviatrix Remote Syslog forwarder to send syslog data from Aviatrix gateways to your configured Datadog instance. Example:

  1. In CoPilot > Settings > Configuration > Logging Services > Remote Syslog, enable the service.

  2. Server: intake.logs.datadoghq.com

  3. Port: 10514

  4. Protocol: TCP

  5. For Optional Custom Template, copy the following string and replace the string DATADOG_API_KEY with your own key.

    DATADOG_API_KEY <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% - - - %msg%\\n

Remote Syslog Configuration on the Remote syslog Server

On the Remote syslog server:

  1. Install rsyslog and rsyslog-gnutls packages

  2. Create a new config file in /etc/rsyslog.d with the similar content as in the following box depends on your rsyslog version to enable tls connection. Please make sure key paths are readable by the syslog user.

  3. Make sure the output directory /var/log is writable by rsyslog user/daemon.

  4. Restart rsyslog service and check port is listening and no error in /var/log/syslog.

  5. Confirm the port is allowed in the security group / fireware for incoming traffic.

    Remote Syslog version 8 or lower:

    $ModLoad imtcp
$InputTCPServerRun 514

$DefaultNetstreamDriver gtls

#Certificate location
$DefaultNetstreamDriverCAFile /etc/cert/rsyslog-ca.pem
$DefaultNetstreamDriverCertFile /etc/cert/rsyslog-crt.pem
$DefaultNetstreamDriverKeyFile /etc/cert/rsyslog-key.pem

$InputTCPServerStreamDriverAuthMode x509/certvalid
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode

# Re-direct logs to host specific directories
$template TmplMsg, "/var/log/aviatrix/%HOSTNAME%/%PROGRAMNAME%"
*.info,mail.none,authpriv.*,cron.none ?TmplMsg
& ~

    Remote Syslog version 8 or higher:

    global(
    DefaultNetstreamDriver="gtls"
    DefaultNetstreamDriverCAFile="/etc/cert/rsyslog-ca.pem"
    DefaultNetstreamDriverCertFile="/etc/cert/rsyslog-crt.pem"
    DefaultNetstreamDriverKeyFile="/etc/cert/rsyslog-key.pem"
)
template(name="TmplMsg" type="list") {
    constant(value="/var/log/aviatrix/")
    property(name="hostname")
    constant(value="/")
    property(name="programname" SecurePath="replace")
    constant(value="")
    }
ruleset(name="remote"){
    *.info;mail.none;authpriv.*;cron.none action(type="omfile" DynaFile="TmplMsg")
}
module(
    load="imtcp"
    StreamDriver.Name="gtls"
    StreamDriver.Mode="1"
    StreamDriver.Authmode="anon"
)
input(type="imtcp" port="514" ruleset="remote")

  1. Go to /var/log/aviatrix directory

  2. Find the directory of the desired Controller or gateway.

    • Controller’s directory name is in a format of Controller-public_IP_of_controller

    • Gateway’s directory name is in a format of GW-gateway_name-public_IP_of_gateway

  3. Each controller/gateway directory should have:

    • auth.log

    • syslog

Format for CA Certificates from External Logging Servers

The Aviatrix Controller expects certificates in PEM format. Convert any certificates downloaded from your external logging server’s documentation into PEM format. Attempting to upload the wrong format may return an Exception Error.