Usage of central logging systems

Preconditions

Standard Logging

We provide central logging for various logs as a systemd-based log system using syslog-ng. The logs are made available on the Jump server in a file structure. If you are using Docker at root360 Cloud Platform, please check out 

Log types

1. Standard logs

The following logs are aggregated by default:

  • Nginx access.log

  • Nginx error.log

  • Apache access.log

  • Apache error.log

  • Deployment Logs

  • PHP-FPM

  • Supervisord Logs

2. Project logs

Logs from the following paths are automatically included in the central logging:

/var/log/application/

Compressed files such as * .gz or * .xz are excluded. A logrotation is automatically executed for all files with the extension .log and all registered log files (see below).

3. Manual registration of individual logs

Furthermore, it is possible to register individual logs at the central logging by script. The procedure is described in Script snippets under "check-log-registration". File system-based log system based on syslog-ng.

Access to logs in the file system

The access to the aggregated logs is realized via the JumpServer (also called natgw).

There are:

  • At a file /var/log/remote/YOUR-PROJECT/ENVIRONMENT structure.

  • It is differentiated into the different components of the environment (eg web, cron, etc.).

  • There are further differentiations per year and month

  • In addition, a distinction is made between so-called project and system logs. The latter are generally not relevant to the customer.

  • The logs themselves contain an indication of the day they were created and are rotated daily

  • The log schema is shown below  

Data retention

All log files that are under control of root360 through any of the above steps 1-3 are rotated daily and stored on the source instance for only 7 days per log file. After transferring to the central logging system, all log files are rotated (including masking of IP addresses) daily and stored for 90 days by default on the JumpServer (also known as natgw). After expiry of these retention periods, the log files are deleted.

Rotation process in central logging system on the Jumpserver

The rotation of the logfiles by default consists of the following steps:

  • masking of IPv6 addresses (we overwrite the last 4 Bytes/32bit)

  • masking of IPv4 addresses (we overwrite the last 2 Bytes/16bit)

  • compression using XZ format

  • renaming to <filename>.<rotation-count>.xz (e.g. apache2-access.log.1.xz)

Log scheme

' Month Day Time (UTC) Server name Log name: "Technology-specific log entry"'

Example

May 4 10:16:18 some-server-i-8d701531 nginx-access: - - [04 / May / 2016: 10: 16: 17 +0000] "HEAD ...

Log analysis examples

Get the number of deliveries of a URL *:

### NGinx xzgrep URL /var/log/remote/PROJECT/ENVIRONMENT/ROLE/YEAR/MONTH/project/nginx-access-DAY.log.1.xz -c   ### Apache xzgrep URL /var/log/remote/PROJECT/ENVIRONMENT/ROLE/YEAR/MONTH/project/apache2-access-DAY.log.1.xz -c


Get the status codes and their number for a URL *:


Determine the delivery times for a URL *:


Determine the size of responses to a URL *:

The capital letter must be replaced accordingly. 

Advanced Logging Kibana

In a addition to our default log system we provide the option to apply for an advanced logging system. The setup is based on Kibana and Elasticsearch. 

Log types

1. Standard logs

The following logs are aggregated by default:

  • Nginx access.log

  • Nginx error.log

  • Apache access.log

  • Apache error.log

  • Deployment Logs

  • PHP-FPM

  • Supervisord Logs

2. Project logs

Logs from the following paths are automatically included in the central logging:

Compressed files such as * .gz or * .xz are excluded. A logrotation is automatically executed for all files with the extension .log and all registered log files (see below).

3. Manual registration of individual logs

Furthermore, it is possible to register individual logs at the central logging by script. The procedure is described in   under "check-log-registration".

Currently this does not support multi-line log files.

4. Log splitting

Both Nginx and Apache access.log files are split by default into their respective parts. This will allow you to sort by reponse codes or request protocol for example.

In case you want this to be extended please contact our Service Team, you can do so by sending an E-Mail to service@root360.de or by creating a ticket in the ticket system.

Data retention

All log files that are under control of root360 through any of the above steps 1-3 are rotated daily and stored on the source instance for only 7 days per log file. After transferring to the central logging system, all log files are rotated daily and stored for 90 days by default in Elasticsearch. After expiry of these retention periods, the log files are deleted via Elasticsearch curator.

Rotation in central logging system in Elasticsearch

The rotation of the logfiles by default consists of the following steps:

  • masking of IPv6 addresses (we overwrite the last 4 Bytes/32bit)

  • masking of IPv4 addresses (we overwrite the last 2 Bytes/16bit)

How to access your log files

Since all logs are stored in Elasticsearch, you can simply follow our instruction on  .

Requesting root360 Advanced Logging

If you want to request root360 Advanced Logging, you can do so by sending an E-Mail to service@root360.de or by creating a ticket in our ticket system.

Required information

Required Information

Explanation

Options

Data retention

Amount of days that a masked log is stored.

default: 90 days

Pricing

For detailed AWS pricing see https://aws.amazon.com/elasticsearch-service/pricing/?nc1=h_ls

For Root360 Managed Service costs please contact our Service team.

 

Related tutorials

Related components

 

 


root360 Knowledge Base - This portal is hosted by Atlassian (atlassian.com | Privacy Policy)