Usage of central logging systems
Preconditions
You need access to the jumphost with your personal OpenSSH access key. Access an environment via OpenSSH or Putty
Standard Logging
We provide central logging for various logs as a systemd-based log system using syslog-ng. The logs are made available on the Jump server in a file structure. If you are using Docker at root360 Cloud Platform, please check out Understanding container logging configuration
Log types
1. Standard logs
The following logs are aggregated by default:
Nginx access.log
Nginx error.log
Apache access.log
Apache error.log
Deployment Logs
PHP-FPM
Supervisord Logs
2. Project logs
Logs from the following paths are automatically included in the central logging:
/var/log/application/
Compressed files such as * .gz or * .xz are excluded. A logrotation is automatically executed for all files with the extension .log and all registered log files (see below).
3. Manual registration of individual logs
Furthermore, it is possible to register individual logs at the central logging by script. The procedure is described in Script snippets under "check-log-registration". File system-based log system based on syslog-ng.
Access to logs in the file system
The access to the aggregated logs is realized via the JumpServer (also called natgw).
There are:
At a file /var/log/remote/YOUR-PROJECT/ENVIRONMENT structure.
It is differentiated into the different components of the environment (eg web, cron, etc.).
There are further differentiations per year and month
In addition, a distinction is made between so-called project and system logs. The latter are generally not relevant to the customer.
The logs themselves contain an indication of the day they were created and are rotated daily
The log schema is shown below
Data retention
All log files that are under control of root360 through any of the above steps 1-3 are rotated daily and stored on the source instance for only 7 days per log file. After transferring to the central logging system, all log files are rotated (including masking of IP addresses) daily and stored for 90 days by default on the JumpServer (also known as natgw). After expiry of these retention periods, the log files are deleted.
Rotation process in central logging system on the Jumpserver
The rotation of the logfiles by default consists of the following steps:
masking of IPv6 addresses (we overwrite the last 4 Bytes/32bit)
masking of IPv4 addresses (we overwrite the last 2 Bytes/16bit)
compression using XZ format
renaming to <filename>.<rotation-count>.xz (e.g. apache2-access.log.1.xz)
Log scheme
' Month Day Time (UTC) Server name Log name: "Technology-specific log entry"'
Example
May 4 10:16:18 some-server-i-8d701531 nginx-access: - - [04 / May / 2016: 10: 16: 17 +0000] "HEAD ...
Log analysis examples
Get the number of deliveries of a URL *:
### NGinx
xzgrep URL /var/log/remote/PROJECT/ENVIRONMENT/ROLE/YEAR/MONTH/project/nginx-access-DAY.log.1.xz -c
### Apache
xzgrep URL /var/log/remote/PROJECT/ENVIRONMENT/ROLE/YEAR/MONTH/project/apache2-access-DAY.log.1.xz -c
Get the status codes and their number for a URL *:
Determine the delivery times for a URL *:
Determine the size of responses to a URL *:
The capital letter must be replaced accordingly.
Advanced Logging Kibana
In a addition to our default log system we provide the option to apply for an advanced logging system. The setup is based on Kibana and Elasticsearch.
Log types
1. Standard logs
The following logs are aggregated by default:
Nginx access.log
Nginx error.log
Apache access.log
Apache error.log
Deployment Logs
PHP-FPM
Supervisord Logs
2. Project logs
Logs from the following paths are automatically included in the central logging:
Compressed files such as * .gz or * .xz are excluded. A logrotation is automatically executed for all files with the extension .log and all registered log files (see below).
3. Manual registration of individual logs
Furthermore, it is possible to register individual logs at the central logging by script. The procedure is described in Script Snippets under "check-log-registration".
Currently this does not support multi-line log files.
4. Log splitting
Both Nginx and Apache access.log files are split by default into their respective parts. This will allow you to sort by reponse codes or request protocol for example.
In case you want this to be extended please contact our Service Team, you can do so by sending an E-Mail to service@root360.de or by creating a ticket in the ticket system.
Data retention
All log files that are under control of root360 through any of the above steps 1-3 are rotated daily and stored on the source instance for only 7 days per log file. After transferring to the central logging system, all log files are rotated daily and stored for 90 days by default in Elasticsearch. After expiry of these retention periods, the log files are deleted via Elasticsearch curator.
Rotation in central logging system in Elasticsearch
The rotation of the logfiles by default consists of the following steps:
masking of IPv6 addresses (we overwrite the last 4 Bytes/32bit)
masking of IPv4 addresses (we overwrite the last 2 Bytes/16bit)
How to access your log files
Since all logs are stored in Elasticsearch, you can simply follow our instruction on .
Requesting root360 Advanced Logging
If you want to request root360 Advanced Logging, you can do so by sending an E-Mail to service@root360.de or by creating a ticket in our ticket system.
Required information
Required Information | Explanation | Options |
---|---|---|
Data retention | Amount of days that a masked log is stored. | default: 90 days |
Pricing
For detailed AWS pricing see https://aws.amazon.com/elasticsearch-service/pricing/?nc1=h_ls
For Root360 Managed Service costs please contact our Service team.
Related tutorials
Related components
Expert
- 1 Preconditions
- 2 Standard Logging
- 3 Advanced Logging Kibana
- 3.1 Log types
- 3.1.1 1. Standard logs
- 3.1.2 2. Project logs
- 3.1.3 3. Manual registration of individual logs
- 3.1.4 4. Log splitting
- 3.2 Data retention
- 3.3 Rotation in central logging system in Elasticsearch
- 3.4 How to access your log files
- 3.5 Requesting root360 Advanced Logging
- 3.6 Required information
- 3.7 Pricing
- 3.1 Log types
root360 Knowledge Base - This portal is hosted by Atlassian (atlassian.com | Privacy Policy)