We provide central logging for various logs as a systemd-based log system using syslog-ng. The logs are made available on the Jump server in a file structure. |
You need access to the jumphost with your personal OpenSSH access key. Access an environment via OpenSSH or Putty
We provide central logging for various logs as a systemd-based log system using syslog-ng. The logs are made available on the Jump server in a file structure. If you are using Docker at root360 Cloud Platform, please check out Understanding container logging configuration
The following logs are aggregated by default:
Nginx access.log
Nginx error.log
Apache access.log
Apache error.log
Deployment Logs
PHP-FPM
Supervisord Logs
Logs from the following paths are automatically included in the central logging:
/var/log/application/ |
Compressed files such as * .gz or * .xz are excluded. A logrotation is automatically executed for all files with the extension .log and all registered log files (see below).
Furthermore, it is possible to register individual logs at the central logging by script. The procedure is described in Script snippets under "check-log-registration". File system-based log system based on syslog-ng.
The access to the aggregated logs is realized via the JumpServer (also called natgw).
There are:
At a file /var/log/remote/YOUR-PROJECT/ENVIRONMENT structure.
It is differentiated into the different components of the environment (eg web, cron, etc.).
There are further differentiations per year and month
In addition, a distinction is made between so-called project and system logs. The latter are generally not relevant to the customer.
The logs themselves contain an indication of the day they were created and are rotated daily
The log schema is shown below
All log files that are under control of root360 through any of the above steps 1-3 are rotated daily and stored on the source instance for only 7 days per log file. After transferring to the central logging system, all log files are rotated (including masking of IP addresses) daily and stored for 90 days by default on the JumpServer (also known as natgw). After expiry of these retention periods, the log files are deleted.
The rotation of the logfiles by default consists of the following steps:
masking of IPv6 addresses (we overwrite the last 4 Bytes/32bit)
masking of IPv4 addresses (we overwrite the last 2 Bytes/16bit)
compression using XZ format
renaming to <filename>.<rotation-count>.xz (e.g. apache2-access.log.1.xz)
' Month Day Time (UTC) Server name Log name: "Technology-specific log entry"'
Example
May 4 10:16:18 some-server-i-8d701531 nginx-access: - - [04 / May / 2016: 10: 16: 17 +0000] "HEAD ... |
Get the number of deliveries of a URL *:
### NGinx xzgrep URL /var/log/remote/PROJECT/ENVIRONMENT/ROLE/YEAR/MONTH/project/nginx-access-DAY.log.1.xz -c ### Apache xzgrep URL /var/log/remote/PROJECT/ENVIRONMENT/ROLE/YEAR/MONTH/project/apache2-access-DAY.log.1.xz -c |
Get the status codes and their number for a URL *:
### NGinx xzgrep URL /var/log/remote/PROJECT/ENVIRONMENT/ROLE/YEAR/MONTH/project/nginx-access-DAY.log.1.xz | grep -oE '"[0-9] [0-9] [0-9]' | sort | uniq -c ### Apache xzgrep URL /var/log/remote/PROJECT/ENVIRONMENT/ROLE/YEAR/MONTH/project/apache2-access-DAY.log.1.xz | grep -oE '"[0-9] [0-9] [0-9]' | sort | uniq -c |
Determine the delivery times for a URL *:
### NGinx xzgrep URL /var/log/remote/PROJECT/ENVIRONMENT/ROLE/YEAR/MONTH/project/nginx-access-DAY.log.1.xz | awk '{print $ NF}' ### Apache xzgrep URL /var/log/remote/PROJECT/ENVIRONMENT/ROLE/YEAR/MONTH/project/apache2-access-DAY.log.1.xz | awk '{print $ NF}' |
Determine the size of responses to a URL *:
### NGinx xzgrep URL /var/log/remote/PROJECT/ENVIRONMENT/ROLE/YEAR/MONTH/project/nginx-access-DAY.log.1.xz | grep -oE '"[0-9] [0-9] [0-9] [0-9] [0-9] *' | cut -f3 -d '' ### Apache xzgrep URL /var/log/remote/PROJECT/ENVIRONMENT/ROLE/YEAR/MONTH/project/apache2-access-DAY.log.1.xz | grep -oE '"[0-9] [0-9] [0-9] [0-9] [0-9] *' | cut -f3 -d '' |
The capital letter must be replaced accordingly.
In a addition to our default log system we provide the option to apply for an advanced logging system. The setup is based on Kibana and Elasticsearch.
The following logs are aggregated by default:
Nginx access.log
Nginx error.log
Apache access.log
Apache error.log
Deployment Logs
PHP-FPM
Supervisord Logs
Logs from the following paths are automatically included in the central logging:
/var/log/application/ |
Compressed files such as * .gz or * .xz are excluded. A logrotation is automatically executed for all files with the extension .log and all registered log files (see below).
Furthermore, it is possible to register individual logs at the central logging by script. The procedure is described in Script Snippets under "check-log-registration".
Currently this does not support multi-line log files.
Both Nginx and Apache access.log files are split by default into their respective parts. This will allow you to sort by reponse codes or request protocol for example.
In case you want this to be extended please contact our Service Team, you can do so by sending an E-Mail to service@root360.de or by creating a ticket in the ticket system.
All log files that are under control of root360 through any of the above steps 1-3 are rotated daily and stored on the source instance for only 7 days per log file. After transferring to the central logging system, all log files are rotated daily and stored for 90 days by default in Elasticsearch. After expiry of these retention periods, the log files are deleted via Elasticsearch curator.
The rotation of the logfiles by default consists of the following steps:
masking of IPv6 addresses (we overwrite the last 4 Bytes/32bit)
masking of IPv4 addresses (we overwrite the last 2 Bytes/16bit)
Since all logs are stored in Elasticsearch, you can simply follow our instruction on How to access Kibana dashboard? .
If you want to request root360 Advanced Logging, you can do so by sending an E-Mail to service@root360.de or by creating a ticket in our ticket system.
Required Information | Explanation | Options |
---|---|---|
Data retention | Amount of days that a masked log is stored. | default: 90 days |
For detailed AWS pricing see https://aws.amazon.com/elasticsearch-service/pricing/?nc1=h_ls
For Root360 Managed Service costs please contact our Service team.
Pick one level and also add as tag to page
EXPERT