Everything logs to syslog (I generally use rsyslog) in JSON format.
All syslog instances push to a central instance, also running rsyslog. This allows us to tail logs on each instance, as well as tail / grep system-wide on the central instance.
Central instance pushes everything directly into elasticsearch.
Using Kibana for searching and aggregating. Using simple scripts for generating alarms and reports.
Every day a snapshot of the previous day is uploaded to S3 and indexes from 14 days ago are removed. This allows us to easily restore historical data from the past, but also keeps our ES instance relatively thin for daily usage / tracking / debugging. It also makes it possible to replace our central log instance without losing too much.
All devs use some simple convention (ideally built into the logging libs) to make searching and tracing relatively easy. These include "request ids" for all logs pertaining to a single process of work, and "thread ids" for tracing multiple related "requests".
How do you change everything on a system to use JSON format? My syslog (Debian) is filled with text-line entries, and I've not seen a setting to change this.
By "Everything" in my post, I mean all of our own applications. Some services allow you to format logs to json like nginx using log_format[1]. For others, you may find app-specific configuration or plugins for log formatting or simply use plain grep / kibana text search.
I imagine in those cases something like logstash may help, but I don't really know as I tend to avoid logstash.
All syslog instances push to a central instance, also running rsyslog. This allows us to tail logs on each instance, as well as tail / grep system-wide on the central instance.
Central instance pushes everything directly into elasticsearch.
Using Kibana for searching and aggregating. Using simple scripts for generating alarms and reports.
Every day a snapshot of the previous day is uploaded to S3 and indexes from 14 days ago are removed. This allows us to easily restore historical data from the past, but also keeps our ES instance relatively thin for daily usage / tracking / debugging. It also makes it possible to replace our central log instance without losing too much.
All devs use some simple convention (ideally built into the logging libs) to make searching and tracing relatively easy. These include "request ids" for all logs pertaining to a single process of work, and "thread ids" for tracing multiple related "requests".
I documented how I have rsyslog and elasticsearch set up here: https://www.reddit.com/r/devops/comments/9g1nts/rsyslog_elas...