I'm starting a new project. I need to refresh our aging syslog infrastructure.
Currently, we have a FreeBSD based central syslog server. It receives about 10 million syslog messages per day. They messages come from a mix of Solaris, Cisco, Windows, VMware, and NetApp servers.
The only automation is a simple perl scripts that generates an email when an event is alert or higher severity. I would like to change the way we react to syslog messages. Instead of reacting solely on the basis of severity, I would like to process all the well known, frequently hit cases. Over time we could reduce the number of unknown log messages that appear in the system.
In the end, we will have fewer alerts and a deeper understanding of what is going on in our systems.