Centralized logging in 10 minutes
Adam Wiggins' 12 Factor App site is a great read full of practical advice for building distributed systems out of small programs. The logging section along with Adam's other logging post made me realize that I've been overcomplicating the way I handle logging in my apps.
By combining STDOUT, logger, rsyslog/syslog-ng, and a hosted log aggregator like Loggly, you can have centralized logging in a few minutes.
Register for Loggly
There are several companies that offer hosted centralized logging. Here's a few I found:
I chose Loggly because of their free tier (200MB/day, 7 day retention), but any hosted service that accepts logs via syslog should work. I'm not affiliated with Loggly in any way.
Create a Loggly Input
Loggly will prompt you to create an Input. Make sure to select "Syslog TCP".
ssh to your server and run the command they provide, which registers this host to receive log messages.
curl -X POST -u coopernurse http://bitmechanic.loggly.com/api/inputs/1111/adddevice
Enter the password associated with your account when curl prompts you. You should see some JSON data come back from the curl request.
Configure rsyslogd
I'm using rsyslogd because it is installed on Ubuntu by default. If you're using a different Linux distribution, you might opt for syslog-ng instead. The original syslogd won't work with Loggly, as it doesn't support TCP forwarding. Loggly has a table of syslog servers that support TCP forwarding.
Edit /etc/rsyslog.conf and add this to the bottom:
# custom log format that includes severity
$template myformat,"%HOSTNAME%,%syslogseverity-text%,%syslogtag% %msg%\n"
# replace [PORT] with the port supplied by Loggly
*.* @@logs.loggly.com:[PORT];myformat
Then restart rsyslogd:
sudo service rsyslog restart
Configure your app to log to syslog
Most logging packages support syslog (e.g. Python's logger module, Java's log4j, etc). Consult the docs to see how to log to syslog as appropriate.
If you aren't using a logging package you can simply write to STDOUT and
STDERR and use /usr/bin/logger to send the messages to syslog. Here's
an example:
myapp.py > >(/usr/bin/logger -p local0.info) 2> >(/usr/bin/logger -p local0.err)
This sends STDOUT to the local0 facility, with a level of info, and STDERR to the local0 facility with a level of err.
Note: Be aware of buffered I/O. For example, by default Python buffers STDOUT, so you wouldn't see those messages line by line. Use the -u flag to python to use unbuffered I/O. Java's System.out.println() uses a PrintStream, which flushes automatically at each newline, which avoids this issue.
Check out your shiny logs
After you restart your app, /var/log/syslog should still contain all your log messages. If you login to your Loggly account, you should begin seeing logs there too. For example:
Consult the Loggly search docs for more info on how to search your logs using their web interface.
