Web statistics using goaccess.

[kzurell]

This generates weekly web usage statistics and emails the report to the administrator. It installs and configures GoAccess, which is standard on Ubuntu.

I wanted web stats for my MIAB static sites, but without using a separate web property or service. Not looking for minute-by-minute analytics, only a casual, general sense of usage. Another contributor proposed adding AWStats to MIAB some time ago, which would mean a separate process and web interface.

This patch installs GoAccess, configures it for 7 days of database retention, runs it on logrotate and to generate the HTML stats report on Mondays along with the other reports. It emails the report to the site administrator as an attachment, a self-contained HTML page with lots of graphs and numbers.

Breaking changes: this patch alters the Nginx log file to include virtual host name (and port, not used). I have altered the Munin jail which watches access.log, and @JoshData's setup tracking script (not fully tested). It changes the stock Nginx log_format in what I read is a correct way (turn off globally, define w/ custom in each server), would like confirmation.

Left to do: the GoAccess configuration is largely stock, including dashboard layout. It can do geographic analysis, which is very interesting but requires extra configuration & files which may or may not be straightforward. The /mail/ etc. URLs appear in the stats, which might not be desired. Not sure what happens if GoAccess runs on logrotate and daily_tasks at the same time. GoAccess can't be made completely --quiet, need to verify this won't cause problems.

• Why GoAccess?
  It can run completely batch. The reports are presentable (& self-contained, no remote dependencies per browser console). Development is ongoing. There are others, the same approach could easily work for a different batch processor.
• Why not use GoAccess' live HTML statistics page?
  GoAccess can serve an HTML page and use WebSockets to update the dashboard. This means a separate web presence, which could be done but adds complexity.
  If the administrator wants live access they can ssh in, run goaccess and use the ncurses interface, which is also very slick, probably enough to diagnose in-the-moment issues.
• Why not use separate virtual host access logs?
  Could do, but that's a bigger discussion.
• Why not make the report HTML mail instead of an attachment?
  The static report that GoAccess generates is largely script and JSON, RoundCube strips out the scripts and metas.

It is far along enough to fail usefully. It works in Vagrant for setup and normal running. Did my best idiomatic python and bash, but, y'know.... Would like to see if there's any appetite before I go any further. --Thanks, K.