Minimal Web Analytics provider.
In other words, syslog drainer (works with Heroku) that stores logs in a PostgreSQL DB, transforms them into aggregatable information and displays them in a friendly manner.
The goal is to build something simple yet useful.
In your application, log the following line while processing your request:
LOG_LINE_KEY=JSON_ENCODED_DATA
LOG_LINE_KEY is a uniquely identifiable string that will be used to pick or ignore log lines from being processed. Configurable via the LOG_LINE_KEY env var.
JSON_ENCODED_DATA is a JSON string that contains the following keys:
| Key | Description | Example | Required |
|---|---|---|---|
| path | Requested path | /index | yes |
| referrer | Value of the referrer header | https://myblog.com/ | no |
| query_string | Value of the query params | utm_source=Facebook&igshid=1234ABCD | no |
| http_method | HTTP Verb of the request | GET | yes |
| format | MIME type of the request | text/html | yes |
| ip | Remote IP | 182.22.98.233 | yes |
| user_agent | User agent | Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36 | yes |
This app provides an HTTP POST endpoint to feed logs into the logger (POST /logs).
HTTP header Logplex-Msg-Count must be set with the amount of messages sent in the body of the request, see Heroku's docs.
Logs are parsed according to RFC5424.
Once parsed they are stored in the logs table, and enqueued to be transformed in the background.
Transformation process is run in background jobs, via Sidekiq.
Current transformations are:
- IP to country code
- ...
Logs that do not contain transformable information are removed.
Once the logs are transformed, they are displayed in the Dashboard page.
It uses chart.js for the line chart, and GeoChart from Google Charts for the country chart.
Basic filters are in place, although they need to be improved.
HTTP Basic Auth is used to protect all endpoints.
Set the env vars HTTP_BASIC_AUTH_NAME and HTTP_BASIC_AUTH_PASSWORD with your desired credentials.
Set the CONFIG_KEEP_LOGS to true in order to make the app maintain all logs.
- Install ruby (2.7.x preferred),
gem install bundlerandbundle install. - Create the database
bundle exec rails db:create db:migrate. - Populate some data with
bundle exec rails runner 'FactoryBot.create_list(:entry, 1000)'. - Copy the .env.dist file and update with your data
cp .env.dist .env - Download the GeoLite2 Country database from your MaxMind's account and set the following env vars:
- GEOIP_GEOLITE2_PATH the path to your downloaded MaxMind database
- GEOIP_GEOLITE2_COUNTRY_FILENAME the file name of the downloaded MaxMind database
- Run the server
bundle exec rails s
- Follow first two steps from the Development section
- Copy the .env.test file
cp .env.test .env.test.local - Set the GeoLite2 env vars in the .env.test.local file the same way as in the previous Development section
- Run the specs
bundle exec rspec
- Add the buildpack https://github.com/danstiner/heroku-buildpack-geoip-geolite2 to download MaxMind's database at deploy time.
- Get a MaxMind license key and set the env var
MAXMIND_LICENSE_KEYwith the license key. - Set the env vars from .env.dist in your Heroku app — no need to set the GeoLite2 env vars as they are automatically set by the buildpack previously installed.
- The rest is a normal heroky deploy
git push heroku master
Once the app is deployed, add it as a log drainer for the web app you want. In Heroku it's done with:
heroku drains:add https://HTTP_BASIC_AUTH_NAME:HTTP_BASIC_AUTH_PASSWORD@APPLOGGER_DOMAIN/logs -a NAME_OF_YOUR_APP
Make sure to replace HTTP_BASIC_AUTH_NAME, HTTP_BASIC_AUTH_PASSWORD and APPLOGGER_DOMAIN with the correct values.