Skip to content

A dockerized solution to generate Azure Data Collection Rule (DCR) JSON Sample Files from unstructured log sources.

License

Notifications You must be signed in to change notification settings

dcodev1702/azure_dcr_json_generator

Repository files navigation

Azure DCR JSON Generator using Docker

A dockerized solution using Logstash (v8.8.0) to generate Azure Data Collection Rule (DCR) JSON Sample Files from unstructured logs.
An Apache2 Access log is used in this example. Limited testing has been done with single line log entries (Bind9 DNS & Apache2).

This is inspired from using a Logstash container to ship unstructured logs to Log Analytics using the Custom Table (Log) DCR API.
See cited resources below for a more detailed explanation.

Cited Resources:

Microsoft: Azure DCR API Tutorial
Medium: Azure DCR API Tutorial

Assumptions:

  1. Docker is installed
  2. You're connected to the Internet
  3. You have the required access to build and run Docker containers
  4. You can clone or download and unzip this repository

Instructions:

git clone https://github.com/dcodev1702/azure_dcr_json_generator.git
cd azure_dcr_json_generator
docker build -t logstash_azure_dcr .
chmod 755 generate_json_for_dcr_docker.sh
./generate_json_for_dcr_docker.sh

If you want to delete existing sample JSON files stored in ./tmp do the following

./generate_json_for_dcr_docker.sh 1

Post Condition:

The sample file: e.g. sampleFilexxxxxxxxxx.json will be in $(pwd)/tmp
You can use this file when creating a Custom Log in Log Analytics via DCR API at the KQL transformation stage

Below is the JSON from the sample file stored in ./tmp. Once uploaded to Log Analytrics - Custom Table (DCR)
you can write your KQL to normalize/transform the unstructured data to a structured format to your liking.
See KQL_Apache2_AccessLog_Transformation.txt for an example.

Apache2_Logstash_Create_DCR_JSON

The Apache2 Access Log is now parsed (transformed) at ingestion time into several different distinct and queryable fields.

Apache2_Logstash_Create_DCR_Parsed

About

A dockerized solution to generate Azure Data Collection Rule (DCR) JSON Sample Files from unstructured log sources.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published