This repository contains a set of Terraform modules in the modules folder for setup a data export from the Nobl9 application to object storage and further integration.

• The modules folder contains several standalone, reusable, production-grade modules that you can use to setup a data export from Nobl9 to AWS S3 and set up a further integration with Snowflake.

• The root folder is an example of how to use modules to export data from Nobl9 to S3 bucket and set up a further integration with Snowflake.
  The Terraform Registry requires the root of every repo to contain Terraform code. Thus, we've put one of the examples there. The example that represents a typical scenario for learning and experimenting is included in the registry. Use the underlying modules from the modules folder if you need fine-grained control.

• The nobl9 module creates an S3 bucket and IAM role, which gives the Nobl9 app write access

• The snowflake module creates an IAM role which gives Snowflake read access to an existing S3 bucket (for an instance provisioned with the above module) and configures notifications about file upload for Snowpipe.

The root module uses Nobl9 and Snowflake to provide end-to-end setup for export from Nobl9 to S3 and integration with Snowflake. The following is a manual on how to use it. When a more sophisticated configuration is required, use modules Nobl9 and Snowflake.

This module can be imported directly (it is advised to pin to a specific version by adding ?ref=<REF>, e.g., ?ref=0.1.0).

module "aws_snowflake" {
  source = "git::[email protected]:nobl9/export-from-n9-terraform.git"
}

Parameters must be passed as described in the following step-by-step instructions.

1. Obtain the AWS external ID for your organization in Nobl9 UI or with the sloctl command-line tool.

2. Run the following command in sloctl:

   sloctl get dataexport --aws-external-id

   Output

   <EXTERNAL_ID_FOR_ORGANIZATION>

3. Enter the variables for Terraform.
   For example, create the file input.auto.tfvars in the root module with the following content:

   aws_region = "<AWS_REGION_WHERE_TO_DEPLOY_RESOURCES>"  # Region where Terraform provsion S3 bucket.
   external_id_provided_by_nobl9 = "<EXTERNAL_ID_FOR_ORGANIZATION>"  # Previously obtained from Nobl9 external id.
   # Specify the desired name for a bucket that must be globally unique, e.g., n9-exported-data-1234
   # (do not include s3:// in the passed name); when omitted, a random name will be generated.
   s3_bucket_name = "<S3_BUCKET_FOR_N9_DATA_NAME>"
   
   # Optionally, tags to add for every created resource.
   tags = {
       "key": "value"
   }
   
   # Other available variables.
   
   # Specify the desired name for the IAM role, which gives Nobl9 access to the created bucket
   # when omitted, default name "nobl9-exporter" is used
   iam_role_to_assume_by_nobl9_name = "<NAME_OF_CREATED_ROLE_FOR_N9>"
   
   # Specify whether all objects should be deleted from the previously created S3 bucket when using terraform destroy
   # This will allow destroying the non-empty S3 bucket without errors
   # When omitted, the default value false is used
   s3_bucket_force_destroy = <S3_BUCKET_FOR_N9_FORCE_DESTROY>

4. Initialize a new or existing Terraform working directory by running the following command:

   terraform init

   next

   terraform apply

5. Wait for the Terraform outputs.

   iam_role_to_assume_by_nobl9 = "arn:aws:iam::<AWS_ACCOUNT_ID>:role/<NAME_OF_CREATED_ROLE_FOR_N9>"
   s3_bucket_name = "<S3_BUCKET_FOR_N9_DATA_NAME>"

6. Copy the above to the configuration of DataExport in the N9 App with YAML configuration. The data is exported every hour by the Nobl9 app to the S3 bucket.

   The following is an example Nobl9 YAML for DataExport, and can be applied with sloctl or configured in the UI. The field value for roleArn is obtained from Terraform output.

   apiVersion: n9/v1alpha
   kind: DataExport
   metadata:
     name: data-export-s3
     project: default
   spec:
     exportType: S3
     spec:
       bucketName: "<S3_BUCKET_FOR_N9_DATA_NAME>"
       roleArn: "arn:aws:iam::<AWS_ACCOUNT_ID>:role/<NAME_OF_CREATED_ROLE_FOR_N9>"

Snowflake can automatically pull data from this bucket on every automatic upload done by N9 and make them available in the database. Steps related to Snowflake have to be performed in its UI.

1. Create the database, table, and format for Nobl9 data in Snowflake.
   Default names are used in the following setup (for example, nobl9_slo for database, etc.). Feel free to use different names.

   create database nobl9_slo;

   create or replace table nobl9_data(
     timestamp datetime not null,
     organization string not null,
     project string not null,
     measurement string not null,
     value double not null,
     time_window_start datetime,
     time_window_end datetime,
     slo_name string not null,
     slo_description string,
     error_budgeting_method string not null,
     budget_target double not null,
     objective_display_name string,
     objective_value double,
     objective_operator string,
     service string not null,
     service_display_name string,
     service_description string,
     slo_time_window_type string not null,
     slo_time_window_duration_unit string not null,
     slo_time_window_duration_count int not null,
     slo_time_window_start_time timestamp_tz,
     composite boolean,
     objective_name string,
     slo_labels string,
     slo_display_name string
   );

   create or replace file format nobl9_csv_format
     type = csv
     field_delimiter = ','
     skip_header = 1
     null_if = ('NULL', 'null')
     empty_field_as_null = true
     field_optionally_enclosed_by = '"'
     error_on_column_count_mismatch = false
     compression = gzip;

2. Create the Snowflake integration with S3, and enter the following with the desired values:

   • <AWS_ACCOUNT_ID> - ID of AWS account where the S3 bucket for Nobl9 was created
   • <SNOWFLAKE_ROLE_NAME> - name of the IAM role to create to be assumed by Snowflake, when omitted in Terraform configuration, default name snowflake-integration must be used
   • <BUCKET_NAME> - the name of previously created S3 bucket (from Terraform output)

   create or replace storage integration nobl9_s3
     type = external_stage
     storage_provider = s3
     enabled = true
     storage_aws_role_arn = 'arn:aws:iam::<AWS_ACCOUNT_ID>:role/<SNOWFLAKE_ROLE_NAME>'
     storage_allowed_locations = ('s3://<BUCKET_NAME>');

3. Obtain the following <STORAGE_AWS_IAM_USER_ARN> value and <STORAGE_AWS_EXTERNAL_ID> by running the following command:

   desc integration nobl9_s3;

4. Add the output values of the above command to the Terraform variables to the input.auto.tfvars file

   snowflake_storage_aws_iam_user_arn = "<STORAGE_AWS_IAM_USER_ARN>"
   snowflake_storage_aws_external_id = "<STORAGE_AWS_EXTERNAL_ID>"
   # Previously referenced in Snowlake configuration, gives access to bucket.
   snowflake_iam_role_name = "<SNOWFLAKE_ROLE_NAME>" # Omit when default name snowflake-integration is used.

5. Apply the Terraform variables and wait.

   terraform apply

6. Run the following command in Snowflake worksheet to set up the integration with S3:

   create or replace stage s3_export_stage
     url = 's3://<BUCKET_NAME>'
     file_format = nobl9_csv_format
     storage_integration = nobl9_s3;

7. Start configuring Snowpipe:

   create pipe nobl9_data_pipe auto_ingest=true as
     copy into nobl9_data
       from @s3_export_stage;

   The above command will end successfully only when the configuration of access by Snowflake (previous steps) to S3 was done correctly.

   desc pipe nobl9_data_pipe;

8. Add to Terraform variables as previous (e.g., to file input.auto.tfvars) ARN of SQS queue from Snowflake where notification about a new file in S3 will be sent.

   snowflake_sqs_notification_arn = "<notification_channel>"

9. Run the apply command for the last time.

   terraform apply

From now on, the data from every file exported by Nobl9 to the dedicated S3 bucket should be available automatically in the Snowflake database nobl9_slo.

• The following is an example query to execute on data:

  select distinct
    service_display_name,
    service,
    project,
    slo_name,
    objective_name,
    objective_value,
    budget_target * 100 as target
  from nobl9_data
  order by service, slo_name;

1. In Snowflake worksheet, use the following command to delete the setup:

   drop database nobl9_slo;

   drop storage integration nobl9_s3;

2. For resources created with Terraform, use the following to delete the setup:

   terraform destroy

   Objects in the S3 bucket prevent deletion unless s3_bucket_force_destroy variable is set to true. The above allows destroying the S3 bucket with its content.

3. The configuration of object DataExport should be deleted in Nobl9 too.