Pull Request

Issue link(s):

Summary - What I changed

Please merge #3955 into this PR before merging to main

This PR adds the ability to specify a config variable to auto generate a custom schema file for any missing non-ecs fields. To trigger this behavior add the following line to your _config.yaml

auto_gen_schema_file: <path_to_schema_json_file>

I addition, there are a number of other minor fixes/updates to the import-rules-to-repo process including:

• Added error skipping for rule-prompt and import-rules-to-repo.
• Modified --require-only behavior in import-rules-to-repo to populate all fields but only prompt for required ones.
• Improved extract_error_field function to handle non alpha-numeric characters.
• Introduced CLI argument to remove null values from TOML rule files.
• Added support for rule.actions.frequency.throttle field, required in Kibana.
• Implemented a _add_known_nulls function to handle null fields in TOML files.
• Added support for action connectors.
• Enhanced rule prompt with skip errors support.
• Included rule name in error.txt output for easier debugging.
• Updated config file initialization to include directories for actions, action connectors, and exceptions.
• Added helper functions for auto-generating schema.
• Made Exception List optional within an Exception Container.
• Cleaned up exceptions and action connectors build and parse functions.
• Updated Generic Loader to support Action Connectors and fixed a bug.
• Added support for endpoint integration.
• Fixed formatting issue with single lines ending in a double quote in a TOML file.
• Improved formatting support for threat query, setup, and description.
• Fixed a bug in extract_error_field function related to quotes around the field value.
• Fixed version printing bug in all_versions function.
• Updated rule name definitions for customer rules to provide less restriction.
• Implemented minor bug fixes.

How To Test

Autogen Custom Schema

To test this, first add the appropriate config variable. Next, run view-rule on a rule that has fields that are not currently present in a schema. The rule should validate successfully and your schema file should be updated.

Example: python -m detection_rules view-rule custom_rules/rules/dac_demo_dev_rule_1.toml

[metadata]
creation_date = "2024/07/29"
maturity = "production"
updated_date = "2024/07/29"

[rule]
actions = []
author = ["DAC User"]
description = "Test Rule"
enabled = true
exceptions_list = []
false_positives = []
from = "now-6m"
index = ["the-best-integration-ever*"]
interval = "5m"
language = "eql"
max_signals = 100
name = "DAC Demo Dev Rule 1"
references = []
risk_score = 47
risk_score_mapping = []
rule_id = "794d2fc0-ecd0-4963-99da-fd587666b80d"
setup = "Test Setup"
severity = "medium"
severity_mapping = []
tags = []
threat = []
to = "now"
type = "eql"

query = '''
process where host.os.type.fakeData == "linux" and process.name.okta.thread == "updated"
'''
[[rule.related_integrations]]
package = "endpoint"
version = "^8.2.0"

[[rule.required_fields]]
ecs = true
name = "host.os.type"
type = "keyword"

[[rule.required_fields]]
ecs = true
name = "process.name"
type = "keyword"

This will occur in any command where rule contents are validated against a schema, including import-rules-to-repo, kibana export-rules and others.

Additionally added, is the --skip-errors flag to import-rules-to-repo to follow similar functionality from this same flag for the kibana export-rules command. With this flag one should expect output similar to the following when importing rules.

1347 results exported
1099 rules converted
248 exceptions exported
49 errors saved to /home/forteea1/Code/clean_mains/detection-rules/custom_rules/rules/_errors.txt

Also, rule names will no longer be validated against our naming conversions for custom rules. Any valid string name will now validate.

Testing support for a required field rule.actions.frequency.throttle

Use the following example rule file to import from ndjson and export back to a kibana instance. You will get a connector id is missing error but this is expected and as far of an implmentation of actions as this PR expects. If the desired result is not working you would see a throttle error instead.

test_action_export.ndjson.txt

[metadata]
creation_date = "2024/08/03"
maturity = "development"
updated_date = "2024/08/03"

[rule]
author = ["INFOSEC_TEST"]
description = "Test"
filters = []
from = "now-360s"
index = [
    "apm-*-transaction*",
    "auditbeat-*",
    "endgame-*",
    "filebeat-*",
    "logs-*",
    "packetbeat-*",
    "traces-apm*",
    "winlogbeat-*",
    "-*elastic-cloud-logs-*",
]
interval = "5m"
language = "kuery"
max_signals = 100
name = "TestActionRule"
note = "None"
revision = 3
risk_score = 21
rule_id = "e818bf1b-dcc8-4746-80fa-a155a94a7f6b"
setup = "None"
severity = "low"
to = "now"
type = "query"
version = 1

query = '''
process.command_line : "FakeRoot"
'''


[[rule.actions]]
action_type_id = ".email"
group = "default"
id = "elastic-cloud-email"
uuid = "c405d3e6-f47d-4ee2-bfc2-9fa786051c66"

[rule.actions.frequency]
notifyWhen = "onActiveAlert"
summary = true
[rule.actions.params]
message = "Rule {{context.rule.name}} generated {{state.signals_count}} alerts"
subject = "TestEmail"
to = ["[email protected]"]
[[rule.actions]]
action_type_id = ".webhook"
group = "default"
id = "478b2165-83fb-480d-8a4a-bb47cfcafd4c"
uuid = "fd2c83b6-414b-4e31-939a-27f399c06203"

[rule.actions.frequency]
notifyWhen = "onActiveAlert"
summary = true
[rule.actions.params]
body = "{}"

[rule.meta]
from = "1m"
kibana_siem_app_url = "https://dev-deployment-2c684a.kb.us-central1.gcp.cloud.es.io:9243/s/infosec/app/security"

Note you may have to download this to watch the example, needed it to be mp4 due to length and not sure how well github renders mp4.

Checklist

• Added a label for the type of pr: bug, enhancement, schema, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?