Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(misconf): use slog #7295

Merged
merged 12 commits into from
Aug 23, 2024
Merged

refactor(misconf): use slog #7295

merged 12 commits into from
Aug 23, 2024

Conversation

nikpivkin
Copy link
Contributor

@nikpivkin nikpivkin commented Aug 1, 2024

Description

This PR improves the logging in the IaC package. The slog package is now used for logging, which aligns the logging style with Trivy.

Changes:

  • Use the slog package instead of debug
  • Added messages when parameter values are missing in CloudFormation and Terraform variables when they are not set by default and passed through arguments
  • Reduced logging noise when scanning Terraform

Small refactoring: since no options are now passed to IaC parsers, unnecessary interfaces have been removed.

Related issues

Before

2024-08-13T18:46:33 06:00	DEBUG	Initializing scan cache...	type="memory"
2024-08-13T18:46:33 06:00	DEBUG	Scanning files for misconfigurations...	scanner="Helm"
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.295939500 helm.scanner.rego                Overriding filesystem for checks!
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.297375000 helm.scanner.rego                Loaded 3 embedded libraries.
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.334477000 helm.scanner.rego                Loaded 192 embedded policies.
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.376318000 helm.scanner.rego                Loaded 195 checks from disk.
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.376573000 helm.scanner.rego                Overriding filesystem for data!
2024-08-13T18:46:33 06:00	DEBUG	Scanning files for misconfigurations...	scanner="CloudFormation"
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.580120000 cloudformation.scanner.rego      Overriding filesystem for checks!
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.580801000 cloudformation.scanner.rego      Loaded 3 embedded libraries.
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.613006000 cloudformation.scanner.rego      Loaded 192 embedded policies.
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.650282000 cloudformation.scanner.rego      Loaded 195 checks from disk.
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.650551000 cloudformation.scanner.rego      Overriding filesystem for data!
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.863397000 cloudformation.scanner           Found 1 results for AVD-AWS-0086
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.863424000 cloudformation.scanner           Found 1 results for AVD-AWS-0087
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.863430000 cloudformation.scanner           Found 1 results for AVD-AWS-0088
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.863433000 cloudformation.scanner           Found 1 results for AVD-AWS-0090
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.863437000 cloudformation.scanner           Found 1 results for AVD-AWS-0132
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.863442000 cloudformation.scanner           Found 1 results for AVD-AWS-0091
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.863460000 cloudformation.scanner           Found 1 results for AVD-AWS-0092
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.863464000 cloudformation.scanner           Found 1 results for AVD-AWS-0093
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.863519500 cloudformation.scanner           Found 1 results for AVD-AWS-0094
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.863698000 cloudformation.scanner.rego      Scanning 1 inputs...
2024-08-13T18:46:33 06:00	DEBUG	Scanning files for misconfigurations...	scanner="Terraform"
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.869866000 terraform.scanner                Scanning [&{%!s(*mapfs.file=&{ [] {. 256 2147484096 {13954107996550735840 2495660001 0x109cfd2a0} <nil>} {{{0 0} {[] {} 0x14005a44aa0} map[] 0}}}) .}] at '.'...
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.872081000 terraform.scanner.rego           Overriding filesystem for checks!
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.872710000 terraform.scanner.rego           Loaded 3 embedded libraries.
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.906372000 terraform.scanner.rego           Loaded 192 embedded policies.
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.937903000 terraform.scanner.rego           Loaded 195 checks from disk.
2024-08-13T18:46:33 06:00	DEBUG	[misconf] 46:33.938182000 terraform.scanner.rego           Overriding filesystem for data!
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.147922000 terraform.parser.<root>          Setting project/module root to 'tf'
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.147945000 terraform.parser.<root>          Parsing FS from 'tf'
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.147964000 terraform.parser.<root>          Parsing 'tf/main.tf'...
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.148512000 terraform.parser.<root>          Added file tf/main.tf.
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.148586000 terraform.scanner                Scanning root module 'tf'...
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.148591000 terraform.parser.<root>          Setting project/module root to 'tf'
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.148863000 terraform.parser.<root>          Parsing FS from 'tf'
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.148601000 terraform.parser.<root>          Parsing 'tf/main.tf'...
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.148630000 terraform.parser.<root>          Added file tf/main.tf.
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.148634000 terraform.parser.<root>          Evaluating module...
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.148649500 terraform.parser.<root>          Read 1 block(s) and 0 ignore(s) for module 'root' (1 file[s])...
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.148657000 terraform.parser.<root>          Added 0 variables from tfvars.
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.148669500 terraform.parser.<root>          Working directory for module evaluation is "/Users/nikita/projects/trivy-test/iac-test"
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.148882000 terraform.parser.<root>.evaluator Filesystem key is 'fa37a03cfd8ac454b5e2679e7861189a66b5bd655e64b6f2c3a2a19d93c9482c'
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.148887000 terraform.parser.<root>.evaluator Starting module evaluation...
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.149641000 terraform.parser.<root>.evaluator Starting submodule evaluation...
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.149645000 terraform.parser.<root>.evaluator All submodules are evaluated at i=0
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.149650000 terraform.parser.<root>.evaluator Starting post-submodule evaluation...
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.149711000 terraform.parser.<root>.evaluator Finished processing 0 submodule(s).
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.149714000 terraform.parser.<root>.evaluator Module evaluation complete.
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.149717000 terraform.parser.<root>          Finished parsing module 'root'.
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.149720000 terraform.executor               Adapting modules...
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.151064000 terraform.executor               Adapted 1 module(s) into defsec state data.
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.151068000 terraform.executor               Using max routines of 7
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.151144000 terraform.executor               Initialized 487 rule(s).
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.151147000 terraform.executor               Created pool with 7 worker(s) to apply rules.
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.151580000 terraform.scanner.rego           Scanning 1 inputs...
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.153173000 terraform.executor               Finished applying rules.
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.153188000 terraform.executor               Applying ignores...
2024-08-13T18:46:34 06:00	DEBUG	Scanning files for misconfigurations...	scanner="Dockerfile"
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.157011000 dockerfile.scanner.rego          Overriding filesystem for checks!
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.157643000 dockerfile.scanner.rego          Loaded 3 embedded libraries.
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.193945000 dockerfile.scanner.rego          Loaded 192 embedded policies.
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.226278000 dockerfile.scanner.rego          Loaded 195 checks from disk.
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.226545000 dockerfile.scanner.rego          Overriding filesystem for data!
2024-08-13T18:46:34 06:00	DEBUG	[misconf] 46:34.441370000 dockerfile.scanner.rego          Scanning 1 inputs...
2024-08-13T18:46:34 06:00	DEBUG	OS is not detected.
2024-08-13T18:46:34 06:00	INFO	Detected config files	num=4
2024-08-13T18:46:34 06:00	DEBUG	Scanned config file	file_path="cf/main.yaml"
2024-08-13T18:46:34 06:00	DEBUG	Scanned config file	file_path="dockerfiles/Dockerfile.dev"
2024-08-13T18:46:34 06:00	DEBUG	Scanned config file	file_path="tf"
2024-08-13T18:46:34 06:00	DEBUG	Scanned config file	file_path="tf/main.tf"
2024-08-13T18:46:34 06:00	DEBUG	[vex] VEX filtering is disabled

After

2024-08-13T18:46:53 06:00       DEBUG   Initializing scan cache...      type="memory"
2024-08-13T18:46:53 06:00       DEBUG   Scanning files for misconfigurations... scanner="Helm"
2024-08-13T18:46:53 06:00       DEBUG   [rego] Overriding filesystem for checks
2024-08-13T18:46:53 06:00       DEBUG   [rego] Embedded libraries are loaded    count=3
2024-08-13T18:46:53 06:00       DEBUG   [rego] Embedded checks are loaded       count=192
2024-08-13T18:46:53 06:00       DEBUG   [rego] Checks from disk are loaded      count=195
2024-08-13T18:46:53 06:00       DEBUG   [rego] Overriding filesystem for data
2024-08-13T18:46:53 06:00       DEBUG   Scanning files for misconfigurations... scanner="Terraform"
2024-08-13T18:46:53 06:00       DEBUG   [terraform scanner] Scanning directory  file_path="."
2024-08-13T18:46:53 06:00       DEBUG   [rego] Overriding filesystem for checks
2024-08-13T18:46:53 06:00       DEBUG   [rego] Embedded libraries are loaded    count=3
2024-08-13T18:46:53 06:00       DEBUG   [rego] Embedded checks are loaded       count=192
2024-08-13T18:46:53 06:00       DEBUG   [rego] Checks from disk are loaded      count=195
2024-08-13T18:46:53 06:00       DEBUG   [rego] Overriding filesystem for data
2024-08-13T18:46:54 06:00       DEBUG   [terraform parser] Setting project/module root  module="root" file_path="tf"
2024-08-13T18:46:54 06:00       DEBUG   [terraform parser] Parsing FS   module="root" file_path="tf"
2024-08-13T18:46:54 06:00       DEBUG   [terraform parser] Parsing      module="root" file_path="tf/main.tf"
2024-08-13T18:46:54 06:00       DEBUG   [terraform parser] Added file   module="root" file_path="tf/main.tf"
2024-08-13T18:46:54 06:00       INFO    [terraform scanner] Scanning root module        file_path="tf"
2024-08-13T18:46:54 06:00       DEBUG   [terraform parser] Setting project/module root  module="root" file_path="tf"
2024-08-13T18:46:54 06:00       DEBUG   [terraform parser] Parsing FS   module="root" file_path="tf"
2024-08-13T18:46:54 06:00       DEBUG   [terraform parser] Parsing      module="root" file_path="tf/main.tf"
2024-08-13T18:46:54 06:00       DEBUG   [terraform parser] Added file   module="root" file_path="tf/main.tf"
2024-08-13T18:46:54 06:00       DEBUG   [terraform parser] Loading module       module="root" module="root"
2024-08-13T18:46:54 06:00       DEBUG   [terraform parser] Read block(s) and ignore(s)  module="root" blocks=1 ignores=0
2024-08-13T18:46:54 06:00       DEBUG   [terraform parser] Added input variables from tfvars    module="root" count=0
2024-08-13T18:46:54 06:00       DEBUG   [terraform parser] Working directory for module evaluation      module="root" file_path="/Users/nikita/projects/trivy"
2024-08-13T18:46:54 06:00       DEBUG   [terraform evaluator] Starting module evaluation...     path="tf"
2024-08-13T18:46:54 06:00       DEBUG   [terraform evaluator] Starting iteration        iteration=0
2024-08-13T18:46:54 06:00       DEBUG   [terraform evaluator] Starting iteration        iteration=1
2024-08-13T18:46:54 06:00       DEBUG   [terraform evaluator] Context unchanged iteration=1
2024-08-13T18:46:54 06:00       DEBUG   [terraform evaluator] Starting post-submodules evaluation...
2024-08-13T18:46:54 06:00       DEBUG   [terraform evaluator] Starting iteration        iteration=0
2024-08-13T18:46:54 06:00       DEBUG   [terraform evaluator] Starting iteration        iteration=1
2024-08-13T18:46:54 06:00       DEBUG   [terraform evaluator] Context unchanged iteration=1
2024-08-13T18:46:54 06:00       DEBUG   [terraform evaluator] Module evaluation complete.
2024-08-13T18:46:54 06:00       DEBUG   [terraform parser] Finished parsing module      module="root" module="root"
2024-08-13T18:46:54 06:00       DEBUG   [terraform executor] Adapting modules...
2024-08-13T18:46:54 06:00       DEBUG   [terraform executor] Adapted module(s) into state data. count=1
2024-08-13T18:46:54 06:00       DEBUG   [terraform executor] Using max routines count=7
2024-08-13T18:46:54 06:00       DEBUG   [terraform executor] Initialized rule(s).       count=487
2024-08-13T18:46:54 06:00       DEBUG   [rego] Scannning inputs count=1
2024-08-13T18:46:54 06:00       DEBUG   [terraform executor] Finished applying rules.
2024-08-13T18:46:54 06:00       DEBUG   [terraform executor] Applying ignores...
2024-08-13T18:46:54 06:00       DEBUG   Scanning files for misconfigurations... scanner="CloudFormation"
2024-08-13T18:46:54 06:00       DEBUG   [cloudformation parser] Context loaded from source      file_path="cf/main.yaml"
2024-08-13T18:46:54 06:00       DEBUG   [rego] Overriding filesystem for checks
2024-08-13T18:46:54 06:00       DEBUG   [rego] Embedded libraries are loaded    count=3
2024-08-13T18:46:54 06:00       DEBUG   [rego] Embedded checks are loaded       count=192
2024-08-13T18:46:54 06:00       DEBUG   [rego] Checks from disk are loaded      count=195
2024-08-13T18:46:54 06:00       DEBUG   [rego] Overriding filesystem for data
2024-08-13T18:46:54 06:00       DEBUG   [rego] Scannning inputs count=1
2024-08-13T18:46:54 06:00       DEBUG   Scanning files for misconfigurations... scanner="Dockerfile"
2024-08-13T18:46:54 06:00       DEBUG   [rego] Overriding filesystem for checks
2024-08-13T18:46:54 06:00       DEBUG   [rego] Embedded libraries are loaded    count=3
2024-08-13T18:46:54 06:00       DEBUG   [rego] Embedded checks are loaded       count=192
2024-08-13T18:46:54 06:00       DEBUG   [rego] Checks from disk are loaded      count=195
2024-08-13T18:46:54 06:00       DEBUG   [rego] Overriding filesystem for data
2024-08-13T18:46:54 06:00       DEBUG   [rego] Scannning inputs count=1
2024-08-13T18:46:54 06:00       DEBUG   OS is not detected.
2024-08-13T18:46:54 06:00       INFO    Detected config files   num=4
2024-08-13T18:46:54 06:00       DEBUG   Scanned config file     file_path="cf/main.yaml"
2024-08-13T18:46:54 06:00       DEBUG   Scanned config file     file_path="dockerfiles/Dockerfile.dev"
2024-08-13T18:46:54 06:00       DEBUG   Scanned config file     file_path="tf"
2024-08-13T18:46:54 06:00       DEBUG   Scanned config file     file_path="tf/main.tf"
2024-08-13T18:46:54 06:00       DEBUG   [vex] VEX filtering is disabled

Checklist

  • I've read the guidelines for contributing to this repository.
  • I've followed the conventions in the PR title.
  • I've added tests that prove my fix is effective or that my feature works.
  • I've updated the documentation with the relevant information (if needed).
  • I've added usage information (if the PR introduces new options)
  • I've included a "before" and "after" example to the description (if the PR is a user interface change).

Comment on lines 199 to 210
p.logger.Debug("Added input variables from tfvars", log.Int("count", len(inputVars)))

for _, varBlock := range blocks.OfType("variable") {
if varBlock.GetAttribute("default") == nil {
if _, ok := inputVars[varBlock.TypeLabel()]; !ok {
p.logger.Warn(
"Variable was not found in the environment or variable files. Evaluating may not work correctly.",
log.String("variable", varBlock.TypeLabel()),
)
}
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a test for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done 1cb0263
Also changed the wording of the logs to mean missing values rather than missing variables/parameters.

Comment on lines 62 to 63
opt.Debug("Found a token for the registry at %s", hostname)
opt.DebugLogger.Debug("Found a token for the registry", log.String("hostname", hostname))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems a little too verbose to have opt.DebugLogger.Debug. Could we rename it better?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed DebugLogger to Logger b2cdb99

@simar7 simar7 self-requested a review August 22, 2024 00:02
Copy link
Member

@simar7 simar7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@simar7
Copy link
Member

simar7 commented Aug 22, 2024

@nikpivkin looks like we have some merge conflicts to resolve prior to merging.

@nikpivkin
Copy link
Contributor Author

@simar7 Done

@simar7 simar7 added this pull request to the merge queue Aug 23, 2024
Merged via the queue into aquasecurity:main with commit bfdf5cf Aug 23, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bug(misconf): Inconsistent in the issue count If terraform variables are not passed
2 participants