Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update put_s3_bucket_encryption.fp #68

Merged
merged 6 commits into from
Sep 3, 2024
Merged

Conversation

gcasilva
Copy link
Contributor

@gcasilva gcasilva commented Aug 26, 2024

Fixing source file name for s3 bucket encryption function

Checklist

Fixing source file name for s3 bucket encryption function
Updating to clarify allowed values in sse_algorithm.
Fixing jsonencode to use the correct attribute response for the step "function" "build_enctryption_config"
@misraved misraved requested a review from khushboo9024 August 27, 2024 05:33
@misraved misraved changed the base branch from main to release/v0.4.1 August 27, 2024 05:34
change source path in s3 bucket encryption pipeline to relative path when running flowpipe mod install
@misraved
Copy link
Contributor

Hi @gcasilva,

Thank you so much for the fix! 👍

I have a couple of questions and suggestions regarding the PR:

  • To maintain consistency, I suggest we update the folder name under functions from update_s3_bucket_encryption to put_s3_bucket_encryption.
  • Is there a particular reason you used source = ".flowpipe/mods/github.com/turbot/[email protected]/pipelines/s3/functions/update_s3_bucket_encryption" instead of ./pipelines/s3/functions/put_s3_bucket_encryption? I made this change locally to use source = "./pipelines/s3/functions/put_s3_bucket_encryption", and it seemed to work fine.

Please feel free to reach out if you have any further questions. I"m happy to help! 👍

gcasilva and others added 2 commits August 28, 2024 10:56
@gcasilva
Copy link
Contributor Author

Hi @misraved please check my comments below:

  1. Made the changes to the folder so its put_s3_bucket_encryption
  2. source = ".flowpipe/mods/github.com/turbot/[email protected]/pipelines/s3/functions/update_s3_bucket_encryption" is needed as when user runs flowpipe mod install github.com/turbot/flowpipe-mod-aws to install the flowpipe aws mod dependency it installs in this specific path convention .flowpipe/mods/github.com/turbot/[email protected]/ as tested by me and documented in https://flowpipe.io/docs/build/mod-dependencies
    If we don"t have this source path corrected user would have to manually create this directory in their current path or they would run into an error where the flowpipe function wouldn"t find the source whenever they try to use the s3 bucket encryption fp, which is what happened to me before this fix.

The only caveat with this is that whenever flowpipe-mod-aws has a new release you"d have to change this line of code to reflect whatever new version is (e.g. v0.4.2) .flowpipe/mods/github.com/turbot/[email protected]/

A better approach would be to change flowpipe logic to use the tag "latest" for release instead of a version number which is what is uses today, but that"s outside of what I can help on this PR.

@judell
Copy link

judell commented Aug 29, 2024

@gcasilva I"m not seeing that problem. I am in ~/flowpipe-mod-aws with these changes only.

  1. rename functions/update_s3_bucket_encryption -> functions/put_s3_bucket_encryption

-      ["--server-side-encryption-configuration", jsonencode(step.function.build_encryption_config.result)],
+      ["--server-side-encryption-configuration", jsonencode(step.function.build_encryption_config.response)],

Without these changes, this command fails as you reported.

flowpipe pipeline run aws.pipeline.put_s3_bucket_encryption   --arg "region=us-east-1"   --arg "bucket=jon-turbot-test-bucket-01"   --arg "sse_algorithm=AWS256"   --arg "bucket_key_enabled=true"`

With the changes, it succeeds.

What am I missing?

@judell
Copy link

judell commented Aug 29, 2024

OK I see the problem.

If I run the command directly from the tweaked ~/flowpipe-mod-aws it"s OK.

But:

mkdir apn
cd apn 
flowpipe mod install ~/flowpipe-mod-aws

Now we are back to

Bad Request: "source" not found for function: aws.pipeline.put_s3_bucket_encryption.function.build_encryption_config

This indeed appears to be a Flowpipe bug that"s beyond the scope of this PR.

@misraved, @khushboo9024 can we merge this PR without that change, and address other issue separately?

@johnsmyth
Copy link

@judell @misraved, @khushboo9024 @gcasilva

FYI - I believe the bug with the relative path in source is fixed in 0.8.1 (thanks @vhadianto!):
https://flowpipe.io/changelog/flowpipe-cli-v0-8-1

@judell
Copy link

judell commented Aug 30, 2024

Excellent, thanks @vhadianto!

@misraved misraved merged commit 07313dd into turbot:release/v0.4.1 Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants