Destination Redshift - additional check method check, fix s3 file deletion #34186

jbfbell · 2024-01-11T19:18:20Z

This changes Destination Redshift's check method to verify access to the svv_table_info table as well as fixes a bug related to the removal of s3 staging files if users selected this option

vercel · 2024-01-11T19:18:23Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Jan 16, 2024 8:59pm

github-actions · 2024-01-11T19:18:40Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

Co-authored-by: Marcos Marx <[email protected]>

… be parsed (#32589)

)

…33707)

…l/redshift-check-updates

edgao

mostly nits. curious what you think about overriding check directly vs exposing the method in the base class. lgtm otherwise

...ions/src/main/java/io/airbyte/cdk/integrations/destination/jdbc/AbstractJdbcDestination.java

.../io/airbyte/integrations/destination/redshift/operations/RedshiftS3StagingSqlOperations.java

...n-redshift/src/main/java/io/airbyte/integrations/destination/redshift/util/RedshiftUtil.java

docs/integrations/destinations/redshift.md

jbfbell · 2024-01-12T00:06:15Z

mostly nits. curious what you think about overriding check directly vs exposing the method in the base class. lgtm otherwise

@edgao opted to not override check because of the connection and exception handling already present. I wanted the additional checks to run inside of all that without duplicating, I think overriding the check method I would have to something more sequential like

super.check()
try {
additionalChecks()
} catch (Exception e) {
...
}
finally {
close()
}

Signed-off-by: Gireesh Sreepathi <[email protected]>

…l/redshift-check-updates

jbfbell · 2024-01-16T20:25:53Z

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/7547136162
✅ Successfully published Java CDK version=0.12.1!

…l/redshift-check-updates

## What * Resurrect removed signatures to fix compilation errors in Snowflake connector. * Partial revert of #34186

…d methods and vars (#34467) ## What * Reverting few signature changes from #34186

…etion (airbytehq#34186) Signed-off-by: Gireesh Sreepathi <[email protected]> Co-authored-by: Sitaram Shelke <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: Cynthia Yin <[email protected]> Co-authored-by: Baz <[email protected]> Co-authored-by: bazarnov <[email protected]> Co-authored-by: kekiss <[email protected]> Co-authored-by: Anatolii Yatsuk <35109939 [email protected]> Co-authored-by: Edward Gao <[email protected]> Co-authored-by: Augustin <[email protected]> Co-authored-by: Joe Reuter <[email protected]> Co-authored-by: Alexandre Cuoci <[email protected]> Co-authored-by: perangel <[email protected]> Co-authored-by: Aaron ("AJ") Steers <[email protected]> Co-authored-by: Ben Church <[email protected]> Co-authored-by: Gireesh Sreepathi <[email protected]>

## What * Resurrect removed signatures to fix compilation errors in Snowflake connector. * Partial revert of airbytehq#34186

…d methods and vars (airbytehq#34467) ## What * Reverting few signature changes from airbytehq#34186

…etion (airbytehq#34186) Signed-off-by: Gireesh Sreepathi <[email protected]> Co-authored-by: Sitaram Shelke <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: Cynthia Yin <[email protected]> Co-authored-by: Baz <[email protected]> Co-authored-by: bazarnov <[email protected]> Co-authored-by: kekiss <[email protected]> Co-authored-by: Anatolii Yatsuk <35109939 [email protected]> Co-authored-by: Edward Gao <[email protected]> Co-authored-by: Augustin <[email protected]> Co-authored-by: Joe Reuter <[email protected]> Co-authored-by: Alexandre Cuoci <[email protected]> Co-authored-by: perangel <[email protected]> Co-authored-by: Aaron ("AJ") Steers <[email protected]> Co-authored-by: Ben Church <[email protected]> Co-authored-by: Gireesh Sreepathi <[email protected]>

## What * Resurrect removed signatures to fix compilation errors in Snowflake connector. * Partial revert of airbytehq#34186

…d methods and vars (airbytehq#34467) ## What * Reverting few signature changes from airbytehq#34186

jbfbell added 3 commits January 11, 2024 09:49

Actually Delete S3 Objects

409c357

add svv table check

557ad12

docs update

8110a34

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit connectors/destination/redshift labels Jan 11, 2024

vercel bot deployed to Preview January 11, 2024 19:19 View deployment

jbfbell and others added 9 commits January 11, 2024 11:28

typos and docs

0a4f94c

version updates

95002d5

formatting

98c6a29

docs: Mention that DBT Worker not supported on kubernetes (#34087)

9b2b243

Co-authored-by: Marcos Marx <[email protected]>

🧹 Destination Redshift: clean up DAT classes (#34134)

b82d983

🎉 Airbyte CDK (File-based CDK): Stop the sync if the record could not…

99c35d8

… be parsed (#32589)

🤖 Bump patch version of Python CDK

19142a3

Docs: Updated grammar and formatting for clarity and consistency (#34080

f061bca

)

✨Source Google Ads: Add possibility to sync all connected accounts (#…

cf66abb

…33707)

octavia-squidington-iii added the connectors/source/google-ads label Jan 11, 2024

jbfbell and others added 2 commits January 11, 2024 12:03

Merge branch 'master' into joseph.bell/redshift-check-updates

3c9c1b1

build gradle setting

9aa2ee0

This was linked to issues Jan 11, 2024

[Redshift Dv2] Handle svv_table_info permissions #33552

Closed

redshift-destination: Verify purge stage data flag is honored and files deleted from S3 #33983

Closed

jbfbell added 2 commits January 11, 2024 13:16

Merge branch 'master' of github.com:airbytehq/airbyte into joseph.bel…

1052b89

…l/redshift-check-updates

Merge branch 'master' of github.com:airbytehq/airbyte into joseph.bel…

1fc244c

…l/redshift-check-updates

jbfbell marked this pull request as ready for review January 11, 2024 21:16

jbfbell requested a review from a team as a code owner January 11, 2024 21:16

edgao approved these changes Jan 11, 2024

View reviewed changes

gisripa and others added 6 commits January 12, 2024 15:34

Destination Bigquery: Clean up dependencies with TD/CDK (#34226)

07be323

Signed-off-by: Gireesh Sreepathi <[email protected]>

Destination Snowflake: Cleanup dependencies and upgrade CDK (#34227)

59b70e2

Signed-off-by: Gireesh Sreepathi <[email protected]>

version updates

e082c3e

actually limit stuff

ea989df

remove getStageName method

48d56b5

merge conflicts

0719b69

vercel bot deployed to Preview January 12, 2024 23:38 View deployment

jbfbell added 2 commits January 12, 2024 16:14

formatting

18d0f9f

Merge branch 'master' of github.com:airbytehq/airbyte into joseph.bel…

2c6438d

…l/redshift-check-updates

vercel bot deployed to Preview January 16, 2024 18:23 View deployment

use newly published cdk

89ba758

jbfbell enabled auto-merge (squash) January 16, 2024 20:58

Merge branch 'master' of github.com:airbytehq/airbyte into joseph.bel…

9a705bb

…l/redshift-check-updates

jbfbell merged commit db83e14 into master Jan 16, 2024
26 checks passed

jbfbell deleted the joseph.bell/redshift-check-updates branch January 16, 2024 21:23

cynthiaxyin mentioned this pull request Jan 19, 2024

Redshift: Crash in hasUnprocessedRecords() #34273

Closed

This was referenced Jan 24, 2024

Destination Redshift: Update CDK, revert signatures to add back unused methods and vars #34467

Merged

Partial CDK revert of #34186 #34461

Merged

gisripa added a commit that referenced this pull request Jan 24, 2024

Partial CDK revert of #34186 (#34461)

852cb59

## What * Resurrect removed signatures to fix compilation errors in Snowflake connector. * Partial revert of #34186

gisripa added a commit that referenced this pull request Jan 24, 2024

Destination Redshift: Update CDK, revert signatures to add back unuse…

7123222

…d methods and vars (#34467) ## What * Reverting few signature changes from #34186

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

Destination Redshift: Update CDK, revert signatures to add back unuse…

10539e6

…d methods and vars (airbytehq#34467) ## What * Reverting few signature changes from airbytehq#34186

jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024

Destination Redshift: Update CDK, revert signatures to add back unuse…

6e7b9ba

…d methods and vars (airbytehq#34467) ## What * Reverting few signature changes from airbytehq#34186

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Destination Redshift - additional check method check, fix s3 file deletion #34186

Destination Redshift - additional check method check, fix s3 file deletion #34186

jbfbell commented Jan 11, 2024 •

edited

Loading

vercel bot commented Jan 11, 2024 •

edited

Loading

github-actions bot commented Jan 11, 2024 •

edited by jbfbell

Loading

edgao left a comment

jbfbell commented Jan 12, 2024

jbfbell commented Jan 16, 2024 •

edited by github-actions bot

Loading

Destination Redshift - additional check method check, fix s3 file deletion #34186

Destination Redshift - additional check method check, fix s3 file deletion #34186

Conversation

jbfbell commented Jan 11, 2024 • edited Loading

vercel bot commented Jan 11, 2024 • edited Loading

github-actions bot commented Jan 11, 2024 • edited by jbfbell Loading

Before Merging a Connector Pull Request

edgao left a comment

Choose a reason for hiding this comment

jbfbell commented Jan 12, 2024

jbfbell commented Jan 16, 2024 • edited by github-actions bot Loading

jbfbell commented Jan 11, 2024 •

edited

Loading

vercel bot commented Jan 11, 2024 •

edited

Loading

github-actions bot commented Jan 11, 2024 •

edited by jbfbell

Loading

jbfbell commented Jan 16, 2024 •

edited by github-actions bot

Loading