Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Destination Redshift: Implement refreshes #40567

Merged
merged 1 commit into from
Jul 12, 2024
Merged

Destination Redshift: Implement refreshes #40567

merged 1 commit into from
Jul 12, 2024

Conversation

edgao
Copy link
Contributor

@edgao edgao commented Jun 26, 2024 •
edited
Loading

new test class (RedshiftS3StagingStorageOperationTest) verifies some of the error handling behavior:

  • creating writing to temp raw tables
  • overwriting the real raw table from the temp raw table
  • copying records from the temp raw table to the real raw table

tested manually in https://cloud.airbyte.com/workspaces/b61bc266-ef3c-460c-af2f-f70da4a2993c/connections/e1e1f841-ca63-4524-9c87-9da81f928a79/job-history with a merge refresh normal sync:

  • no airbyte_tmp tables after successful sync
  • generation IDs correctly incremented, OVERWRITE stream correctly truncated

sample queries:

SELECT 'products' as stream, _airbyte_generation_id, count(1) FROM "integrationtests"."v2_staging_inserts"."products" group by _airbyte_generation_id
UNION ALL
SELECT 'purchases' as stream, _airbyte_generation_id, count(1) FROM "integrationtests"."v2_staging_inserts"."purchases" group by _airbyte_generation_id
UNION ALL
SELECT 'users' as stream, _airbyte_generation_id, count(1) FROM "integrationtests"."v2_staging_inserts"."users" group by _airbyte_generation_id
order by stream;


SELECT 'products' as stream, _airbyte_generation_id, count(1) FROM "integrationtests"."airbyte_internal"."v2_staging_inserts_raw__stream_products" group by _airbyte_generation_id
UNION ALL
SELECT 'purchases' as stream, _airbyte_generation_id, count(1) FROM "integrationtests"."airbyte_internal"."v2_staging_inserts_raw__stream_purchases" group by _airbyte_generation_id
UNION ALL
SELECT 'users' as stream, _airbyte_generation_id, count(1) FROM "integrationtests"."airbyte_internal"."v2_staging_inserts_raw__stream_users" group by _airbyte_generation_id
order by stream;

@vercel Vercel
Copy link

vercel bot commented Jun 26, 2024 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jul 12, 2024 6:03pm

@edgao Graphite App
Copy link
Contributor Author

edgao commented Jun 26, 2024 •
edited
Loading

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @edgao and the rest of your teammates on Graphite Graphite

@edgao edgao mentioned this pull request Jun 26, 2024
Merged
@edgao edgao force-pushed the edgao/redshift_generation_id branch from ddf8630 to d84dd36 Compare June 26, 2024 23:49
@edgao edgao force-pushed the edgao/redshift_generation_id branch from d84dd36 to af94f4d Compare June 26, 2024 23:58
@edgao edgao force-pushed the edgao/redshift_refreshes branch 2 times, most recently from 821a36d to 9429b1b Compare June 27, 2024 01:01
@edgao edgao force-pushed the edgao/redshift_generation_id branch from af94f4d to 9c7b93d Compare June 27, 2024 14:52
@edgao edgao force-pushed the edgao/redshift_generation_id branch from 9c7b93d to 4398a4b Compare June 27, 2024 14:59
@edgao edgao force-pushed the edgao/redshift_generation_id branch from 4398a4b to 20ead36 Compare June 27, 2024 15:01
@edgao edgao force-pushed the edgao/redshift_refreshes branch 3 times, most recently from af77244 to d0a640f Compare June 27, 2024 15:07
@edgao edgao force-pushed the edgao/redshift_generation_id branch from 20ead36 to eae7629 Compare June 27, 2024 16:20
@edgao edgao force-pushed the edgao/redshift_generation_id branch from eae7629 to b73d090 Compare June 27, 2024 16:22
@edgao edgao force-pushed the edgao/redshift_generation_id branch from b73d090 to b6e1b30 Compare June 27, 2024 16:23
@edgao edgao force-pushed the edgao/redshift_generation_id branch from b6e1b30 to e03677b Compare June 27, 2024 16:54
@edgao edgao force-pushed the edgao/redshift_generation_id branch from e03677b to fd5042b Compare June 27, 2024 18:29
@edgao edgao force-pushed the edgao/redshift_generation_id branch from fd5042b to 9b5a05f Compare June 27, 2024 18:38
@edgao edgao requested a review from a team as a code owner July 2, 2024 15:06
@edgao edgao force-pushed the edgao/redshift_refreshes branch from 8701aad to 9f967b4 Compare July 2, 2024 15:19
@edgao edgao force-pushed the edgao/redshift_refreshes branch from 9f967b4 to 80239d9 Compare July 2, 2024 17:12
@edgao edgao force-pushed the edgao/redshift_refreshes branch from 80239d9 to 38c6c77 Compare July 2, 2024 19:54
@edgao edgao force-pushed the edgao/redshift_refreshes branch from 38c6c77 to 68d8bb2 Compare July 2, 2024 22:07
@edgao edgao mentioned this pull request Jul 2, 2024
Merged
This was referenced Jul 2, 2024
Merged
Merged
@edgao edgao force-pushed the edgao/redshift_refreshes branch from 68d8bb2 to 3314670 Compare July 9, 2024 17:11
@edgao edgao force-pushed the edgao/redshift_refreshes branch from 3314670 to 7bed45e Compare July 9, 2024 21:21
@edgao edgao requested a review from a team as a code owner July 12, 2024 17:49
@edgao
Copy link
Contributor Author

edgao commented Jul 12, 2024 •
edited by github-actions bot
Loading

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/9912026151
✅ Successfully published Java CDK version=0.41.2!

@edgao edgao enabled auto-merge (squash) July 12, 2024 18:00
@edgao edgao mentioned this pull request Jul 12, 2024
Merged
@edgao edgao merged commit 355ac15 into master Jul 12, 2024
33 of 34 checks passed
@edgao edgao deleted the edgao/redshift_refreshes branch July 12, 2024 18:23
@edgao edgao mentioned this pull request Jul 26, 2024
Merged
@edgao edgao mentioned this pull request Aug 14, 2024
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stephane-airbyte stephane-airbyte stephane-airbyte approved these changes

Assignees
No one assigned
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit connectors/destination/redshift
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@edgao @stephane-airbyte @octavia-squidington-iii