Destination Redshift: pull in cdk update for refreshes bugfix #42506
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
This was referenced Jul 24, 2024
This stack of pull requests is managed by Graphite. Learn more about stacking. |
2 tasks
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
d59f5d8
to
3d9cf10
Compare
edgao
force-pushed
the
edgao/redshift_refreshes_bugfix
branch
from
ec4786e
to
87cdc45
Compare
stephane-airbyte
approved these changes
Jul 24, 2024
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
3d9cf10
to
341ae0d
Compare
This was referenced Jul 26, 2024
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
341ae0d
to
8a53e44
Compare
edgao
force-pushed
the
edgao/redshift_refreshes_bugfix
branch
from
87cdc45
to
f974dfe
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
8a53e44
to
3922cf7
Compare
edgao
force-pushed
the
edgao/redshift_refreshes_bugfix
branch
from
f974dfe
to
328d04b
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
3922cf7
to
b9bf10d
Compare
edgao
force-pushed
the
edgao/redshift_refreshes_bugfix
branch
from
328d04b
to
247cec9
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
2 times, most recently
from
087b55b
to
7fdef81
Compare
edgao
force-pushed
the
edgao/redshift_refreshes_bugfix
branch
from
247cec9
to
e47bb1c
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
2 times, most recently
from
ad24e27
to
1d5f768
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
1d5f768
to
dd2e935
Compare
edgao
force-pushed
the
edgao/redshift_refreshes_bugfix
branch
from
e47bb1c
to
990fef6
Compare
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
from
dd2e935
to
707b94d
Compare
edgao
force-pushed
the
edgao/redshift_refreshes_bugfix
branch
from
990fef6
to
d78a87e
Compare
edgao
force-pushed
the
edgao/more_refresh_tests
branch
from
600b41a
to
19305f5
Compare
edgao
force-pushed
the
edgao/redshift_refreshes_bugfix
branch
2 times, most recently
from
d469631
to
2a7328b
Compare
edgao
force-pushed
the
edgao/more_refresh_tests
branch
from
19305f5
to
23a01c7
Compare
edgao
force-pushed
the
edgao/redshift_refreshes_bugfix
branch
4 times, most recently
from
e249ff8
to
805863d
Compare
edgao
commented
Aug 14, 2024
| @@ -100,7 100,7 @@ class RedshiftStagingStorageOperation( | |||
| return null | |||
| } | |||
|
|
|||
| return generation.first()[JavaBaseConstants.COLUMN_NAME_AB_GENERATION_ID].asLong() | |||
| return generation.first()[JavaBaseConstants.COLUMN_NAME_AB_GENERATION_ID]?.asLong() ?: 0 | |||
There was a problem hiding this comment.
previously we only ever queried the temp raw table for generation ID, so it was always nonnull. Now we have to handle the real raw table (where pre-generationId-migration rows have null generation ID), so add this logic.
edgao
force-pushed
the
edgao/more_refresh_tests
branch
from
23a01c7
to
5fb7304
Compare
edgao
force-pushed
the
edgao/redshift_refreshes_bugfix
branch
2 times, most recently
from
6c75be4
to
07297ca
Compare
octavia-squidington-iii
added
the
area/documentation
edgao
force-pushed
the
edgao/more_refresh_tests
branch
from
5fb7304
to
95ac402
Compare
edgao
force-pushed
the
edgao/redshift_refreshes_bugfix
branch
from
07297ca
to
bcb8299
Compare
edgao
commented
Aug 16, 2024
| @@ -7,4 7,4 @@ | |||
| // And append the records from the second sync | |||
| {"_airbyte_extracted_at": "1970-01-01T00:00:02.000000Z", "_airbyte_data": {"id1": 1, "id2": 200, "updated_at": "2000-01-02T00:00:00Z", "_ab_cdc_deleted_at": null, "name": "Alice", "address": {"city": "Seattle", "state": "WA"}}, "_airbyte_generation_id": 43, "_airbyte_meta": {"sync_id": 42, "changes": []}} | |||
| {"_airbyte_extracted_at": "1970-01-01T00:00:02.000000Z", "_airbyte_data": {"id1": 1, "id2": 201, "updated_at": "2000-01-02T00:00:00Z", "_ab_cdc_deleted_at": null, "name": "Bob", "address": {"city": "New York", "state": "NY"}}, "_airbyte_generation_id": 43, "_airbyte_meta": {"sync_id": 42, "changes": []}} | |||
| {"_airbyte_extracted_at": "1970-01-01T00:00:02.000000Z", "_airbyte_data": {"id1": 1, "id2": 201, "updated_at": "2000-01-02T00:01:00Z", "_ab_cdc_deleted_at": "1970-01-01T00:00:00Z", "_airbyte_generation_id": 43, "_airbyte_meta": {"sync_id": 42, "changes": []}}} | |||
| {"_airbyte_extracted_at": "1970-01-01T00:00:02.000000Z", "_airbyte_data": {"id1": 1, "id2": 201, "updated_at": "2000-01-02T00:01:00Z", "_ab_cdc_deleted_at": "1970-01-01T00:00:00Z"}, "_airbyte_generation_id": 43, "_airbyte_meta": {"sync_id": 42, "changes": []}} | |||
There was a problem hiding this comment.
no idea how this was working previously. The gen ID meta columns were in the airbyte_data blob. Moved the closing curly brace to the correct position.
gisripa
approved these changes
Aug 19, 2024
Base automatically changed from
edgao/more_refresh_tests
to
edgao/refreshes_recover_from_real_raw_table
August 19, 2024 16:27
edgao
force-pushed
the
edgao/refreshes_recover_from_real_raw_table
branch
2 times, most recently
from
7270022
to
840e346
Compare
Base automatically changed from
edgao/refreshes_recover_from_real_raw_table
to
master
August 19, 2024 18:07
edgao
force-pushed
the
edgao/redshift_refreshes_bugfix
branch
from
bcb8299
to
63c8503
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
How
Review guide
User Impact
Can this PR be safely reverted and rolled back?