Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Integration][DBT] Fix compatibility with DBT v1.8 #3228

Merged
merged 3 commits into from
Nov 8, 2024
Merged

[Integration][DBT] Fix compatibility with DBT v1.8 #3228

merged 3 commits into from
Nov 8, 2024

Conversation

NJA010
Copy link
Contributor

@NJA010 NJA010 commented Nov 7, 2024 •
edited
Loading

Signed-off-by: Niels Janssen [email protected]

Problem

DbtArtifactProcessor.parse_assertions() reads the node's name from a test_metadata field. However, the schema of the DBT manifest has changed and the name of a node is no longer in the test_metadata field. Instead, name is a field of each node.

Closes: #3222

Solution

Check manifest version whether to select test_metadata or not

One-line summary:

Fix bug due to DBT manifest change in DBT v1.8

Checklist

  • You've signed-off your work
  • Your pull request title follows our guidelines
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • Your comment includes a one-liner for the changelog about the specific purpose of the change (not required for changes to tests, docs, or CI config)
  • You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
  • You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2024 contributors to the OpenLineage project

@NJA010 NJA010 requested a review from a team as a code owner November 7, 2024 09:00
@boring-cyborg boring-cyborg bot added area:integration/common openlineage-integration-common language:python Uses Python programming language labels Nov 7, 2024
@boring-cyborg boring-cyborg
Copy link

boring-cyborg bot commented Nov 7, 2024

Thanks for opening your first OpenLineage pull request! We appreciate your contribution. If you haven't already, please make sure you've reviewed our guide for new contributors (https://github.com/OpenLineage/OpenLineage/blob/main/CONTRIBUTING.md).

@NJA010 NJA010 force-pushed the bug/fix-dbt-manifest branch from 9ca28e7 to 5be48b2 Compare November 7, 2024 09:24
@NJA010 NJA010 force-pushed the bug/fix-dbt-manifest branch from 5be48b2 to 06a81c3 Compare November 7, 2024 09:25
@NJA010 NJA010 marked this pull request as draft November 7, 2024 09:33
@NJA010 NJA010 marked this pull request as ready for review November 7, 2024 10:59
@mobuchowski
Copy link
Member

@NJA010 could you add a test for that (from, for example jaffle shop in 1.8)?

@codecov-commenter
Copy link

codecov-commenter commented Nov 7, 2024 •
edited
Loading

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.75%. Comparing base (49ec821) to head (3ee9176).

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #3228       /-   ##
==========================================
  Coverage   85.73%   85.75%    0.02%     
==========================================
  Files          54       54              
  Lines        3119     3124        5     
==========================================
  Hits         2674     2679        5     
  Misses        445      445

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@boring-cyborg boring-cyborg bot added the area:tests Testing code label Nov 7, 2024
@NJA010
Copy link
Contributor Author

NJA010 commented Nov 7, 2024

@mobuchowski I've copied over the jaffle shop test folder and transformed the manifest and run_results to a reflect a dbt test run using dbt 1.8. Changed some fields to make it compatible, but not all fields (like execution time) make sense. Hope it's OK. Let me know if it's not.

mobuchowski
mobuchowski approved these changes Nov 7, 2024
View reviewed changes
Copy link
Member

@mobuchowski mobuchowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution @NJA010!

@mobuchowski mobuchowski merged commit 0399867 into OpenLineage:main Nov 8, 2024
31 checks passed
@boring-cyborg boring-cyborg
Copy link

boring-cyborg bot commented Nov 8, 2024

Great job! Congrats on your first merged pull request in OpenLineage!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mobuchowski mobuchowski mobuchowski approved these changes

Assignees
No one assigned
Labels
area:integration/common openlineage-integration-common area:tests Testing code language:python Uses Python programming language
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG] DBT integration not compatible with DBT v1.8
3 participants
@NJA010 @mobuchowski @codecov-commenter