Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

spec/facet/tests: adding example for change in #2756 #2801

Merged
merged 11 commits into from
Jul 3, 2024
Merged

spec/facet/tests: adding example for change in #2756 #2801

merged 11 commits into from
Jul 3, 2024

Conversation

Sheeri
Copy link
Contributor

@Sheeri Sheeri commented Jun 25, 2024

Problem

The customLineage facet was updated in #2756 but the test case is not up to date.

Closes: #2799

Solution

This example gives all the customLineage for the outputDataSet from this query:
CREATE VIEW CUSTOMER_DISCOUNTS AS
SELECT NAME, AMOUNT_OFF, STARTS_AT, ENDS_AT
FROM DISCOUNTS INNER JOIN CUSTOMERS ON (DISCOUNTS.CUSTOMERS_ID=CUSTOMERS.ID)

Each of the 4 columns (NAME, AMOUNT_OFF, STARTS_AT, ENDS_AT) has 3 inputFields - its own identity from the input, plus 2 indirect inputFields from the join clause, CUSTOMERS_ID and ID.

  • Your change modifies the core OpenLineage model
  • Your change modifies one or more OpenLineage facets

One-line summary:

Updating the customLineage facet test for the new syntax created in #2756

Checklist

  • You've signed-off your work
  • Your pull request title follows our guidelines
  • [ LULZ] Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • Your comment includes a one-liner for the changelog about the specific purpose of the change (not required for changes to tests, docs, or CI config)
  • [N/A] You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
  • [N/A] You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2024 contributors to the OpenLineage project

@Sheeri Sheeri requested a review from a team as a code owner June 25, 2024 14:18
@boring-cyborg boring-cyborg bot added area:spec Specifications and standards for the project area:tests Testing code labels Jun 25, 2024
Sheeri added 2 commits June 25, 2024 10:18
@Sheeri
adding example for 2756
0b894ff 
Signed-off-by: Sheeri K. Cabral <[email protected]>
@Sheeri
adding signoff.
a253023 
Signed-off-by: Sheeri K. Cabral <[email protected]>
@Sheeri Sheeri force-pushed the 2799_add_ex_for_2756 branch from ec0f191 to a253023 Compare June 25, 2024 14:19
tnazarew
tnazarew reviewed Jun 26, 2024
View reviewed changes
Copy link
Contributor

@tnazarew tnazarew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the test update!

@Sheeri
Copy link
Contributor Author

Sheeri commented Jun 28, 2024

@tnazarew the unsuccessful test fails because of diffs that don't actually look different (maybe tab instead of whitespace?) - I'm not sure what to do here:

@@ -139,12  139,12 @@
               }
             ]
           },
-          {   
           {
             "namespace": "SnowflakeOpenLineage",
             "name": "CUSTOMERS",
             "field": "ID",
             "transformations": [
-              { 
               {
                 "type": "INDIRECT",
                 "subtype": "JOIN",
                 "description": "ON (DISCOUNTS.CUSTOMERS_ID=CUSTOMERS.ID)"
@@ -160,7  160,7 @@
                 "type": "INDIRECT",
                 "subtype": "JOIN",
                 "description": "ON (DISCOUNTS.CUSTOMERS_ID=CUSTOMERS.ID)"
-              } 
               }
             ]
           }
         ],

Exited with code exit status 1

tnazarew
tnazarew previously requested changes Jul 2, 2024
View reviewed changes
spec/tests/ColumnLineageDatasetFacet/1.json Outdated Show resolved Hide resolved
@Sheeri Sheeri requested a review from tnazarew July 3, 2024 13:11
@mobuchowski mobuchowski merged commit 253398a into OpenLineage:main Jul 3, 2024
68 checks passed
@mobuchowski
Copy link
Member

Thanks for the contribution @Sheeri

lmassaoy pushed a commit to nubank/NuOpenLineage that referenced this pull request Jul 5, 2024
@Sheeri @lmassaoy
spec/facet/tests: adding example for change in OpenLineage#2756 (Open…
064f84b 
…Lineage#2801)

* adding example for 2756

Signed-off-by: Sheeri K. Cabral <[email protected]>

* adding signoff.

Signed-off-by: Sheeri K. Cabral <[email protected]>

* updating to change subtype to JOIN, and adding subtype IDENTITY for the direct relationships

Signed-off-by: Sheeri K. Cabral <[email protected]>

* changing subType to subtype

Signed-off-by: Sheeri K. Cabral <[email protected]>

* adding _producer and _schemaURL as they are required.

Signed-off-by: Sheeri K. Cabral <[email protected]>

* fixing prettier commit errors

Signed-off-by: Sheeri K. Cabral <[email protected]>

---------

Signed-off-by: Sheeri K. Cabral <[email protected]>
@Sheeri Sheeri deleted the 2799_add_ex_for_2756 branch July 5, 2024 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mobuchowski mobuchowski mobuchowski approved these changes

@tnazarew tnazarew Awaiting requested review from tnazarew

Assignees
No one assigned
Labels
area:spec Specifications and standards for the project area:tests Testing code
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG] Indirect lineage requires an output field
3 participants
@Sheeri @mobuchowski @tnazarew