Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Return valid JSON for failed logical plan serialization #2892

Conversation

arturowczarek
Copy link
Collaborator

@arturowczarek arturowczarek commented Jul 30, 2024

  • The field "preCanonicalized" which may cause infinite recursive calls is skipped during serialization.
  • Modified the test to return truly valid JSON from the serialization method

Problem

  1. LogicalPlanSerializer returns empty string in case of the Exception. Empty string is not a valid JSON and may render the whole event unparseable when used.
  2. The field preCanonicalized can cause StackOverflowError because of infinite recursion

Solution

  1. LogicalPlanSerializer returns "<failed-to-serialize-logical-plan>" for failed serialization instead of empty string.
  2. The field preCanonicalized was excluded from serialization.

Note: All schema changes require discussion. Please link the issue for context.

  • Your change modifies the core OpenLineage model
  • Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).

One-line summary:

Checklist

  • You've signed-off your work
  • Your pull request title follows our guidelines
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • Your comment includes a one-liner for the changelog about the specific purpose of the change (not required for changes to tests, docs, or CI config)
  • You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
  • You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2024 contributors to the OpenLineage project

@arturowczarek arturowczarek requested a review from a team as a code owner July 30, 2024 07:23
@boring-cyborg boring-cyborg bot added area:integration/spark language:java Uses Java programming language labels Jul 30, 2024
@arturowczarek arturowczarek force-pushed the pr/fix-nonrelocated-jackson branch from 91ba04a to dc95d76 Compare July 30, 2024 08:34
@boring-cyborg boring-cyborg bot added the area:tests Testing code label Jul 30, 2024
@arturowczarek arturowczarek force-pushed the pr/fix-nonrelocated-jackson branch from dc95d76 to 220c1e0 Compare July 30, 2024 08:35
@arturowczarek arturowczarek changed the title fix: Relocated and not-relocated Jacksons working together. fix: Fix problems with serializing LogicalPlan with non-relocated Jackson Jul 30, 2024
@arturowczarek arturowczarek force-pushed the pr/fix-nonrelocated-jackson branch from 220c1e0 to 4eabd79 Compare July 30, 2024 08:49
@arturowczarek arturowczarek force-pushed the pr/fix-nonrelocated-jackson branch from 4eabd79 to a22b057 Compare July 30, 2024 11:32
@arturowczarek arturowczarek changed the title fix: Fix problems with serializing LogicalPlan with non-relocated Jackson fix: Return valid JSON for failed logical plan serialization Jul 30, 2024
@arturowczarek arturowczarek force-pushed the pr/fix-nonrelocated-jackson branch 2 times, most recently from f24597d to a618e81 Compare July 31, 2024 07:21
* The field "preCanonicalized" which may cause infinite recursive calls is skipped during serialization.
* Modified the test to return truly valid JSON from the serialization method

Signed-off-by: Artur Owczarek <[email protected]>
@arturowczarek arturowczarek force-pushed the pr/fix-nonrelocated-jackson branch from a618e81 to 7d815f5 Compare August 1, 2024 09:34
@pawel-big-lebowski pawel-big-lebowski merged commit 30eedfa into OpenLineage:main Aug 1, 2024
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:integration/spark area:tests Testing code language:java Uses Java programming language
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants