@ddebowczyk92 GCP -> Dataplex?

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.70%. Comparing base (162e837) to head (724b320).
Report is 3 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

@@           Coverage Diff           @@
##             main    #3043    /-   ##
=======================================
  Coverage   85.70%   85.70%           
=======================================
  Files          54       54           
  Lines        3112     3112           
=======================================
  Hits         2667     2667           
  Misses        445      445           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

My concern is: is this code specific to OpenLineage or certain lineage backend? In case of the latter: are we happy to build and release transports for all the other OL backends in case someone wants it? What are the advantages of openlineage community taking care of this?

I believe there is no problem with us maintaining wrappers (transports are wrappers for specific libraries) for sending events to significant OpenLineage backends. I don't see a difference between DataplexTransport and KafkaTransport here for example, given that they both produce consumable JAR that the transport uses.

Adding this as a separate releasable Maven package is actually a step forward for modularity: even if something breaks inside, it affects only users that use it - it's not the case with Kafka or Kinesis transports now for example.

The disadvantage of this approach is that: we take responsibility for the code without being able to verify its correctness. One can easily make a change within DataplexTransport which breaks the integration while passing all the tests. It's difficult to maintain such package, as even upgrading dependencies like `com.google.cloud:google-cloud-datalineage' is risky.

I don't agree with this, the analogy is comparable to KafkaTransport. We don't test if 'org.apache.kafka:kafka-clients:3.8.0' can emit data to Kafka. If Google breaks their dependency, it will be on Google to fix it. We don't have to do any workarounds for this, we can purely point to the older version of a dependency or disable release for particular transport.

If we want to maintain such code, I think we need an integration test with real DataPlex credentials, same as we have for BigQuery.

@tnazarew is writing integration consumer tests that would be able to cover whether particular transport produces information that's relevant to the particular consumer.