Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix][io] Upgrade Debezium connector version to 2.6.1.Final #23078

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[fix][io] Upgrade Debezium connector version to 2.6.1.Final #23078

wants to merge 2 commits into from

Conversation

mukesh154
Copy link
Contributor

Fixes #22626

Motivation

To Upgrade debezium version to 2.6.1.Final for all source connectors.

It will address this comment as well.

Modifications

Upgraded debezium version from 1.9.7.Final to 2.6.1.Final all debezium source connectors which includes mysql, postgresql, mongodb, mssql & oracle.

Verifying this change

  • Make sure that the change passes the CI checks.

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

Check the box below or label this PR directly (if you have committer privilege).

Need to update docs?

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jul 26, 2024
lhotari
lhotari reviewed Jul 29, 2024
View reviewed changes
...n/src/test/java/org/apache/pulsar/tests/integration/containers/DebeziumMongoDbContainer.java Outdated Show resolved Hide resolved
...ion/src/test/java/org/apache/pulsar/tests/integration/containers/DebeziumMySQLContainer.java Outdated Show resolved Hide resolved
...rc/test/java/org/apache/pulsar/tests/integration/containers/DebeziumPostgreSqlContainer.java Outdated Show resolved Hide resolved
pom.xml
@@ -201,7 201,7 @@ flexible messaging model and an intuitive client API.</description>
<json-smart.version>2.4.10</json-smart.version>
<opensearch.version>1.2.4</opensearch.version>
<elasticsearch-java.version>8.12.1</elasticsearch-java.version>
<debezium.version>1.9.7.Final</debezium.version>
<debezium.version>2.6.1.Final</debezium.version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to https://debezium.io/ , latest stable Debezium release series is 2.7.x . Could we upgrade directly to 2.7.x instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From 2.6.2.Final onwards the postgres tester is not working. The consumer is expecting a message event just after the insertion into the table but the source is taking some time to process the event. I have tested this locally as well.
I think it is due to PostgreSQL offset flush race condition. More details are here:

  1. Release Notes
  2. DBZ-7816

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From 2.6.2.Final onwards the postgres tester is not working.

DBZ-7816 is fixed in 2.6.2.Final. Perhaps something needs to be adjusted in the Pulsar Debezium integration?

lhotari
lhotari reviewed Jul 29, 2024
View reviewed changes
pom.xml Show resolved Hide resolved
lhotari
lhotari reviewed Jul 30, 2024
View reviewed changes
pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/PulsarDatabaseHistory.java
* topic, and that recovers the history by establishing a Kafka Consumer re-processing all messages on that topic.
*/
@Slf4j
@ThreadSafe
public final class PulsarDatabaseHistory extends AbstractDatabaseHistory {
public final class PulsarDatabaseHistory extends AbstractSchemaHistory {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this change needed? I don't know Debezium very well, but it seems that "database history" and "schema history" are separate concepts.

Copy link
Member

@lhotari lhotari Jul 30, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nvm. found the answer, DBZ-5043 Replace 'database history' with 'database schema history' and
DBZ-5043 Rename DatabaseHistory to SchemaHistory

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be more consistent naming if PulsarDatabaseHistory is renamed to PulsarSchemaHistory. For example, in Debezium, KafkaDatabaseHistory was renamed to KafkaSchemaHistory.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They renamed DatabaseHistory to SchemaHistory in 2.0 version of debezium. See here

@lhotari lhotari added this to the 3.4.0 milestone Jul 30, 2024
@lhotari
Copy link
Member

lhotari commented Jul 30, 2024 •
edited
Loading

I'd recommend running tests for this PR in your own fork so that you get CI feedback without consuming apache/pulsar CI resources.

apache/pulsar pull requests should be first tested in your own fork since the apache/pulsar CI based on
GitHub Actions has constrained resources and quota. GitHub Actions provides separate quota for
pull requests that are executed in a forked repository.

  1. Go to https://github.com/cognitree/pulsar/tree/master and ensure that your master branch is up to date
    with https://github.com/apache/pulsar
    Sync your fork if it's behind.
  2. Open a pull request to your own fork. You can use this link to create the pull request in
    your own fork:
    Create PR in fork for running tests
  3. Edit the description of the pull request [fix][io] Upgrade Debezium connector version to 2.6.1.Final #23078 and add the link to the PR that you opened to your own fork
    so that the reviewer can verify that tests pass in your own fork.
  4. Ensure that tests pass in your own fork. Your own fork will be used to run the tests during the PR review
    and any changes made during the review. You as a PR author are responsible for following up on test failures.
    Please report any flaky tests as new issues at https://github.com/apache/pulsar/issues
    after checking that the flaky test isn't already reported.
  5. When the PR is approved, it will be possible to restart the Pulsar CI workflow within apache/pulsar
    repository by adding a comment "/pulsarbot rerun-failure-checks" to the PR.
    An alternative for the PR approval is to add a ready-to-test label to the PR. This can be done
    by Apache Pulsar committers.
  6. When tests pass on the apache/pulsar side, the PR can be merged by a Apache Pulsar Committer.

@mukesh154
Copy link
Contributor Author

mukesh154 commented Jul 30, 2024 •
edited
Loading

The PR checks passed on my fork for 2.6.1.Final. Check here.

@lhotari
Copy link
Member

lhotari commented Jul 30, 2024

Since upgrading Debezium will be a breaking change. We need to document the upgrade notes for our users. What is the upgrade path and instructions for existing installations? Is it possible to upgrade an existing installation without loosing events etc.?

@mukesh154
Copy link
Contributor Author

Here is the doc for the breaking changes.

@mukesh154 mukesh154 requested a review from lhotari August 20, 2024 16:30
@lhotari lhotari modified the milestones: 4.0.0, 4.1.0 Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lhotari lhotari Awaiting requested review from lhotari

At least 1 approving review is required to merge this pull request.

Assignees

@mukesh154 mukesh154

Labels
doc-not-needed Your PR changes do not impact docs
Projects
None yet
Milestone
4.1.0
Development

Successfully merging this pull request may close these issues.

[Bug] Infinispan Client Hotrod has a vulnerability CVE-2023-4586
2 participants
@mukesh154 @lhotari