Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to change the offsets of the source database by kcat. #371

Open
lawrenceee1 opened this issue Feb 23, 2022 · 8 comments
Open

Failed to change the offsets of the source database by kcat. #371

lawrenceee1 opened this issue Feb 23, 2022 · 8 comments

Comments

@lawrenceee1
Copy link

I failed to change the offsets of the source database By using the method on the official website.
The following is my operation process.

1.I installed kcat successful.
2.I found out the name of the topic that contains plugin-offsets,it is "connect-offsets-01".
3.I found out the last offset for the given connector, key under which it is stored and identify the partition used to store the offset
/usr/bin/kcat -b 10.7.102.11:9092 -C -t connect-offsets-01 -f 'Partition(%p) %k %s\n'

Partition(6) ["mysqltest0216",{"server":"mysqltest021601"}] {"transaction_id":null,"ts_sec":1644913815,"file":"mysql-bin.002655","pos":1062699299,"row":1,"server_id":10011,"event":3}
Partition(6) ["mysqltest0216",{"server":"mysqltest021601"}] {"transaction_id":null,"ts_sec":1644914105,"file":"mysql-bin.002655","pos":1067626895,"row":1,"server_id":10011,"event":3}
Partition(6) ["mysqltest0216",{"server":"mysqltest021601"}] {"transaction_id":null,"ts_sec":1644979421,"file":"mysql-bin.002656","pos":497395301,"row":1,"server_id":10011,"event":3}

4.I stopped my connector and then executed the following command
echo '["mysqltest0216",{"server":"mysqltest021601"}]|{"ts_sec":1644979430,"file":"mysql-bin.002655","pos":1062699299,"row":1,"server_id":10011,"event":3}' |
/usr/bin/kcat -P -b 10.7.102.11:9092 -t connect-offsets-01 -K | -p 6

The command was ececuted successful,but the offset of the topic is unchanged,it doesn't seem to have the desired effect.

Is there something wrong with my process?
How to change the offsets of the source database?
what is the expected perfect result?
Thanks so much for your answer.

@OneCricketeer
Copy link

Are you sure the command ran correctly? The pipe character after the K option is piping your kcat command into some -p 6 command, which wouldn't be valid.

You'll need to quote your key separator, or use a different one that's not interpreted by the shell

@lawrenceee1
Copy link
Author

Are you sure the command ran correctly? The pipe character after the K option is piping your kcat command into some -p 6 command, which wouldn't be valid.

You'll need to quote your key separator, or use a different one that's not interpreted by the shell

Thank you for your reply.Sorry for the typo here.But in fact,I have this symbol (\)when I execute this command
echo '["mysqltest0216",{"server":"mysqltest021601"}]|{"ts_sec":1644913815,"file":"mysql-bin.002655","pos":1062699299,"row":1,"server_id":10011,"event":3}' | /usr/bin/kcat -P -b 10.7.102.13:9092 -t connect-offsets-01 -K | -p 6

@lawrenceee1
Copy link
Author

When I copy this history command to here,the symbol (\) is automatically lost

@OneCricketeer
Copy link

You can use triple backticks to write code in github markdown

echo '["mysqltest0216",{"server":"mysqltest021601"}]|{"ts_sec":1644913815,"file":"mysql-bin.002655","pos":1062699299,"row":1,"server_id":10011,"event":3}' | /usr/bin/kcat -P -b 10.7.102.13:9092 -t connect-offsets-01 -K \| -p 6

But you could still change the delimiter to not use something interpreted by the shell

@OneCricketeer
Copy link

Overall, I don't think this is a kcat issue if you're able to produce and consume records to the topic... You could be using any other client to write the exact same message, and you'll have the same issues with the connector.

Perhaps open a ticket against the JDBC connector / Debezium repository for the connector you're using?

More specifically, how do you know that the combination of "ts_sec" "file" "pos" are correct?

@lawrenceee1
Copy link
Author

I can't confirm if that is correct,I referenced the information on this site:https://debezium.io/documentation/faq/#how_to_change_the_offsets_of_the_source_database

@OneCricketeer
Copy link

Still, assuming kcat actually produced the correct event, the issue of whether the connector reset, or not, is a Debezium issue, and they have their own issue board which will provide better support for that

@lawrenceee1
Copy link
Author

Thank you so much ,I will submit this question on their site.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants