KAFKA-12724: Add 2.8.0 to system tests and streams upgrade tests. #10602

kamalcph · 2021-04-28T13:46:49Z

Ran ./gradlew streams:testAll to verify the streams upgrade test.
Ran individual system test in my local setup.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

kamalcph · 2021-04-28T13:49:25Z

...ms/upgrade-system-tests-28/src/test/java/org/apache/kafka/streams/tests/SmokeTestClient.java

+            }
+        });
+
+        streams.setUncaughtExceptionHandler(e -> {


Updated this part of the code as the streams#setUncaughtExceptionHandler(Thread.UncaughtExceptionHandler handler) method is deprecated

kamalcph · 2021-04-28T13:49:45Z

@chia7712 @mjsax @ijuma
Ping for review

mjsax · 2021-04-28T19:54:02Z

...upgrade-system-tests-28/src/test/java/org/apache/kafka/streams/tests/StreamsUpgradeTest.java

+
+        final Properties streamsProperties = Utils.loadProps(propFileName);
+
+        System.out.println("StreamsTest instance started (StreamsUpgradeTest v2.7)");


Should be updated to 2.8

mjsax · 2021-04-28T19:54:54Z

...upgrade-system-tests-28/src/test/java/org/apache/kafka/streams/tests/StreamsUpgradeTest.java

+
+            @Override
+            public void init(final ProcessorContext context) {
+                System.out.println("[2.6] initializing processor: topic=data taskId="   context.taskId());


Should be updated to 2.8 (Seem to be a c&p error) could you fix this in 2.7 and update to 2.7 ?

Maybe worth to introduce a variable that encodes the version?

Updated the version to 2.8.

mjsax · 2021-04-28T19:55:40Z

\cc @vvcephei who was RM for 2.8 for tracking.

tests/docker/Dockerfile

jolshan · 2021-04-29T16:38:52Z

Can we also add some of the newer versions todowngrade_test.py? We've fallen behind on some of the versions there.

jolshan · 2021-05-04T21:20:20Z

tests/kafkatest/tests/streams/streams_upgrade_test.py


 # broker 0.10.0 is not compatible with newer Kafka Streams versions
 broker_upgrade_versions = [str(LATEST_0_10_1), str(LATEST_0_10_2), str(LATEST_0_11_0), str(LATEST_1_0), str(LATEST_1_1), \
                           str(LATEST_2_0), str(LATEST_2_1), str(LATEST_2_2), str(LATEST_2_3), \
-                           str(LATEST_2_4), str(LATEST_2_5), str(LATEST_2_6), str(LATEST_2_7), str(DEV_BRANCH)]
+                           str(LATEST_2_4), str(LATEST_2_5), str(LATEST_2_6), str(LATEST_2_7), str(LATEST_2_8), str(DEV_BRANCH)]


I noticed we are updating the versions for one test (test_upgrade_downgrade_brokers), but not for the other one (test_metadata_upgrade)

The metadata_1_versions and metadata_2_versions are the versions used in this test. Should we keep updating those as well? I'm not super familiar with the streams tests here, only noticed this when running the system test files changed in this PR.

The tests that were not ignored passed btw :)

Added v2.0 and above to the metadata_upgrade test. As per KIP-268, one rolling restart is enough for apps upgrading from 2.x to the latest version. I kept the test behavior unchanged, we can fix this later if required in a follow-up PR. Yeah, the upgrade_downgrade_brokers test was already disabled, so adding versions to this test won't help.

Sorry for late reply.

vvcephei · 2021-07-06T16:44:20Z

Hi @kamalcph , what's the status of this PR? I noticed there are some unresolved code review comments, and there are some conflicts.

kamalcph · 2021-07-07T02:53:05Z

Hi @kamalcph , what's the status of this PR? I noticed there are some unresolved code review comments, and there are some conflicts.

I'll address the pending review comments and resolve the conflicts.

kamalcph · 2021-07-08T09:17:36Z

@vvcephei
Resolved the conflicts. Ran the below tests:

./gradlew :streams:testAll
 TC_PATHS="tests/kafkatest/tests/streams/streams_upgrade_test.py" bash tests/docker/run_tests.sh

The metadata_upgrade test fails for versions above 2.4.1 with Could not detect Kafka Streams version 3.1.0-SNAPSHOT. Please take a look when you get a chance.

vvcephei · 2021-07-08T21:11:16Z

Thanks, @kamalcph ! I'll take a look now.

vvcephei · 2021-07-08T21:28:46Z

Hmm, I had some trouble with the required pip version... I went ahead and kicked off https://jenkins.confluent.io/job/system-test-kafka-branch-builder/4586/ while I debug my local env.

vvcephei · 2021-07-08T21:42:27Z

I haven't verified it, but I think your problem is that you needed to do ./gradlew clean systemTestLibs instead of :streams:testAll.

mjsax · 2021-08-03T21:36:19Z

tests/kafkatest/tests/streams/streams_upgrade_test.py


 metadata_1_versions = [str(LATEST_0_10_0)]
 metadata_2_versions = [str(LATEST_0_10_1), str(LATEST_0_10_2), str(LATEST_0_11_0), str(LATEST_1_0), str(LATEST_1_1)]
+metadata_3_10_versions = [str(LATEST_2_0), str(LATEST_2_1), str(LATEST_2_2), str(LATEST_2_3), str(LATEST_2_4),


What is 3_10 ?

It's probably a typo, but it didn't matter, since this was the wrong test to add new versions to, anyway. I backed out these changes and added the missing versions to streams_application_upgrade_test.py.

mjsax · 2021-08-03T21:37:53Z

...ms/upgrade-system-tests-28/src/test/java/org/apache/kafka/streams/tests/SmokeTestClient.java

+            System.out.println(name   ": FATAL: An unexpected exception is encountered: "   e);
+            e.printStackTrace(System.out);
+            uncaughtException = true;
+            return StreamsUncaughtExceptionHandler.StreamThreadExceptionResponse.SHUTDOWN_CLIENT;


Wondering if we should stop the whole app for this case? \cc @vvcephei

That probably makes more sense, but this more closely mimics the previous behavior of the test. I think I'll leave it to someone who's feeling energetic to make this change and test it.

mjsax · 2021-08-03T21:39:41Z

streams/upgrade-system-tests-28/src/test/java/org/apache/kafka/streams/tests/SmokeTestUtil.java

+                    @Override
+                    public void init(final ProcessorContext context) {
+                        super.init(context);
+                        System.out.println("[DEV] initializing processor: topic="   topic   " taskId="   context.taskId());


Why DEV -- should be 2.8?

Good eye :)

vvcephei · 2021-08-04T04:30:35Z

Ok, I figured it out. The problem was that this change was adding the new version(s) to the metadata upgrade test, but there isn't a metadata upgrade between 2.x and 3.0. Since we're down to the wire now, I just went ahead and pushed a fix to this branch.

It passes for me locally now:

[john@arcturus kafka]$ TC_PATHS="tests/kafkatest/tests/streams/streams_application_upgrade_test.py::StreamsUpgradeTest.test_app_upgrade" _DUCKTAPE_OPTIONS='--parameters '\''{"from_version":"2.8.0","to_version":"3.1.0-SNAPSHOT","bounce_type":"full"}'\' bash tests/docker/run_tests.sh

docker exec ducker01 bash -c "cd /opt/kafka-dev && ducktape --cluster-file /opt/kafka-dev/tests/docker/build/cluster.json  ./tests/kafkatest/tests/streams/streams_application_upgrade_test.py::StreamsUpgradeTest.test_app_upgrade --parameters '{"from_version":"2.8.0","to_version":"3.1.0-SNAPSHOT","bounce_type":"full"}'"
[INFO:2021-08-03 21:27:59,868]: starting test run with session id 2021-08-03--012...
[INFO:2021-08-03 21:27:59,868]: running 1 tests...
[INFO:2021-08-03 21:27:59,869]: Triggering test 1 of 1...
[INFO:2021-08-03 21:27:59,878]: RunnerClient: Loading test {'directory': '/opt/kafka-dev/tests/kafkatest/tests/streams', 'file_name': 'streams_application_upgrade_test.py', 'cls_name': 'StreamsUpgradeTest', 'method_name': 'test_app_upgrade', 'injected_args': {'from_version': '2.8.0', 'to_version': '3.1.0-SNAPSHOT', 'bounce_type': 'full'}}
[INFO:2021-08-03 21:27:59,883]: RunnerClient: kafkatest.tests.streams.streams_application_upgrade_test.StreamsUpgradeTest.test_app_upgrade.from_version=2.8.0.to_version=3.1.0-SNAPSHOT.bounce_type=full: Setting up...
[INFO:2021-08-03 21:27:59,884]: RunnerClient: kafkatest.tests.streams.streams_application_upgrade_test.StreamsUpgradeTest.test_app_upgrade.from_version=2.8.0.to_version=3.1.0-SNAPSHOT.bounce_type=full: Running...
[INFO:2021-08-03 21:29:44,814]: RunnerClient: kafkatest.tests.streams.streams_application_upgrade_test.StreamsUpgradeTest.test_app_upgrade.from_version=2.8.0.to_version=3.1.0-SNAPSHOT.bounce_type=full: PASS
[INFO:2021-08-03 21:29:44,816]: RunnerClient: kafkatest.tests.streams.streams_application_upgrade_test.StreamsUpgradeTest.test_app_upgrade.from_version=2.8.0.to_version=3.1.0-SNAPSHOT.bounce_type=full: Tearing down...
[INFO:2021-08-03 21:30:03,898]: RunnerClient: kafkatest.tests.streams.streams_application_upgrade_test.StreamsUpgradeTest.test_app_upgrade.from_version=2.8.0.to_version=3.1.0-SNAPSHOT.bounce_type=full: Summary: 
[INFO:2021-08-03 21:30:03,899]: RunnerClient: kafkatest.tests.streams.streams_application_upgrade_test.StreamsUpgradeTest.test_app_upgrade.from_version=2.8.0.to_version=3.1.0-SNAPSHOT.bounce_type=full: Data: None
================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.8.1
session_id:       2021-08-03--012
run time:         2 minutes 4.072 seconds
tests run:        1
passed:           1
failed:           0
ignored:          0
================================================================================
test_id:    kafkatest.tests.streams.streams_application_upgrade_test.StreamsUpgradeTest.test_app_upgrade.from_version=2.8.0.to_version=3.1.0-SNAPSHOT.bounce_type=full
status:     PASS
run time:   2 minutes 4.014 seconds
--------------------------------------------------------------------------------

vvcephei · 2021-08-04T04:31:18Z

I went ahead and kicked off a full system test build to evaluate the other tests that got changed in this PR: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/4633/console

EDIT: Results: http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/2021-08-04--001.system-test-kafka-branch-builder--1628079409--kamalcph--KAFKA-12724--fbdce206b/report.html

kamalcph · 2021-08-04T06:42:43Z

@vvcephei
Thanks for figuring out the issue. Feel free to take over this patch.

vvcephei · 2021-08-04T13:53:09Z

Thanks @kamalcph . Sorry it took me so long to look into it.

…ests

vvcephei · 2021-08-04T14:29:58Z

There were a number of failures in the prior run. Some were timeouts, and some of the Streams failures were due to the HA assignor giving only standbys to some nodes. I spot-checked by re-running a few of the tests locally and confirmed that they were flaky failures.

I re-ran the tests (https://jenkins.confluent.io/job/system-test-kafka-branch-builder/4635/)

with "spot instance" disabled
having added a change to the Streams tests intended to disable the HA assignor logic in favor of a one-shot balanced assignment.

I'll keep an eye on that test run to make sure it's clean.

vvcephei · 2021-08-04T22:29:26Z

There were still 21 test failures, but they do not look related to this change. There are also fixes ongoing in parallel to other tests, so I'm going to go ahead and merge this one. If it turns out to be the cause of new problems, we should just revert it.

…0602) Also adjusted the acceptable recovery lag to stabilize Streams tests. Reviewers: Justine Olshan <[email protected]>, Matthias J. Sax <[email protected]>, John Roesler <[email protected]>

…ache#10602) Also adjusted the acceptable recovery lag to stabilize Streams tests. Reviewers: Justine Olshan <[email protected]>, Matthias J. Sax <[email protected]>, John Roesler <[email protected]>

kamalcph commented Apr 28, 2021

View reviewed changes

mjsax reviewed Apr 28, 2021

View reviewed changes

kamalcph commented Apr 29, 2021

View reviewed changes

tests/docker/Dockerfile Outdated Show resolved Hide resolved

kamalcph mentioned this pull request Apr 29, 2021

MINOR: Update tests to include the 2.8.0 release #10562

Closed

jolshan reviewed May 4, 2021

View reviewed changes

kamalcph added 4 commits July 8, 2021 10:08

KAFKA-12724: Add 2.8.0 to system tests and streams upgrade tests.

1c0cd74

Addressed the review comments.

b7edf8a

Adding version 2.6, 2.7 and 2.8 to downgrade test.

09aa5e2

Add v2.x and above to metadata_upgrade test.

20406c9

kamalcph force-pushed the KAFKA-12724 branch from d05215b to 20406c9 Compare July 8, 2021 09:09

mjsax reviewed Aug 3, 2021

View reviewed changes

fix tests

fbdce20

disable HA rebalance logic in favor of static assignments in system t…

4872bef

…ests

vvcephei merged commit a103c95 into apache:trunk Aug 4, 2021

kamalcph deleted the KAFKA-12724 branch August 9, 2021 06:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-12724: Add 2.8.0 to system tests and streams upgrade tests. #10602

KAFKA-12724: Add 2.8.0 to system tests and streams upgrade tests. #10602

kamalcph commented Apr 28, 2021

kamalcph Apr 28, 2021

kamalcph commented Apr 28, 2021

mjsax Apr 28, 2021

kamalcph Apr 29, 2021

mjsax Apr 28, 2021

kamalcph Apr 29, 2021

mjsax commented Apr 28, 2021

jolshan commented Apr 29, 2021

jolshan May 4, 2021

kamalcph Jul 8, 2021

vvcephei commented Jul 6, 2021

kamalcph commented Jul 7, 2021

kamalcph commented Jul 8, 2021 •

edited

Loading

vvcephei commented Jul 8, 2021

vvcephei commented Jul 8, 2021

vvcephei commented Jul 8, 2021 •

edited

Loading

mjsax Aug 3, 2021

vvcephei Aug 4, 2021

mjsax Aug 3, 2021

vvcephei Aug 4, 2021

mjsax Aug 3, 2021

vvcephei Aug 4, 2021

vvcephei commented Aug 4, 2021

vvcephei commented Aug 4, 2021 •

edited

Loading

kamalcph commented Aug 4, 2021 •

edited

Loading

vvcephei commented Aug 4, 2021

vvcephei commented Aug 4, 2021 •

edited

Loading

vvcephei commented Aug 4, 2021


		final Properties streamsProperties = Utils.loadProps(propFileName);

		System.out.println("StreamsTest instance started (StreamsUpgradeTest v2.7)");

KAFKA-12724: Add 2.8.0 to system tests and streams upgrade tests. #10602

KAFKA-12724: Add 2.8.0 to system tests and streams upgrade tests. #10602

Conversation

kamalcph commented Apr 28, 2021

Committer Checklist (excluded from commit message)

Choose a reason for hiding this comment

kamalcph commented Apr 28, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjsax commented Apr 28, 2021

jolshan commented Apr 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vvcephei commented Jul 6, 2021

kamalcph commented Jul 7, 2021

kamalcph commented Jul 8, 2021 • edited Loading

vvcephei commented Jul 8, 2021

vvcephei commented Jul 8, 2021

vvcephei commented Jul 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vvcephei commented Aug 4, 2021

vvcephei commented Aug 4, 2021 • edited Loading

kamalcph commented Aug 4, 2021 • edited Loading

vvcephei commented Aug 4, 2021

vvcephei commented Aug 4, 2021 • edited Loading

vvcephei commented Aug 4, 2021

kamalcph commented Jul 8, 2021 •

edited

Loading

vvcephei commented Jul 8, 2021 •

edited

Loading

vvcephei commented Aug 4, 2021 •

edited

Loading

kamalcph commented Aug 4, 2021 •

edited

Loading

vvcephei commented Aug 4, 2021 •

edited

Loading