This is a program for bulk inspection of redpanda test results, to report on which tests are unstable.
- Python 3
requestsmodule
Generate yourself a Buildkite API key:
- Visit https://buildkite.com/user/api-access-tokens
- Create a new token with scopes as follows:
- Organization access: vectorized
- Read artifacts
- Read builds
- Read build logs
- Read user
Create a ~/.buildkite-auth file:
{
"buildkite_token": "<your token here>",
"artifact_auth": ["vectorized", "<password>"]
}
The artifact auth setting is HTTP basic auth for the vectorized artifact server, see https://vectorizedio.atlassian.net/wiki/spaces/CORE/pages/27033620/Buildkite
Examples:
# Report on the last 24 hours results on `dev`
python main.py dev 24h
# Report on the last week on `test-staging`
python main.py test-staging 1w
Example output:
$ python main.py dev 24h
# ... snip HTTP logs to stderr ...
Ignored test: NodesDecommissioningTest.test_decommissioning_working_node (26/26 runs ignored)
Ignored test: PartitionMovementTest.test_dynamic (26/26 runs ignored)
Ignored test: RpkToolTest.test_consume_from_partition (26/26 runs ignored)
Ignored test: ArchivalTest.test_isolate (26/26 runs ignored)
Unstable test: RaftAvailabilityTest.test_one_node_down (2/26 runs failed)
Most recent failure at 2021-09-28T07:30:30.421Z: MetricCheckFailed('vectorized_raft_received_vote_requests_total', 2.0, 6.0)
in job https://buildkite.com/vectorized/redpanda/builds/2660#7a4a3e18-9500-4faf-ab45-b6ac2d22400d
Unstable test: ctest.test_cluster_rpunit (2/31 runs failed)
Most recent failure at 2021-09-28T07:14:10.501Z: None
in job https://buildkite.com/vectorized/redpanda/builds/2660#5be114bc-30b8-44e8-bb57-569f3bc9cbd5
Unstable test: multiarch-image.multiarch-image (7/11 runs failed)
Most recent failure at None: None
in job https://buildkite.com/vectorized/redpanda/builds/2661#2307e629-e7ff-4408-b1e5-f43d2fcbbe71
Unstable test: k8s-operator.k8s-operator (5/11 runs failed)
Most recent failure at 2021-09-28T08:52:04.384Z: None
in job https://buildkite.com/vectorized/redpanda/builds/2660#3d720045-0651-4fd8-9bec-306291a97888
Unstable test: k8s-helm.k8s-helm (1/11 runs failed)
Most recent failure at 2021-09-28T00:51:11.050Z: None
in job https://buildkite.com/vectorized/redpanda/builds/2643#5f019d04-f25d-4b98-b5aa-decbca507dc8