Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [date 9.11]1.2-dev tke regression: sysbench no-pk point select test reported operation error S3: GetObject, exceeded maximum number of attempts, 3 #18725

Open
1 task done
heni02 opened this issue Sep 12, 2024 · 24 comments
Open
1 task done

[Bug]: [date 9.11]1.2-dev tke regression: sysbench no-pk point select test reported operation error S3: GetObject, exceeded maximum number of attempts, 3 #18725

heni02 opened this issue Sep 12, 2024 · 24 comments
Assignees
@reusee
Labels
kind/bug Something isn't working severity/s0 Extreme impact: Cause the application to break down and seriously affect the use
Milestone
2.1.0

Comments

@heni02
Copy link
Contributor

heni02 commented Sep 12, 2024 •
edited by sukki37
Loading

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Branch Name

1.2-dev,main

Commit ID

2fed24b

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/10813870929/job/30021691067
企业微信截图_0c258cbf-35b5-404c-a5c1-053ea71d0b4e

FATAL: mysql_stmt_execute() returned error 1105 (operation error S3: GetObject, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "https://mo-nightly-gz-1308875761.cos.ap-guangzhou.myqcloud.com/mo-benchmark-10813870929/data/0191e397-5f91-7c0a-a0fa-1b927613232f_00000?x-id=GetObject": dial tcp 169.254.0.47:443: connect: cannot assign requested address) for query 'SELECT c FROM sbtest10 WHERE id=?'

mo log：
https://grafana.ci.matrixorigin.cn/explore?panes={"qFM":{"datasource":"loki","queries":[{"refId":"A","expr":"{namespace=\"mo-branch-nightly-2fed24bdf-20240911\"} |= `S3: GetObject, exceeded maximum number of attempts, 3`","queryType":"range","datasource":{"type":"loki","uid":"loki"},"editorMode":"builder"}],"range":{"from":"1726045516630","to":"1726131916630"}}}&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

sysbench  --mysql-host=172.16.110.95 --mysql-port=6001 --mysql-user=dump --mysql-password=111  oltp_point_select.lua --mysql-db=sysbench_db --tables=10 --table_size=10000000 --threads=100 --time=150 --report-interval=10 --range_selects=off --point_selects=1 --secondary=on  --create_secondary=off --auto_inc=off prepare

sysbench  --mysql-host=172.16.110.95 --mysql-port=6001 --mysql-user=dump --mysql-password=111  oltp_point_select.lua --mysql-db=sysbench_db --tables=10 --table_size=10000000 --threads=100 --time=150 --report-interval=10 --range_selects=off --point_selects=1 --secondary=on  --create_secondary=off --auto_inc=off run

Additional information

No response
@heni02 heni02 added kind/bug Something isn't working severity/s0 Extreme impact: Cause the application to break down and seriously affect the use labels Sep 12, 2024
@heni02 heni02 added this to the 1.3.0 milestone Sep 12, 2024
This was referenced Sep 13, 2024
Merged
Merged
@reusee
Copy link
Contributor

reusee commented Sep 24, 2024

无进展

2 similar comments
@reusee
Copy link
Contributor

reusee commented Sep 29, 2024

无进展

@reusee
Copy link
Contributor

reusee commented Sep 29, 2024

无进展

@reusee reusee mentioned this issue Oct 8, 2024
Merged
7 tasks
@reusee
Copy link
Contributor

reusee commented Oct 11, 2024

无进展

@aressu1985 aressu1985 modified the milestones: 2.0.0, 1.2.3 Oct 14, 2024
@reusee
Copy link
Contributor

reusee commented Oct 16, 2024

无进展

@reusee
Copy link
Contributor

reusee commented Oct 21, 2024

已提pr替换成腾讯云sdk

@sukki37 sukki37 modified the milestones: 1.2.3, 1.2.x Oct 23, 2024
@reusee
Copy link
Contributor

reusee commented Oct 24, 2024

无进展

@aressu1985 aressu1985 modified the milestones: 1.2.x, 2.0.1 Oct 29, 2024
@aressu1985
Copy link
Contributor

repo link:
https://github.com/matrixorigin/mo-nightly-regression/actions/runs/11556246019

@aressu1985
Copy link
Contributor

repo link:
https://github.com/matrixorigin/mo-nightly-regression/actions/runs/11556246019/job/32194544455

image

@reusee reusee mentioned this issue Nov 1, 2024
Merged
7 tasks
@reusee
Copy link
Contributor

reusee commented Nov 6, 2024

已优化，观察中

@fengttt fengttt modified the milestones: 2.0.1, 2.1.0 Nov 8, 2024
@reusee
Copy link
Contributor

reusee commented Nov 11, 2024

无进展

@reusee
Copy link
Contributor

reusee commented Nov 14, 2024

无进展

@reusee
Copy link
Contributor

reusee commented Nov 19, 2024

pr 已合并，观察中

1 similar comment
@reusee
Copy link
Contributor

reusee commented Nov 22, 2024

pr 已合并，观察中

This was referenced Nov 25, 2024
Merged
Merged
@reusee
Copy link
Contributor

reusee commented Nov 28, 2024

已增加连接profile，等复现时再分析

@aressu1985
Copy link
Contributor

repro link:
https://github.com/matrixorigin/mo-nightly-regression/actions/runs/12121603248/job/33824088373

@reusee
Copy link
Contributor

reusee commented Dec 3, 2024

http-conns profile
CN_30393131-3063-6434-3734-303961326266_http-conns_01938ae1-e44d-7877-9115-2c719983570c.gz
CN_30393131-3063-6434-3734-303961326266_http-conns_01938ae3-43fc-7001-b986-fb557fdfaabd.gz

@reusee
Copy link
Contributor

reusee commented Dec 3, 2024

从上面两个 profile 的 diff 看，30s内，http连接计数从 143947 增长到 172187，新增 28240 个连接，和日志里记录的活跃连接数量一致：
Screenshot From 2024-12-03 14-55-20
Screenshot From 2024-12-03 14-55-51

@reusee
Copy link
Contributor

reusee commented Dec 3, 2024

考虑从这个路径入手优化
Screenshot From 2024-12-03 14-58-32

This was referenced Dec 3, 2024
Merged
Merged
Open
@reusee
Copy link
Contributor

reusee commented Dec 6, 2024

修复已合入2.0分支，继续观察

@reusee
Copy link
Contributor

reusee commented Dec 11, 2024

继续观察

@reusee
Copy link
Contributor

reusee commented Dec 16, 2024

如上

This was referenced Dec 17, 2024
Open
Merged
Merged
@reusee
Copy link
Contributor

reusee commented Dec 20, 2024

继续观察

1 similar comment
@reusee
Copy link
Contributor

reusee commented Dec 25, 2024

继续观察

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@reusee reusee

Labels
kind/bug Something isn't working severity/s0 Extreme impact: Cause the application to break down and seriously affect the use
Projects
None yet
Milestone
2.1.0
Development

No branches or pull requests
5 participants
@fengttt @reusee @aressu1985 @sukki37 @heni02