Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Forward errors and unref owner #788

Open
palateo93 opened this issue Feb 1, 2021 · 0 comments
Open

Forward errors and unref owner #788

palateo93 opened this issue Feb 1, 2021 · 0 comments

Comments

@palateo93
Copy link

Hi,
I'm having problems with Dynomite cluster.
The cluster is composed by one dc and 2 racks with 1 server each.

From Grafana I see a lot of errors as reported below:
grafana_redis_errors

From the logs:

Feb  1 18:00:56 redisa200 dynomite[6495]: [2021-02-01 18:00:56.236] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae5826b70 -1 from '10.57.24.2:60345'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:01 redisa200 dynomite[6495]: [2021-02-01 18:01:01.456] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:42732'>
Feb  1 18:01:01 redisa200 dynomite[6495]: [2021-02-01 18:01:01.456] conn_recv_data:361 <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:42732'> recv eof rb 0 sb 0
Feb  1 18:01:01 redisa200 dynomite[6495]: [2021-02-01 18:01:01.456] core_close:414 close <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:42732'> on event FFFF eof 1 done 1 rb 0 sb 0
Feb  1 18:01:01 redisa200 dynomite[6495]: [2021-02-01 18:01:01.456] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae5826b70 -1 from '10.57.24.2:42732'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:01 redisa200 dynomite[6495]: [2021-02-01 18:01:01.826] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:43423'>
Feb  1 18:01:01 redisa200 dynomite[6495]: [2021-02-01 18:01:01.829] conn_recv_data:361 <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:43423'> recv eof rb 57 sb 5
Feb  1 18:01:01 redisa200 dynomite[6495]: [2021-02-01 18:01:01.829] core_close:414 close <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:43423'> on event 00FF eof 1 done 1 rb 57 sb 5
Feb  1 18:01:01 redisa200 dynomite[6495]: [2021-02-01 18:01:01.829] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae5826b70 -1 from '10.57.24.2:43423'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.269] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:64541'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.307] conn_recv_data:361 <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:64541'> recv eof rb 156 sb 173
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.307] core_close:414 close <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:64541'> on event 00FF eof 1 done 1 rb 156 sb 173
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.307] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae5826b70 -1 from '10.57.24.2:64541'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.340] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:61110'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.341] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ae582c410 20 from '10.57.24.2:47900'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.356] conn_recv_data:361 <CONN_CLIENT 0x564ae582c410 20 from '10.57.24.2:47900'> recv eof rb 156 sb 173
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.356] core_close:414 close <CONN_CLIENT 0x564ae582c410 20 from '10.57.24.2:47900'> on event 00FF eof 1 done 1 rb 156 sb 173
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.356] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae582c410 -1 from '10.57.24.2:47900'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.387] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ae582c410 20 from '10.57.24.2:25249'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.394] conn_recv_data:361 <CONN_CLIENT 0x564ae582c410 20 from '10.57.24.2:25249'> recv eof rb 114 sb 254
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.394] core_close:414 close <CONN_CLIENT 0x564ae582c410 20 from '10.57.24.2:25249'> on event 00FF eof 1 done 1 rb 114 sb 254
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.394] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae582c410 -1 from '10.57.24.2:25249'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.418] conn_recv_data:361 <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:61110'> recv eof rb 348 sb 45
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.418] core_close:414 close <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:61110'> on event 00FF eof 1 done 1 rb 348 sb 45
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.418] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae5826b70 -1 from '10.57.24.2:61110'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.472] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:54618'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.476] conn_recv_data:361 <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:54618'> recv eof rb 63 sb 5
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.476] core_close:414 close <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:54618'> on event 00FF eof 1 done 1 rb 63 sb 5
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.476] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae5826b70 -1 from '10.57.24.2:54618'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.479] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:63288'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.488] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ae582c410 20 from '10.57.24.2:38423'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.488] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ad41e08e0 22 from '10.57.24.2:27381'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.491] conn_recv_data:361 <CONN_CLIENT 0x564ae582c410 20 from '10.57.24.2:38423'> recv eof rb 63 sb 5
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.491] core_close:414 close <CONN_CLIENT 0x564ae582c410 20 from '10.57.24.2:38423'> on event 00FF eof 1 done 1 rb 63 sb 5
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.491] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae582c410 -1 from '10.57.24.2:38423'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.510] conn_recv_data:361 <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:63288'> recv eof rb 156 sb 173
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.510] core_close:414 close <CONN_CLIENT 0x564ae5826b70 16 from '10.57.24.2:63288'> on event 00FF eof 1 done 1 rb 156 sb 173
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.510] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae5826b70 -1 from '10.57.24.2:63288'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.518] conn_recv_data:361 <CONN_CLIENT 0x564ad41e08e0 22 from '10.57.24.2:27381'> recv eof rb 156 sb 173
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.518] core_close:414 close <CONN_CLIENT 0x564ad41e08e0 22 from '10.57.24.2:27381'> on event 00FF eof 1 done 1 rb 156 sb 173
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.518] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ad41e08e0 -1 from '10.57.24.2:27381'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.530] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ad41e08e0 16 from '10.57.24.2:42535'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.531] conn_recv_data:361 <CONN_CLIENT 0x564ad41e08e0 16 from '10.57.24.2:42535'> recv eof rb 53 sb 7
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.531] core_close:414 close <CONN_CLIENT 0x564ad41e08e0 16 from '10.57.24.2:42535'> on event 00FF eof 1 done 1 rb 53 sb 7
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.532] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ad41e08e0 -1 from '10.57.24.2:42535'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.545] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ad41e08e0 16 from '10.57.24.2:14005'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.548] conn_recv_data:361 <CONN_CLIENT 0x564ad41e08e0 16 from '10.57.24.2:14005'> recv eof rb 58 sb 5
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.549] core_close:414 close <CONN_CLIENT 0x564ad41e08e0 16 from '10.57.24.2:14005'> on event 00FF eof 1 done 1 rb 58 sb 5
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.549] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ad41e08e0 -1 from '10.57.24.2:14005'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.573] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ad41e08e0 16 from '10.57.24.2:23622'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.606] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ae5826b70 20 from '10.57.24.2:50494'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.613] conn_recv_data:361 <CONN_CLIENT 0x564ae5826b70 20 from '10.57.24.2:50494'> recv eof rb 114 sb 254
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.613] core_close:414 close <CONN_CLIENT 0x564ae5826b70 20 from '10.57.24.2:50494'> on event 00FF eof 1 done 1 rb 114 sb 254
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.613] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae5826b70 -1 from '10.57.24.2:50494'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.624] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ae5826b70 20 from '10.57.24.2:4975'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.627] conn_recv_data:361 <CONN_CLIENT 0x564ae5826b70 20 from '10.57.24.2:4975'> recv eof rb 53 sb 5
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.627] core_close:414 close <CONN_CLIENT 0x564ae5826b70 20 from '10.57.24.2:4975'> on event 00FF eof 1 done 1 rb 53 sb 5
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.627] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae5826b70 -1 from '10.57.24.2:4975'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.632] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ae5826b70 20 from '10.57.24.2:18597'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.633] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ae582c410 22 from '10.57.24.2:52918'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.633] proxy_accept:203 <CONN_PROXY 0x564ac92f5f40 7 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x564ae5830fc0 23 from '10.57.24.2:2473'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.634] conn_recv_data:361 <CONN_CLIENT 0x564ae5826b70 20 from '10.57.24.2:18597'> recv eof rb 59 sb 5
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.634] core_close:414 close <CONN_CLIENT 0x564ae5826b70 20 from '10.57.24.2:18597'> on event 00FF eof 1 done 1 rb 59 sb 5
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.634] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae5826b70 -1 from '10.57.24.2:18597'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.636] conn_recv_data:361 <CONN_CLIENT 0x564ae582c410 22 from '10.57.24.2:52918'> recv eof rb 63 sb 5
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.636] core_close:414 close <CONN_CLIENT 0x564ae582c410 22 from '10.57.24.2:52918'> on event 00FF eof 1 done 1 rb 63 sb 5
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.636] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae582c410 -1 from '10.57.24.2:52918'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.637] conn_recv_data:361 <CONN_CLIENT 0x564ae5830fc0 23 from '10.57.24.2:2473'> recv eof rb 53 sb 5
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.637] core_close:414 close <CONN_CLIENT 0x564ae5830fc0 23 from '10.57.24.2:2473'> on event 00FF eof 1 done 1 rb 53 sb 5
Feb  1 18:01:02 redisa200 dynomite[6495]: [2021-02-01 18:01:02.637] client_unref_internal_try_put:94 <CONN_CLIENT 0x564ae5830fc0 -1 from '10.57.24.2:2473'> unref owner <POOL 0x564ac92eb550 'dyn_o_mite'>

Here the nodes configurations:

redisa100:~$ cat /etc/dynomite/dynomite.yaml
dyn_o_mite:
  auto_eject_hosts: true
  datacenter: ams1
  dyn_listen: 0.0.0.0:8101
  dyn_port: 8101
  dyn_seed_provider: simple_provider
  dyn_seeds:
  - redisa200:8101:zone2:ams1:0
  listen: 0.0.0.0:8102
  preconnect: true
  rack: zone1
  read_consistency: DC_SAFE_QUORUM
  write_consistency: DC_SAFE_QUORUM
  server_failure_limit: 3
  server_retry_timeout: 30000
  servers:
  - 127.0.0.1:6379:1
  timeout: 300000
  tokens: 0

redisa200:~$ cat /etc/dynomite/dynomite.yaml
dyn_o_mite:
  auto_eject_hosts: true
  datacenter: ams1
  dyn_listen: 0.0.0.0:8101
  dyn_port: 8101
  dyn_seed_provider: simple_provider
  dyn_seeds:
  - redisa100:8101:zone1:ams1:0
  listen: 0.0.0.0:8102
  preconnect: true
  rack: zone2
  read_consistency: DC_SAFE_QUORUM
  write_consistency: DC_SAFE_QUORUM
  server_failure_limit: 3
  server_retry_timeout: 30000
  servers:
  - 127.0.0.1:6379:1
  timeout: 300000
  tokens: 0

I've already read these two pages #487 and #484, but I did understand what I have to base the mbuf_size value on.

Could you please help me?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@palateo93