Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cartridge role reload resets metrics data #334

Closed
DifferentialOrange opened this issue Dec 22, 2021 · 4 comments · Fixed by #341
Closed

Cartridge role reload resets metrics data #334

DifferentialOrange opened this issue Dec 22, 2021 · 4 comments · Fixed by #341
Assignees
@yngvar-antonsson
Labels
bug Something isn't working

Comments

@DifferentialOrange
Copy link
Member

metrics package should support role reload, while it resets all observations and collectors.

Steps to reproduce

  1. Create template application

    cartridge create --name test-app

  2. Change init.lua

    local ok, err = cartridge.cfg({
    roles_reload_allowed = true,
    roles = {
        'cartridge.roles.vshard-storage',
        'cartridge.roles.vshard-router',
        'cartridge.roles.metrics',
        'app.roles.custom',
    },
})

  3. Start an application

    cartridge build
cartridge start -d
cartridge replicasets setup --bootstrap-vshard

  4. Connect to instance

    cartridge enter router

  5. Execute reload

    test-reload.router> metrics = require('metrics')
---
...

test-reload.router> metrics.counter('c'):inc()
---
...

test-reload.router> metrics.collect()
---
- - label_pairs:
     alias: router
   timestamp: 1640183661554368
   value: 1
   metric_name: c
...

test-reload.router> require("cartridge.roles").reload()
---
- true
...

test-reload.router> metrics = require('metrics')
---
...

test-reload.router> metrics.collect()
---
- []
...

test-reload.router> metrics.collectors()
---
- []
...
@DifferentialOrange DifferentialOrange added the bug Something isn't working label Dec 22, 2021
@Totktonada
Copy link
Member

Why do you think that this is a problem?

@yngvar-antonsson yngvar-antonsson self-assigned this Dec 22, 2021
@DifferentialOrange
Copy link
Member Author

Why do you think that this is a problem?

I've discussed it with @yngvar-antonsson and he said this behavior is unexpected.

@yngvar-antonsson
Copy link
Collaborator

Reload of metrics package (without Cartridge) doesn't reset collectors, so cartridge hot reload doesn't have to reset them too.

@DifferentialOrange
Copy link
Member Author

I think

package.loaded['cartridge.hotreload']
if hotreload ~= nil then
    hotreload.whitelist_globals({'__metrics_registry'})`
end

is the simplest and satisfying solution.

DifferentialOrange added a commit to tarantool/crud that referenced this issue Dec 24, 2021
@DifferentialOrange
Preserve stats config and values between reloads
72dd358 
After this patch, statistics module state (enabled/disabled, driver)
preserved between package reloads and Tarantool Cartridge role
reloads [1]. Since `metrics` package do not fully support
preserving registry between role reloads [2] now, this feature doesn't
work for `metrics` driver. Corresponding tests are marked with xfail
until issue is resolved for metrics.

1. https://www.tarantool.io/en/doc/latest/book/cartridge/cartridge_api/modules/cartridge.roles/#reload
2. tarantool/metrics#334

Follows up #224
@DifferentialOrange DifferentialOrange mentioned this issue Dec 24, 2021
Merged
3 tasks
DifferentialOrange added a commit to tarantool/crud that referenced this issue Jan 18, 2022
@DifferentialOrange
Preserve stats config and values between reloads
cd3ac0d 
After this patch, statistics module state (enabled/disabled, driver)
preserved between package reloads and Tarantool Cartridge role
reloads [1]. Since `metrics` package do not fully support
preserving registry between role reloads [2] now, this feature doesn't
work for `metrics` driver. Corresponding tests are marked with xfail
until issue is resolved for metrics.

1. https://www.tarantool.io/en/doc/latest/book/cartridge/cartridge_api/modules/cartridge.roles/#reload
2. tarantool/metrics#334

Follows up #224
@yngvar-antonsson yngvar-antonsson mentioned this issue Jan 21, 2022
Merged
2 tasks
DifferentialOrange added a commit to tarantool/crud that referenced this issue Feb 3, 2023
@DifferentialOrange
test: conditional metrics roles reload skip
f941611 
Before this patch, tests were marked with xfail since there was a bug in
metrics module [1]. This bug is fixes in newer versions, so xfail is
replaced with skip based on metrics version.

1. tarantool/metrics#334

Follows #244
DifferentialOrange added a commit to tarantool/crud that referenced this issue Feb 3, 2023
@DifferentialOrange
test: conditional metrics roles reload skip
b18760c 
Before this patch, tests were marked with xfail since there was a bug in
metrics module [1]. This bug is fixes in newer versions, so xfail is
replaced with skip based on metrics version.

1. tarantool/metrics#334

Follows #244
DifferentialOrange added a commit to tarantool/crud that referenced this issue Feb 13, 2023
@DifferentialOrange
test: conditional metrics roles reload skip
e5d2eb2 
Before this patch, tests were marked with xfail since there was a bug in
metrics module [1]. This bug is fixes in newer versions, so xfail is
replaced with skip based on metrics version.

1. tarantool/metrics#334

Follows #244
DifferentialOrange added a commit to tarantool/crud that referenced this issue Feb 13, 2023
@DifferentialOrange
test: conditional metrics roles reload skip
4a06457 
Before this patch, tests were marked with xfail since there was a bug in
metrics module [1]. This bug is fixes in newer versions, so xfail is
replaced with skip based on metrics version.

1. tarantool/metrics#334

Follows #244
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yngvar-antonsson yngvar-antonsson

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Don't reset collectors when Cartridge roles hot reload tarantool/metrics
4 participants
@Totktonada @filonenko-mikhail @DifferentialOrange @yngvar-antonsson