Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Postgres max pool size option not honored #1656

Closed
rubengees opened this issue Oct 13, 2020 · 6 comments
Closed

Postgres max pool size option not honored #1656

rubengees opened this issue Oct 13, 2020 · 6 comments

Comments

@rubengees
Copy link
Contributor

Environment

  • VerneMQ Version: 1.11.0
  • OS: Linux
  • Erlang/OTP version (if building from source): erlang:23-alpine
  • Cluster size/standalone: 1

Expected behaviour

VerneMQ should use the given vmq_diversity.postgres.pool_size or DOCKER_VERNEMQ_VMQ_DIVERSITY__POSTGRES__POOL_SIZE option to limit the database connections to the specified value.

Actual behaviour

Upon setting DOCKER_VERNEMQ_VMQ_DIVERSITY__POSTGRES__POOL_SIZE to 1, VerneMQ uses 10 connections to the database (which is already weird because I only see 5 everywhere in the code). Verified by running SELECT * FROM pg_stat_activity and looking through the result.

I suspected the option to be written incorrectly, but cat /etc/vernemq/vernemq.conf | grep "pool_size" shows vmq_diversity.postgres.pool_size=1.

Is this not currently supported properly (I did not found documentation on that apart from this in the schema) or is the config wrong?

@ioolkos
Copy link
Contributor

ioolkos commented Oct 13, 2020

Thanks, this is a duplicate for #1269 which got closed without further investigation.
But, ok, so we got double confirmation that this gets 10 connection as a default.
Now the question is whether 10 is just the minimal number (for instance enforced by the PG library), or whether this doesn't respect configs at all (also for config value > 10).

@rubengees
Copy link
Contributor Author

I tried around a bit and it seems the config is not respected at all. When I set it to 20, there were still 10 connections.

I also tried overriding this directly in a lua plugin (by copying the default postgres plugin and adding size = 1):

require "auth/auth_commons"
-- import shared database query logic
require "auth/postgres_cockroach_commons"

function auth_on_register(reg)
    return auth_on_register_common(postgres, reg)
end

pool = "auth_postgres"
config = {
    pool_id = pool,
    size = 1
}

postgres.ensure_pool(config)

This also does not change the behaviour.

One interesting effect I saw though is, that my plugin is now run 10 times. I have a log statement at top which normally has been printed only once but now 10 times. Not sure if that connected to the issue.

Is there any known workaround or do you have any idea how I can investigate this further?

@rubengees
Copy link
Contributor Author

rubengees commented Oct 13, 2020

Looks like the 10 comes from here: https://github.com/vernemq/vernemq/blob/master/apps/vmq_diversity/src/vmq_diversity_sup.erl#L77

I can't really read erlang, but it seems that this file is the culprit.

@ioolkos
Copy link
Contributor

ioolkos commented Oct 13, 2020

Technically, it's in the function clause for postgres:

start_all_pools([{pgsql, ProviderConfig}|Rest], Acc) ->

But your totally right. The ProviderConfig there comes in over the vmq_diversity_postgres.erl, but size simply gets reset to default.

Do you want to have a go at it? Can fix it myself, but low prio ;)

@ioolkos
Copy link
Contributor

ioolkos commented Oct 13, 2020

Note to self: we could upgrade to and test epgsql 4.4.0, instead of 4.3.0
https://github.com/epgsql/epgsql/tags

@rubengees
Copy link
Contributor Author

Did a PR here. Seems to work in my environment, but not sure if this has any unintended side effects.

@ioolkos ioolkos closed this as completed Oct 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants