Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multple persistent connections to the same Redis host with different databases: database number changes #1920

Open
1 of 2 tasks
binary-data opened this issue Jan 29, 2021 · 12 comments

Comments

@binary-data
Copy link

Expected behaviour

Redis documentation for the SELECT command says that

New connections always use the database 0

So subsequent SELECT commands on different connections should not change the database number for other connections, even if the database 0 was not explicitly selected.

And if the database for connection was changed for any reason, Redis::getDbNum() method should return database number being used.

Actual behaviour

I have the following script which runs in PHP-FPM:

<?php

$redis0 = new Redis();
$redis0->pconnect('redis', 6379, 0.0, 'redis0');
# No explicit select 0 here

$redis1 = new Redis();
$redis1->pconnect('redis', 6379, 0.0, 'redis1');
$redis1->select(1);

$redis2 = new Redis();
$redis2->pconnect('redis', 6379, 0.0, 'redis2');
$redis2->select(2);

echo $redis0->getDbNum();
$redis0->set('REDIS_0', '0');

echo $redis1->getDbNum();
$redis1->set('REDIS_1', '1');

echo $redis2->getDbNum();
$redis2->set('REDIS_2', '2');

On the first and second requests the keys appear in correct databases (MONITOR output from redis-cli):

# Request 1
1611942079.015152 [0 172.18.0.9:45512] "SELECT" "1"
1611942079.016807 [0 172.18.0.9:45514] "SELECT" "2"
1611942079.017168 [0 172.18.0.9:45510] "SET" "REDIS_0" "0"
1611942079.017464 [1 172.18.0.9:45512] "SET" "REDIS_1" "1"
1611942079.017689 [2 172.18.0.9:45514] "SET" "REDIS_2" "2"

1611942219.691163 [2 172.18.0.1:47622] "flushall" # flushall from redis-cli

# Request 1
1611942223.955243 [0 172.18.0.9:45592] "SELECT" "1"
1611942223.955885 [0 172.18.0.9:45863] "SELECT" "2"
1611942223.956078 [0 172.18.0.9:45590] "SET" "REDIS_0" "0"
1611942223.956166 [1 172.18.0.9:45592] "SET" "REDIS_1" "1"
1611942223.956259 [2 172.18.0.9:45863] "SET" "REDIS_2" "2"

Checking the keys in redis-cli:

127.0.0.1:63799> keys *
1) "REDIS_0"
127.0.0.1:63799> select 1
OK
127.0.0.1:63799[1]> keys *
1) "REDIS_1"
127.0.0.1:63799[1]> select 2
OK
127.0.0.1:63799[2]> keys *
1) "REDIS_2"

However, on the third request (without flushall between requests) the "REDIS_0" key appears in the database 2:

1611942329.811933 [2 172.18.0.9:45514] "ECHO" "phpredis:601449b9c6073:1867ae6f"
1611942329.812692 [1 172.18.0.9:45512] "ECHO" "phpredis:601449b9c6586:3eac0768"
1611942329.813386 [1 172.18.0.9:45512] "SELECT" "1"
1611942329.813804 [0 172.18.0.9:45510] "ECHO" "phpredis:601449b9c6a0c:1bb3c7f6"
1611942329.814144 [0 172.18.0.9:45510] "SELECT" "2"
1611942329.814439 [2 172.18.0.9:45514] "SET" "REDIS_0" "0"
1611942329.814647 [1 172.18.0.9:45512] "SET" "REDIS_1" "1"
1611942329.814945 [2 172.18.0.9:45510] "SET" "REDIS_2" "2"

Checking the keys:

127.0.0.1:63799> keys *
1) "REDIS_0"
127.0.0.1:63799> select 1
OK
127.0.0.1:63799[1]> keys *
1) "REDIS_1"
127.0.0.1:63799[1]> select 2
OK
127.0.0.1:63799[2]> keys *
1) "REDIS_2"
2) "REDIS_0"

Meanwhile, during all the requests $redisN->getDbNum(); outputs correct database numbers: 0 1 2.

I'm seeing this behaviour on

  • OS: Linux 5.10.9-201.fc33.x86_64 x86_64 (redis docker image)
  • PHP: 7.4.13
  • Redis: 6.0.8
  • phpredis: 5.3.2

I've checked

  • There is no similar issue from other users
  • Issue isn't fixed in develop branch - not sure how do I check that
@stuntsman
Copy link

1, found the same issue in our environment.

@sztanpet
Copy link

sztanpet commented Mar 4, 2021

This issue still persists,
this fact either needs documentation with big bold letters, or the module must track the database selected by the connection somehow. We ran into this issue with the module being used as the session handler with options persistent=1&database=2.
The application was using persistent connections and expecting database 0 to be selected, but sometimes got the connection used for session handling with database 2 selected.

@arvincheung-madhead
Copy link

issue still persists on phpredis: 5.3.4

@michalkkkd
Copy link

For me, it looks like last selected database is chosen in all connections.

@binary-data
Copy link
Author

For anyone with the same problem: try to avoid using databases when possible. If you need logical separation of data (e.g. session storage vs cache vs Prometheus data), spin up another Redis instance. It doesn't add much overhead, but it isolates your data and makes debugging so much easier.

@cfgxy
Copy link

cfgxy commented Mar 20, 2023

ini_set('redis.pconnect.pooling_enabled', 0); // Default is 1

Works as expected after set "redis.pconnect.pooling_enabled=0".

I think, "redis.pconnect.pooling_enabled=1" means sharing idle redis connections across php-fpm progress by host and port.

  1. When redis.pconnect.pooling_enabled=0, the pid and client-id bound in pair, and when redis.pconnect.pooling_enabled=1, client-id is random but reused in a static range.
  2. $redis->getDbNum() only shows you the dbIndex recorded in PHP variable, not in redis connection.

I found commit 8206b14749e2583895023312c2143116c2480a50 caused this problem

My Suggest:

  • The default value of "redis.pconnect.pooling_enabled" in phpredis should changed to 0, not 1.
    In most scenarios, a Redis Connection Pool is not necessary, a Long-Lived Connection bound to process is match with PHP's common language conventions and works well.
  • Connection reuse across in pconnect pool, should think about persistent_id

This is my test code:

<?php

// Switch this config and refresh page to test.
// =0; getDbNum() keep same to getDbIndexFromConnection()
// =1; getDbNum() same as getDbIndexFromConnection() on first time, but show random result after.
ini_set('redis.pconnect.pooling_enabled', 0);

function makeConnection($id) {
    $redis = new Redis();
    // Change to your own redis params
    $redis->pconnect('127.0.0.1', 6379, 5, $id);
    return $redis;
}

/**
 * Show Redis ClientID
 */
function getConnectionId($redis) {
    return $redis->client('ID');
}

/**
 * Retrieve  the REAL dbIndex from Redis Connection
 */
function getDbIndexFromConnection($redis) {
    $conns = $redis->client('LIST');
    $clientId = $redis->client('ID');
    foreach ($conns as $conn) {
        if ($conn['id'] === $clientId) {
            return $conn['db'];
        }
    }
    return null;
}

$redis0 = makeConnection('redis0');
$redis1 = makeConnection('redis1');
$redis1->select(1);
$redis2 = makeConnection('redis2');
$redis2->select(2);


echo 'PID: ', getmypid(), PHP_EOL;

echo
    getConnectionId($redis0), ' ',
    $redis0->getDbNum(), ' ',
    getDbIndexFromConnection($redis0), PHP_EOL;
$redis0->set('REDIS_0', '0');

echo
    getConnectionId($redis1), ' ',
    $redis1->getDbNum(), ' ',
    getDbIndexFromConnection($redis1), PHP_EOL;
$redis1->set('REDIS_1', '1');

echo
    getConnectionId($redis2), ' ',
    $redis2->getDbNum(), ' ',
    getDbIndexFromConnection($redis2), PHP_EOL;
$redis2->set('REDIS_2', '2');

@rgrellmann
Copy link

rgrellmann commented Jul 30, 2023

I can still reproduce this behaviour and I consider this a critical bug. At least it should be documented in the description of pconnect().

I am not happy with the recommendation to not use different DBs because saving everything to DB 0 makes it impossible to use FLUSHDB to quickly delete logical portions of the stored data (and other things).

If turning off persistent connections and using several Redis instances, instead of using one existing connection (from the pool) or connecting one single time I would have to connect (for instance) 4 Times during the handling of one Request (if I want to access 4 different instances instead of 4 DBs). This has a huge performance impact, as connecting (an sending a subsequent AUTH command) is probably the slowest thing many scripts do.

Please fix this, my recommendation would be to save the selected DB alongside the connection and if it is not 0, issue a "SELECT 0" before handing a connection from the pool over to another PHP script.

@ruudk
Copy link
Contributor

ruudk commented Jul 31, 2023

Does this issue also happen on develop? It seems that 6.0.0RC1 was tagged 2 days ago (not on PECL though).

@cfgxy
Copy link

cfgxy commented Aug 1, 2023

If turning off persistent connections and using several Redis instances, instead of using one existing connection (from the pool) or connecting one single time I would have to connect (for instance) 4 Times during the handling of one Request (if I want to access 4 different instances instead of 4 DBs). This has a huge performance impact, as connecting (an sending a subsequent AUTH command) is probably the slowest thing many scripts do.

ini_set('redis.pconnect.pooling_enabled', 0); 

Disable pconnect pooling should fix your problem, it just disable pconnect pooling, pconnect is still a long-live connection, the difference is: with redis.pconnect.pooling_enabled=1 connection shares across php-fpm, with redis.pconnect.pooling_enabled=0 connection bound to php-fpm process which created it.

@adri
Copy link

adri commented Aug 11, 2023

Currently, I'm debugging an issue where cache keys were written into the wrong database after adding a new persistent connection (separate persistent_id). After reading about this issue, I'm pretty sure that this is related.

This code seems to not use the persistent_id if redis.pconnect.pooling_enabled is enabled:

phpredis/library.c

Lines 3090 to 3119 in 008ec53

if (redis_sock->persistent) {
if (INI_INT("redis.pconnect.pooling_enabled")) {
p = redis_sock_get_connection_pool(redis_sock);
if (zend_llist_count(&p->list) > 0) {
redis_sock->stream = *(php_stream **)zend_llist_get_last(&p->list);
zend_llist_remove_tail(&p->list);
if (redis_sock_check_liveness(redis_sock) == SUCCESS) {
return SUCCESS;
}
p->nb_active--;
}
int limit = INI_INT("redis.pconnect.connection_limit");
if (limit > 0 && p->nb_active >= limit) {
redis_sock_set_err(redis_sock, "Connection limit reached", sizeof("Connection limit reached") - 1);
return FAILURE;
}
gettimeofday(&tv, NULL);
persistent_id = strpprintf(0, "phpredis_%ld%ld", tv.tv_sec, tv.tv_usec);
} else {
if (redis_sock->persistent_id) {
persistent_id = strpprintf(0, "phpredis:%s:%s", host, ZSTR_VAL(redis_sock->persistent_id));
} else {
persistent_id = strpprintf(0, "phpredis:%s:%f", host, redis_sock->timeout);
}
}
}

  • If redis.pconnect.pooling_enabled is enabled:
    persistent_id = strpprintf(0, "phpredis_%ld%ld", tv.tv_sec, tv.tv_usec);
  • If redis.pconnect.pooling_enabled is disabled and a persistent_id is set:
    persistent_id = strpprintf(0, "phpredis:%s:%s", host, ZSTR_VAL(redis_sock->persistent_id));

Is this not a bug where the persistent_id should be taken into account if redis.pconnect.pooling_enabled is enabled?

What I don't understand yet: what are the implications of disabling redis.pconnect.pooling_enabled when running in PHP-FPM? Will we get issues because multiple workers reuse the same connection?

@uncaught
Copy link

uncaught commented Sep 6, 2023

This hit us when trying to use a dedicated database for a Symfony RedisSessionHandler. Cache keys kept appearing in the wrong database.

In fact, only keys for db 0 end up in db 1, not the other way around. I figure this is because Symfony uses $params['dbindex'] && !$redis->select($params['dbindex']), so only selects if it's not the db 0.

I've asked if they would consider a workaround in Symfony, too.

thor-son added a commit to thor-son/phpredis that referenced this issue Nov 29, 2023
nicolas-grekas added a commit to symfony/symfony that referenced this issue Apr 12, 2024
…tions (uncaught)

This PR was merged into the 5.4 branch.

Discussion
----------

[Cache] Always select database for persistent redis connections

| Q             | A
| ------------- | ---
| Branch?       | 5.4
| Bug fix?      | yes
| New feature?  | no
| Deprecations? | no
| Issues        | Fix #51578
| License       | MIT

Even though the `dbindex` is supposed to be `0` by default for any new redis connection, when dealing with a persistent connection pool, due to a [bug in phpreds](phpredis/phpredis#1920) you can actually end up on a different database.

This PR will call `select` even for a `dbindex` of `0` under these conditions to assure you are actually on the right database.

Commits
-------

ce4f815 bug #51578 [Cache] always select database for persistent redis connections
@wizardist
Copy link

Did you configure redis.pconnect.pool_pattern? It needs to contain i to actually use the persistent_id as part of a hashmap key, if I understand correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests