Concurrent lock writes from different TYPO3 nodes resulting in wait timeouts

[vixe76]

Hello,

we had a massive problem with concurrent lock writes. We have an environment with a single Redis server which handles all the caching and locking. This Redis server is used by 2 TYPO3 nodes. Under some circumstances both instances want to write the same lock which results in long waits (blocked each other) and the php processes on both increased drastically. We had to restart the Redis server for holding the environment alive. We had experimented with 2 shared Redis servers for separating caching and locking but with the same result.

In the meantime we have switched our architecture to separated Redis server per TYPO3 instance to get rid of this locking wait problem.

Btw. many thanks for this extensions!

Best Regards
Falk

[bmack]

hey falk,

did you have this issue with this extension? Can you confirm that they used different databases within the same redis server? are you still using this extension?

thanks for the info!

[vixe76]

Hi Benni,

yes, I've switched from the tourstream extension (which had the same problem) to this/yours extension.

The TYPO3 nodes used the same locking database on the same Redis server.

I still using your extension but now with the separated Redis server instances per node to avoid the described problem.

Do you need more details?

[ryno83]

I think adding a prefix to each key would solve the problem. Something like this :
LocalConfiguration.php or AdditionnalConfiguration.php:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['locking']['redis'] = [
    'redisPrefix' => sha1($GLOBALS['TYPO3_CONF_VARS']['SYS']['encryptionKey']) . '_',
    'database' => 8,
    'hostname' => $redisHost,
    'authentication' => $redisPassword,
    'port' => $redisPort,
];

RedisLockingStrategy.php

private function connectBackend($configuration): \Redis
{
    $backend = new \Redis();
    $backend->connect($configuration['hostname'], (int)$configuration['port']);
    if (!empty($configuration['authentication'])) {
        $backend->auth($configuration['authentication']);
    }
    if (!empty($configuration['redisPrefix'])) {
        $backend->setOption(\Redis::OPT_PREFIX, $configuration['redisPrefix']);
    }
    $backend->select((int)$configuration['database']);
    return $backend;
}

[bmack]

Hey @ryno83 ,
sorry for the delay. Could you provide a PR for this? Seems legit.

[supercaccia]

Hi everyone @vixe76 @ryno83 @bmack

I have 4 Frontend Server Apache, loada balancer, for one TYPO3 istance and shared NFS filesystem.
I'm using php 7.4, TYPO3 10.4.36.

Our site has a lot of web traffic and under some circumstances whe have locking strategy problems which results in long waits and the php processes on both increased drastically, we always have to restart the servers to fix the problem.

• Locking Strategy in typo3temp/var/lock into NFS creates problems, described in the official documentation
• Locking Strategy local in each of the 4 webservers creates problems, described in the official documentation
• Locking Strategy with extension distributed_locks creates problems

TYPO3 log error with distributed_locks:

[CRITICAL] request="cb9658c174eb8" component="TYPO3.CMS.Core.Error.ProductionExceptionHandler": 
Core: Exception handler (WEB): Uncaught TYPO3 Exception: #1561445710: Could not acquire exclusive lock (blocking+exclusive). 
| TYPO3\CMS\Core\Locking\Exception\LockAcquireException thrown in file /vhosts/www/typo3conf/ext/distributed_locks/src/RedisLockingStrategy.php in line 175.
Requested URL: https://test.... - {"TYPO3_MODE":"FE","exception":null}

How should I configure TYPO3 + Redis + distributed_locks correctly in my case?

Can anyone give me some advice please?

Thanks in advance!

[sypets]

@supercaccia I wish I could help but all I can say right now is we do use this extension and Redis with 2 servers but not concurrently, but as a failover. I do not have these errors in the logs . (we use Redis for cache and locking and it works smoothly, but it is a different setup of course)

---

Disclaimer: I am just a user of this extension, not maintainer. (I did write the initial version of the Locking page in the documentation you referenced though and added the information for the NFS stuff. We were affected by that and switched to this extension for locking).

I also wrote the section on Redis server configuration in the docs after tracking down a rather nasty problem. I think this is probably unrelated but might cause problems if you are using Redis cache as well. You might want to check if you are affected. (The impact was that cache entries were not being flushed and number of entries kept increasing. Changing the eviction policy (to maxmemory-policy volatile-ttl) on the Redis server as described in the documentation solved the problem.)

---

It might help if you supply some more information:

• can you list your versions (TYPO3 and b13/distributed-locks)? I don't think it is that relevant here, but for completeness sake might be helpful. Also some things were changed between v11 and v12 in core
• do you use the locking API yourself in extensions or is it only the core?
• do you have a full stack dump so it is possible to see the calling functions (probably TypoScriptFrontendController?)
• what is happening on your redis server when these problems occur? (e.g. can you look at read / writes and if there is anything conspicuous). Just to be sure: Is the memory limit reached on the server and which eviction policy is being used? A big nono is using "no-eviction". Also, if the memory limit is constantly reached and keys evicted this might also cause resource problems (though redis should be able to handle high traffic).
• how much traffic do you have (accessing TYPO3 pages)?

What you can also do is look in the code (in distributed-locks) and see if it is possible to narrow down the problem or create a solution:

/distributed-locks/src/RedisLockingStrategy.php:

while (!$this->isAcquired = $this->lock()) {

    // this blocks till the lock gets released or timeout is reached
    if (!$this->wait()) {
        throw new LockAcquireException('Could not acquire exclusive lock (blocking+exclusive).',
                            1561445710);
      }
}

distributed-locks/src/RedisLockingStrategy.php

Line 175 in 4cc1e06

throw new LockAcquireException('Could not acquire exclusive lock (blocking+exclusive).',

It looks like when the exception is thrown, the timeout is reached. Is it theoretically possible for a deadlock to occur where a lock is not released?

(For the core behaviour of page caching and locking you can look into our old friend TypoScriptFrontendController where the page rendering, caching and locking is done - at least for the page cache. Look at the function acquireLocK. Code is a bit different between v11 and v12. There is a description in v11 TypoScriptFrontendController::aquireLock which explains a 2 phase mechanism with acquiring 2 locks, the accessLock (exclusive and blocking) and the pageLock (exclusive and non-blocking).

---

Also some tips for better readability of the comment (sorry about the nitpicks):

• Why did you list the first 2 points (with NFS and with local) as "ok"? There is documentation for that but that is rather a "don't do this" so there is nothing ok. Can you just change the link to "documentation" to make it more clear or clarify?
• maybe format the exception as code (with three leading and trailing backticks), also to prevent the URL rendered as link

[hannesbochmann]

Thanks for the hints. We had the same problem and got it working after the configuration for eviction policy etc. were adapted as stated in the documentation. I'm not completely sure but using tcp instead of tls might have done the trick, too.

[andreashager]

I had exactly the same problem and I found out, that when the page is generated a global lock on 'pages' is done for each request. With a lot of traffic on different pages that are not cached the lock causes the FPM processes to spawn 😊

See https://forge.typo3.org/issues/105555