Repository navigation
wal files are accumulating #736
Description
Description
I saw the situation a few years ago. A few years ago, I already regulated the accumulation of wal files, and therefore I always tried to keep track of it.
Here's what I've seen: the cluster is working fine, blocks are growing at all nodes, nodes are not restarting, and the leader is not changing. But one node has accumulated wal files. And according to all metrics and logs, the node is fine.
I have now returned to this problem in an attempt to find a case in which this was happening. For example, a node has no connection with the "leader" node, but it makes up for its condition through synchronization. I checked all this on fabric, so it's possible that the blocks are filled up with gossip when transferring "private" data.
I've seen the situation more than once, so I decided to analyze it. The analysis showed that wal files are not truncated in all situations in which they could be. Although maybe that wasn't the reason for the accumulation.
correction option #735
Steps to reproduce
No response
Activity
pfi79 commentedon Oct 25, 2026
@HagarMeir
pfi79 commentedon Nov 3, 2026
I'll close it because there's nothing else to do here yet. It would be nice to find that the wal files continue to accumulate.
The SmartBFT library already collects metrics, they are connected to fabric and they are given away. There is a Grafana dashboard with metrics (it needs to be slightly modified locally). If we connect it at the stands and set up an alert about the excess number of WAL files, we can detect the problem faster.