A bunch of irrelevant pipelines on datanode (copied from https://issues.apache.org/jira/browse/HDDS-11430)

[vtutrinov]

There was a following case:

• 9 datanodes
• a bucket with RATIS/THREE replication configuration
• around 6Tb total disk space
• a bunch of write operations applied and SCM started to respond with error that we haven't enough datanodes to allocate new pipeline
• datanodes's logs contains entries about "volume space is not enough"
• the cluster was living in that state some time (2-3 hours) and clients were continuing their attempts to write keys
• at a certain moment the cluster was rebooted and 6 datanodes not entered to HEALTHY state
• heap dump was fetched from one of that datanodes - a lot of SegmentedRaftLog objects was detected

•  these are the pipeline objects that were trying to be created earlier
• the datanodes were not started correctly until the dfs.container.ratis.log.appender.queue.byte-limit configuration property was decreased to 100KB (the default value is 32MB)
• without reducing the conf property value, data nodes would reach a too large heap space state
• eventually, the datanodes removed unnecessary pipelines and entered a HEALTHY state

QUESTION

Why did SCM try to create pipelines on full datanodes and how to avoid this?

[ChenSammi]

@vtutrinov , what's your Ozone version?
When SCM creates a pipeline, it will filter out DNs which doesn't have at least "ozone.scm.container.size" space size, which by default is 5GB. But they are concurrent pipeline allocations, and other disk activities from other processes on the same DN server, so it's risk to depend on this 5GB to prevent disk from full.
The recommendation is set one of two properties in your ozone-site.xml for DN,

hdds.datanode.dir.du.reserved                  
hdds.datanode.dir.du.reserved.percent

You can refer to the ozone-default.xml for more information about these two properties.

[vtutrinov]

@ChenSammi We use ozone-1.4 and already familiar with the mentioned configuration properties, and, sure thing, they help us eventually to avoid the problem in advance.
But it looks like a bug for me anyway, it seems the datanodes should have some lock to avoid creation of irrelevant pipelines if the size of the volumes are running out (e.g. as like a rollback step in RDBMS if the pipeline was created with error)