Flood Stage Watermark 🌊 in ElasticSearch
![cluster_block_exception [FORBIDDEN/12/index read-only / allow delete (api)];](flood.jpg)
A watermark is simply a threshold set on a ElasticSearch cluster. In this case, Flood Stage Watermark is indicating a major disk space issue, 3 watermarks are related to that:
A watermark is simply a threshold set on a ElasticSearch cluster. In this case, Flood Stage Watermark is indicating a major disk space issue, 3 watermarks are related to that:
- low: 85% of disk usage (default)
- high: 90% of disk usage (default)
- flood_stage: 95% of disk usage (default) <- critical
All watermarks can be verified running a GET requests to the cluster:
curl -X GET -H "Content-Type: application/json" http://[ENDPOINT]:9200/_cluster/settings?include_defaults=true
When the last threshold or watermark is reached, ElasticSearch will force read-only mode. This is the security mecanism to prevent nodes from running out of space, so no write operations are allowed at this stage.
In order to fix this issue, ensure to:
- Remove unused indices
- Remove documents
- Increase disk space
Basically anything you can do to free up space. After that, you will need to exit the read only mode manually:
curl -X PUT -H "Content-Type: application/json" http://[ENDPOINT]:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'