Flood Stage Watermark 🌊 in ElasticSearch

May 1, 2024 #elasticsearch
cluster_block_exception [FORBIDDEN/12/index read-only / allow delete (api)];
A watermark is simply a threshold set on a ElasticSearch cluster. In this case, **Flood Stage Watermark is indicating a major disk space issue**, 3 watermarks are related to that:
  1. low: 85% of disk usage (default)
  2. high: 90% of disk usage (default)
  3. flood_stage: 95% of disk usage (default) <- critical

All watermarks can be verified running a GET requests to the cluster:

1

 curl -X GET -H "Content-Type: application/json" http://[ENDPOINT]:9200/_cluster/settings?include_defaults=true

When the last threshold or watermark is reached, ElasticSearch will force read-only mode. This is the security mecanism to prevent nodes from running out of space, so no write operations are allowed at this stage.

In order to fix this issue, ensure to:

  • Remove unused indices
  • Remove documents
  • Increase disk space

Basically anything you can do to free up space. After that, you will need to exit the read only mode manually:

1

 curl -X PUT -H "Content-Type: application/json" http://[ENDPOINT]:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

Reference

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#cluster-routing-flood-stage