Monitoring

There are a number of things that need to be monitored in the elk stack.

Checking the status of a cluster or an index seems simple, but my brain is having trouble with it.

Note

Adding ?v at the end of some of these (the delimited line output ones anyway) adds headers to each column.

Warning

“Space delimited line” is a bit of a lie. There are multiple spaces. It’s a nightmare, built for looks instead of functionality.

Sysadmin type stuff

Cluster allocation

Output: space delimited line

curl 'http://USER:PASSWORD4321@elastic.example.com:9200/_cat/allocation?v'

Get a list of indexes

Output: space delimited line

curl -sk -XGET 'https://USER:PASSWORD1234@elastic.example.com:9200/_cat/indices'

Restrict the list to only the index names:

curl -sk -XGET 'https://USER:PASSWORD0987@elastic.example.com:9200/_cat/indices?h=index'

Check read-only status

Get the json for the read_only_allow_delete field. Empty {} if the field does not exist. The field will not exist unless specifically set.

Output: json

curl -sk -XGET 'https://USER:PASSWORD1234@elastic.example.com:9200/fluttershy-2019.08.*/_settings/index.blocks.read_only_allow_delete'

Use jq to see do the thing. Will return null if the field does not exist.

Output: “true”|”false”

curl -sk -XGET 'https://USER:PASSWORD1234@elastic.example.com:9200/fluttershy-2019.08.*/_settings/index.blocks.read_only_allow_delete' \
| jq '.["fluttershy-2019.08.02"].settings.index.blocks.read_only_allow_delete'

Reset read-only status

for i in $(curl -s 'http://127.0.0.1:9200/_cat/indices?h=index'); do curl -s -XPUT -H 'Content-Type: application/json' "http://127.0.0.1:9200/${i}/_settings" -d '{  "index": {  "blocks": {  "read_only_allow_delete": "false"  }}}' ; echo "";  done

red|yellow|green

Output: space delimited line

curl -sk -XGET 'https://USER:PASSWORD@elastic.example.com:9200/_cat/health?v'

Get disk usage

List a bunch of the stuff

GET /_cat/allocation?v

List the node and disk avail

GET /_cat/nodes?h=h,name,diskAvail&v

Get the fs usage of each node in a cluster as percent used (along with the roles of the node)

GET /_nodes/stats | jq '.nodes[] | "\(.name) \(.roles) \((.fs.total.free_in_bytes / .fs.total.total_in_bytes) * 100)"'

Lots of stats

GET /_stats

elasticsearch logs

From elastic cloud console, clicking the View in Logs and metrics takes you to Observability/Logs/Stream in Kibana.