Out of memory issue for master and client nodes

[mootezbessifi]

Hi,
i am running a ES cluster on top of k8s (2 es-client, 3 es-master and 3 es-data).
the cluster is used for efk stack.
i configured before the heap size for each one as following:

• each es-client: 2GB
• each es-master: 1GB
• each es-data: 1GB

the cluster was working fine until today when i tried to change the storage with Glusterfs for the es-data and restart all ES deployments.

es-masters and es-clients refuse to run and generate exception OutOfMemory exception.
i was going for es-client beyond 5GB and i still got the same issue

i need help please

[rewt]

Try increasing es-data nodes to at least 4 GB.

[mootezbessifi]

@rewt thank you very much for your response
when i tried to fo this, all es-data pods generated Out of memory issue.
i want to know i the case of huge traffic from fluentd, pushing logs to ES, what is the best configuration tu use (number of es-masters/client/data and what is the memory heap size for each one)
thank you

[mootezbessifi]

the strange think here is that when it was es-data heap = 1G like es-master and increasing it, it was working fine, but when increasing its heap to 4GB as you recommended @rewt , it returns out of memory issue !!!!

[rewt]

@mootezbessifi In my experience, I was giving java heap more RAM than container was able to provide, so while pod worked fine for a while, as time went on, Java heap would attempt to allocate more RAM than available - hence java heap errors.

I modified resource requests in stateful yaml to include memory req and limits and error, configured java_opts within those limits, and errors resolved.

https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/

[mootezbessifi]

@rewt
in my case i am running on top of machines (k8s cluster nodes) with 16GB memory. and as i mentioned before i have 2 es-clients, 3 es-masters and 3 es-data. what is the most suitable memory heap configuration for each es-{type}.

thank you

[rewt]

@mootezbessifi That is essentially your call and depends what kind of data ES needs to handle.

To resolve java heap out of memory errors, start by checking how much RAM the container is currently consuming:

kubectl top pod <data node name>

Then be sure ES_JAVA_OPTS does not exceed how much RAM the pod is currently consuming.

From there you can set limits for the container based on link above.