So we have a jupyterhub that spawns individual notebooks with kubespawner (which creates a pod to run individually), what we need is to get the logs for those individual notebooks with filebeat from the EFK (elastic.co) stack;
The issue that we are having is that if I run filebeat outside the indv pod, it cant access the logs since it cant get into the files/paths of the pod;
Is there a way to get filebeat as a sidecar for this configuration? i now that if it wasnt a pod or if it was just a container running on the same instance we could use the autodiscovery mode of filebeat, but dont know how we can get those logs…
Now we are not closed to using only filbeat, if you have experience with taking the logs out and is with something else, im listening.
The logs from pods started by KubeSpawner will be handled by your K8s cluster’s standard logging. If it’s possible to send the K8s logs to ElasticSearch directly through K8s that might be easier? Kubernetes Container Logs | Elastic docs suggests it’s possible.
If you want to launch a sidecar container in the singleuser pod you can define extra containers:
I thought the same of trying to get them directly from K8s cluster, but for some reason, thats not possible (architecture wise for our case, hate whoever came with this architecture);
Thats why we are trying to get the sidecar; I tried that extra container and seems to work(?) but Seems like we are missing either some flags/args or something cause its not communicating with our elk instance but your input was really helpful
its outside, we now got it working with the extra container config, we needed to mount the filebeat image on the same path/volume where the logs were being written; now we can see some logs on elastic but im not sure if those are the right ones or not, the reason being, the individual servers cant start and we are getting logs that look like they belong to the individual notebook (due to their structure) but not sure if hub is sending those
but your input was really helpful