diff --git a/prometheus_grafana/readme.md b/prometheus_grafana/readme.md index 52563e9..e282148 100644 --- a/prometheus_grafana/readme.md +++ b/prometheus_grafana/readme.md @@ -508,27 +508,45 @@ Including pushing information from windows powershell.

Loki

- Loki is made by grafana team, who call it Prometheus for logs.
- It is a **push** type monitoring, where an agent(**promtail**) - pushes logs on to a Loki instance. + Loki is made by the grafana team, and they call it Prometheus for logs.
+ It is a **push** type monitoring, where an agent - **promtail** + pushes logs on to a Loki instance, or **loki-docker-driver** is installed + on a docker host and log pushing is set either globally in /etc/docker/daemon.json + or per container in compose files. What this example will set out to do is monitor logs of a [minecraft server.](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/master/minecraft) - Have some dashboards about logs and have an alert when a player joins. + Have some dashboards about logs and have an alert when a player joins.
+ Its just proof of concept, as there are prometheus exporters for minecraft, + so digging through logs to get alert is less optimal. - To add to the existing setup: + What needs to be done: - - New container - `loki` added to the compose file. It stores logs and makes + - New container - `loki` added to the compose file. Loki stores logs and makes them available for grafana to visualize. - - New container - `promtail` added to the compose file. Agent that pushes - logs content to loki. This container needs access to logs to be pushed - and so docker host `/var/lib/docker/containers` directory is bind mounted. - New file - `loki-docker-config.yml` bind mounted in the loki container.
The file is [all default](https://github.com/grafana/loki/tree/main/cmd/loki), except for alertmanager url.
- - New file - `promtail-config.yml` mounted in to promtail container
- This files defines source of logs. In here a minecraft container is set, - with some added labels for easier targeting of logs. + - install [loki-docker-driver](https://grafana.com/docs/loki/latest/clients/docker-driver/)
+ `docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions`
+ - adding logging section to compose files of a containers + that should be monitored, like so:
+ ``` + services: + whoami: + image: "containous/whoami" + container_name: "whoami" + hostname: "whoami" + ports: + - 88:80 + logging: + driver: "loki" + options: + loki-url: "http://localhost:3100/loki/api/v1/push" + ``` + + Promtail can be used to push logs to loki, older version of this readme + has details - [here.](https://github.com/DoTheEvo/selfhosted-apps-docker/tree/70830ac8679d81e73265c8522e06312db4905f3b/prometheus_grafana)
docker-compose.yml @@ -650,22 +668,6 @@ Including pushing information from windows powershell. labels: org.label-schema.group: "monitoring" - # LOG AGENT PUSHING LOGS TO LOKI - promtail: - image: grafana/promtail:2.7.3 - container_name: promtail - hostname: promtail - user: root - restart: unless-stopped - volumes: - - /var/log:/var/log:ro - - /var/lib/docker/containers:/var/lib/docker/containers:ro - - ./promtail-config.yml:/etc/promtail-config.yml - command: - - '-config.file=/etc/promtail-config.yml' - labels: - org.label-schema.group: "monitoring" - networks: default: name: $DOCKER_MY_NETWORK @@ -711,40 +713,21 @@ Including pushing information from windows powershell. ```
-
- promtail-config.yml + +Now with driver installed, files in place, compose edited,.. - ```yml - server: - http_listen_port: 9080 - grpc_listen_port: 0 - - positions: - filename: /tmp/positions.yaml - - clients: - - url: http://loki:3100/loki/api/v1/push - - scrape_configs: - - job_name: my-container-logs - static_configs: - - targets: # tells promtail to look for the logs on the current machine/host - - localhost - labels: # labels with which all the following logs should be labelled - job: minecraft-container # label-1 - host: docker-host-archlinux # label-2 - __path__: /var/lib/docker/containers/60a617a7c1376ad42*/60a617a7c1376ad42*.log - ``` -
- -There was also an option to use loki logging driver to just get docker containers. -But any container re-recreated with that setting set took always long time to down.
- -Anyway, now with these files in place.. - -* Loki needs to be added to in grafana as a datasource.
- If everything works as it should, there should be no notice, down left side:
+* In grafana, loki needs to be added as a datasource.
+ If everything works as it should, there should be no red notice, down left side + only gree:
`Data source connected and labels found.` +* In `Explore` section, if the input is set to `Builder`, picking from dropdown + filter menu container_name = minecraft, and hitting run query.. + this should result in seeing minecraft logs. +* In Alert grafana section, + + + +