Skip to content
GitLab
  • Menu
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • H haf
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 46
    • Issues 46
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 20
    • Merge requests 20
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • hivehive
  • haf
  • Merge requests
  • !152

Added sql logs

  • Review changes

  • Download
  • Email patches
  • Plain diff
Open Michal Zander requested to merge mzander/sql-logs into develop Aug 03, 2022
  • Overview 9
  • Commits 8
  • Pipelines 102
  • Changes 24

Firstly we need to create config file promtail-config.yml by creating in specific directory (in my case it is home/dev/docker_volumes/promtail , if you want to change it to another directory you need to change it in docker-compose.yaml aswell to your_directory:/etc/promtail )

---
clients:
  -
    url: "http://loki:3100/loki/api/v1/push"
positions:
  filename: /tmp/positions.yaml
scrape_configs:
  -
    job_name: hive
    pipeline_stages:
      -
        logfmt:
          mapping:
            timestamp:
            user:
            database:
            level:
            context:
            haflog:
    static_configs:
      -
        labels:
          __path__: /var/log/postgresql/*log
          hive: logs
        targets:
          - localhost
  -
    job_name: arch
    static_configs:
      -
        labels:
          __path__: /var/log/postgresql/*log.1
          hive: archive
        targets:
          - localhost
server:
  grpc_listen_port: 0
  http_listen_port: 9080

Next step will be creating our dockers that collect, send and visualize sent logs by creating a docker-compose.yaml file in selected directory

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:2.6.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.6.0
    volumes:
      - /var/log/:/var/log/
      - /home/dev/docker_volumes/promtail:/etc/promtail
      - /var/log/postgresql/:/etc/log/postgresql
    command: -config.file=/etc/promtail/promtail-config.yml
    networks:
      - loki

In postgresql.conf located in /etc/postgresql/12/main (opened with sudo) change log prefixes to

log_line_prefix = 'timestamp="%m" user=%u database=%d level=' 

and log level to highest state

log_min_messages = debug5

After changes in postgresql.conf you need to restart postgres by sudo systemctl restart postgres

Now we can start our dockers by using this command:

docker-compose -f docker-compose.yaml up

in localhost:3000 there is located grafana where we need to setup loki as a source of logs and change url to http://loki:3100

Done.

Edited Aug 31, 2022 by Michal Zander
Assignee
Assign to
Reviewer
Request review from
Time tracking
Source branch: mzander/sql-logs