Imagine you're managing a bustling airport. You need to know not just which planes are landing and taking off, but also their origins, destinations, and if any are experiencing delays.

In the world of software, observability is similar. It's about having the tools to understand the inner workings of your system by examining its outputs, allowing you to answer questions about its behavior, even questions you didn't anticipate needing to ask.

At the heart of Traefik's traffic management are routers. These components act as intelligent directors, examining incoming requests and deciding which service should handle them based on predefined rules. Just as an air traffic controller monitors the movement of aircraft across different runways, understanding the activity of your Traefik routers is essential for ensuring your application's overall health and responsiveness.

Traefik provides several key observability features that can be configured at the router level, offering a granular view into your application's traffic flow. Let's explore these features in a way that's as easy to grasp as explaining it to a curious five-year-old.

Seeing What Your Routers Are Doing: Observability Features

Access Logs: Keeping a Record

Think of access logs as a detailed diary for each of your Traefik routers. Every time a request comes in and is processed by a specific router, an entry is made in this diary. This entry contains important information like the exact time the request arrived, the internet address of the person or system that made the request, what they were asking for (the specific web page or API endpoint), and what happened with their request (was it successful, or did an error occur?).

To put it simply, it's like writing down every car that passes through our airport's main gate, noting the time, where it came from, and where it was headed.

By default, access logs are likely enabled based on the global settings of the EntryPoints that your routers are connected to. However, Traefik gives you the power to specifically control whether a particular router generates these logs. You might want to disable access logs for a very high-traffic route that you don't need detailed information on, or you might want to explicitly enable them for a specific router you're actively troubleshooting.

Here's how you can disable access logs for a router named my-router that handles requests to the path /foo using YAML configuration:

http:
  routers:
    my-router:
      rule: "Path(`/foo`)"
      service: service-foo
      observability:
        accessLogs: false

This ability to configure logging at the router level offers precise control. If you're experiencing issues with a particular feature accessed through a specific route, enabling detailed access logs just for that router can provide focused information without overwhelming you with logs from the entire system. Conversely, for very high-traffic, non-critical routes, you might choose to disable logging to conserve resources.

You can send these logs elsewhere. Elastic Search or AWS CloudWatch

Metrics: Getting the Numbers

Metrics are like taking regular health checkups for your Traefik routers. Instead of just recording every single event like access logs, metrics provide numerical summaries of your router's activity over time. This could include things like the total number of requests the router has handled, the average time it takes to respond to requests, and the number of errors that have occurred.

Think of it as counting how many planes land and take off every hour, and how long passengers have to wait at security checkpoints.

Just like access logs, metrics can be enabled or disabled for individual routers using the metrics option under the observability section in the router's configuration. Here's how you would disable metrics for our my-router in YAML:

http:
  routers:
    my-router:
      rule: "Path(`/foo`)"
      service: service-foo
      observability:
        metrics: false

However, there's a crucial point to understand about metrics in Traefik: even if you enable metrics at the router level, they won't actually be generated unless the global metrics collection is also enabled. This is controlled by options like addEntryPointsLabels, addRoutersLabels, and addServicesLabels in Traefik's main configuration. These global settings act like a master switch for metrics. If the master switch is off, individual routers won't be able to send their metric data. This global dependency is important to remember because a developer might enable metrics for a specific router and then be puzzled when no metrics data appears. The reason is likely that the overall metrics collection hasn't been activated in Traefik's broader configuration.

Tools like Prometheus, AWS CloudWatch or SigNoz can help with metric agreggation

Tracing: Following the Path

Tracing takes observability a step further by allowing you to follow the complete journey of a single request as it travels through Traefik and potentially to other services that are involved in handling that request. Imagine putting a special tracking device on a specific plane as it arrives at our airport, goes through baggage handling, gets refueled, and then departs. Tracing in Traefik lets you visualize this entire path for a web request, helping you identify any delays or issues at each stage.

You can enable or disable tracing for a specific router using the tracing option within the observability section of its configuration. Here's how to disable it for my-router in YAML:

http:
  routers:
    my-router:
      rule: "Path(`/foo`)"
      service: service-foo
      observability:
        tracing: false

Traefik leverages OpenTelemetry, an open standard for observability, for its tracing capabilities. This is significant because it means that the tracing data generated by Traefik can be easily integrated with a wide variety of popular tracing backends, such as Jaeger or Zipkin. When a request involves multiple microservices, it can be challenging to pinpoint the exact location of a problem. Tracing provides a detailed timeline of the request's journey across all involved components, making it much easier to diagnose and resolve issues.

Tools like Jaeger can connect here

Q&A

Where Do Traefik Logs Live?

By default, Traefik writes its access logs to the standard output (stdout). If you're running Traefik in a containerized environment like Docker, this means you'll typically find the access logs mixed in with the other logs generated by the container. The same applies to Traefik's own internal logs, which include information about startup, configuration, and events.

However, Traefik offers the flexibility to configure it to write these logs to specific files. You can achieve this by using the filePath option. For access logs, you'll find this option under the accessLog section in your static configuration. For Traefik's internal logs, the filePath option is located under the log section.

Here's an example of how to configure both access logs and Traefik logs to write to files in YAML:

accessLog:
  filePath: "/var/log/traefik/access.log"

log:
  filePath: "/var/log/traefik/traefik.log"

Additionally, you can control the format of the logs using the format option under both accessLog and log. The default format is common (Common Log Format), but you can also choose json for a more structured output.

While the default of writing logs to stdout is convenient for basic setups and container orchestration platforms, configuring filePath becomes crucial for production environments. Writing logs to files allows for persistent storage and enables integration with dedicated log management systems for analysis and long-term retention. The choice between standard formats like CLF and structured formats like JSON depends on your specific needs for processing and analyzing the log data.

Do I Need a Special Place to Keep My Logs?

Yes, especially if you are running Traefik within a containerized environment like Docker. Containers are often designed to be ephemeral, meaning they can be easily stopped, started, and replaced. When a container stops or is removed, any data stored within its filesystem, including logs written directly into the container, is typically lost.

To ensure that your Traefik logs persist even if the container restarts or is redeployed, you should utilize persistent volumes. Persistent volumes provide a way to store data outside the container's lifecycle, either on the host machine or in a named volume managed by the container runtime.

Here's an example of how you might configure a persistent volume for Traefik logs using docker-compose.yml:

version: '3.7'
services:
  traefik:
    image: traefik:v2.10
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      -./traefik_logs:/var/log/traefik # Mount a host directory to the container's log directory
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--accesslog.filepath=/var/log/traefik/access.log"
      - "--log.filepath=/var/log/traefik/traefik.log"

volumes:
  traefik_logs: # Define the named volume (optional, can also use bind mounts as above)

In this example, we are mounting a directory named traefik_logs on the host machine (represented by ./traefik_logs) to the /var/log/traefik directory inside the Traefik container. If you've configured Traefik to write logs to files within /var/log/traefik, these logs will now be stored on the host and will persist even if the Traefik container is restarted or replaced.

Furthermore, when using persistent volumes for logs, it's essential to consider log rotation. Over time, log files can grow very large and consume significant disk space. Log rotation is a process of automatically archiving and deleting old log files to manage disk usage. You can typically configure log rotation using tools available on your operating system or within your container environment.

You can also leverage Docker native AWS driver to redirect these logs to AWS CloudWatch. Many other drivers are available as well.

How Would This Look on Docker Swarm?

When deploying Traefik in a Docker Swarm environment, configuration is primarily done through labels applied to the Traefik service definition within your docker-compose.yml file or a dedicated Swarm deployment file. Observability settings for individual routers are also configured using labels, specifically within the deploy.labels section of the service definition.

To enable or disable access logs, metrics, or tracing for a specific router in Docker Swarm, you would use labels with the following structure :

• traefik.http.routers.<router_name>.observability.accessLogs=true/false
• traefik.http.routers.<router_name>.observability.metrics=true/false
• traefik.http.routers.<router_name>.observability.tracing=true/false

For example, if you have a router named my-app-router, you could disable access logs for it by adding the following label under the deploy.labels section of your Traefik service:

deploy:
  labels:
    - "traefik.http.routers.my-app-router.observability.accessLogs=false"

Regarding log storage in Docker Swarm, the same principles of persistent volumes apply as in standalone Docker setups. You would define a volume in your docker-compose.yml file and mount it to the Traefik service. Here's an example demonstrating this:

version: '3.7'
services:
  traefik:
    image: traefik:v2.10
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - traefik_logs:/var/log/traefik
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.traefik-dashboard.rule=Host(`traefik.localhost`)"
        - "traefik.http.routers.traefik-dashboard.service=api@internal"
        - "traefik.http.routers.my-app-router.rule=PathPrefix(`/my-app`)"
        - "traefik.http.routers.my-app-router.service=my-app-service@docker"
        - "traefik.http.routers.my-app-router.observability.accessLogs=true"
volumes:
  traefik_logs:

In this Swarm example, we've defined a named volume traefik_logs and mounted it to the /var/log/traefik directory within the Traefik service. This ensures that any logs written to this directory by Traefik will be persisted across service updates or restarts in the Swarm cluster. Remember that you also need to configure Traefik to actually write the logs to this path using the accesslog.filePath and log.filePath options in your static configuration.

Putting It All Together: A Practical Example

Let's look at a simplified docker-compose.yml example for a Docker Swarm setup with a basic web application and Traefik, demonstrating router-level observability:

version: '3.7'
services:
  reverse-proxy:
    image: traefik:v2.10
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - traefik_logs:/var/log/traefik
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.my-app.rule=Host(`myapp.localhost`)"
        - "traefik.http.routers.my-app.service=my-app-service@docker"
        - "traefik.http.routers.my-app.entrypoints=web"
        - "traefik.http.routers.my-app.observability.accessLogs=true"
        - "traefik.http.routers.my-app.observability.metrics=false"
        - "traefik.http.routers.my-app.observability.tracing=false"
        - "traefik.http.services.my-app-service.loadbalancer.server.port=8080"
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.swarmmode=true"
      - "--accesslog.filepath=/var/log/traefik/access.log"

  my-app:
    image: nginx:latest
    ports:
      - "8080"
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.services.my-app-service.loadbalancer.server.port=80"

volumes:
  traefik_logs:

In this example:

• We have a reverse-proxy service running Traefik. It exposes ports 80 and 443 and mounts the Docker socket to discover other services. It also mounts a named volume traefik_logs to /var/log/traefik and is configured to write access logs to /var/log/traefik/access.log.
• The my-app service is a simple Nginx web server.
• In the reverse-proxy service's deploy.labels, we define a router named my-app that listens for requests to myapp.localhost on the web entrypoint and forwards them to the my-app-service. We've explicitly enabled access logs for this router but disabled metrics and tracing.
• The my-app service also has labels to indicate to Traefik that it should be exposed and that its service is running on port 80.

This practical example demonstrates how you can enable access logs for a specific router in a Docker Swarm environment while leaving other observability features disabled for that particular route.

---

If you liked this content I’d appreciate an upvote or a comment. That helps me improve the quality of my posts as well as getting to know more about you, my dear reader.

Muchas gracias!

Follow me for more content like this.

X | PeakD | Rumble | YouTube | Linked In | GitHub | PayPal.me | Medium

Down below you can find other ways to tip my work.

BankTransfer: "710969000019398639", // CLABE
BAT: "0x33CD7770d3235F97e5A8a96D5F21766DbB08c875",
ETH: "0x33CD7770d3235F97e5A8a96D5F21766DbB08c875",
BTC: "33xxUWU5kjcPk1Kr9ucn9tQXd2DbQ1b9tE",
ADA: "addr1q9l3y73e82hhwfr49eu0fkjw34w9s406wnln7rk9m4ky5fag8akgnwf3y4r2uzqf00rw0pvsucql0pqkzag5n450facq8vwr5e",
DOT: "1rRDzfMLPi88RixTeVc2beA5h2Q3z1K1Uk3kqqyej7nWPNf",
DOGE: "DRph8GEwGccvBWCe4wEQsWsTvQvsEH4QKH",
DAI: "0x33CD7770d3235F97e5A8a96D5F21766DbB08c875"