Star
Try Uptrace

Mastering Kubernetes Logging - Detailed Guide to kubectl logs

Effective logging is crucial for maintaining and troubleshooting applications running in Kubernetes clusters. As applications become more complex, ensuring they perform optimally has never been more critical. In this comprehensive guide, we'll explore Kubernetes logging using kubectl, covering everything from basic commands to advanced techniques and best practices.

Kubernetes Logging Overview
Kubernetes Logging Overview

What is Kubernetes Logging?

Kubernetes logging involves collecting, managing, and analyzing log data from containerized applications and the Kubernetes system itself. Logs provide critical insights into application behavior, help diagnose issues, and offer valuable data for monitoring and analytics.

Core Components of Kubernetes Logging

  • Container Logs: Output from applications running in containers.
  • Node Logs: System logs from the Kubernetes nodes.
  • Kubernetes System Component Logs: Logs from Kubernetes control plane components.

Why Choose kubectl for Logging?

kubectl is the command-line interface for interacting with Kubernetes clusters. It offers several benefits for logging:

  • Direct Access: Quickly access logs without additional tools.
  • Real-time Monitoring: Stream logs in real-time for immediate insights.
  • Flexibility: Access logs from various Kubernetes resources (pods, deployments, etc.).
  • Integration: Easily integrate with other command-line tools for advanced log processing.

Basic kubectl logs Commands

Before we dive into specific commands, it's worth noting that kubectl offers a wide range of commands for various Kubernetes operations. For a complete list of kubectl commands and their usage, you can refer to the official Kubernetes documentationopen in new window.

Now, let's start with the fundamental kubectl logs commands:

Viewing Logs from a Pod

kubectl logs <pod-name>

Viewing Logs from a Specific Container in a Pod

kubectl logs <pod-name> -c <container-name>

Following Logs in Real-time

kubectl logs -f <pod-name>

Viewing Logs with Timestamps

kubectl logs --timestamps=true <pod-name>

Advanced kubectl logs Techniques

Tailing Logs

Limit the number of lines returned:

kubectl logs --tail=100 <pod-name>

Logging Since a Specific Time

View logs since a specific time:

kubectl logs --since=1h <pod-name>

Logging from All Containers in a Pod

kubectl logs <pod-name> --all-containers=true

Logging from Previous Container Instances

kubectl logs <pod-name> --previous

Filtering and Searching Logs

Using grep with kubectl logs

kubectl logs <pod-name> | grep "ERROR"

Using jsonpath for Structured Logs

kubectl logs <pod-name> | jq -r '.level + " " + .message'

Logging Multiple Containers and Pods

Logging Multiple Pods with Label Selectors

kubectl logs -l app=myapp

Using stern for Multi-pod Logging

stern <pod-name-pattern>

Troubleshooting Common Logging Issues

Pods in CrashLoopBackOff

When a pod is in CrashLoopBackOff, use --previous to see logs from the last run:

kubectl logs <pod-name> --previous

Container Creating

If a container is stuck in "Creating" state, check events:

kubectl describe pod <pod-name>

Missing Logs

Ensure your application is logging to stdout/stderr. For system components, check node-level logs:

kubectl logs -n kube-system <pod-name>

Best Practices for Kubernetes Logging

Effective logging in Kubernetes requires a strategic approach. Here are some best practices to optimize your logging strategy:

  1. Use Structured Logging (JSON)

    • Implement JSON-formatted logs for easier parsing and analysis.
    • Include relevant metadata such as timestamp, log level, and service name in each log entry.
    • Example:
      {
  "timestamp": "2024-08-27T10:15:30Z",
  "level": "INFO",
  "service": "user-auth",
  "message": "User login successful",
  "userId": "12345"
}

  2. Implement Log Rotation

    • Use Kubernetes' built-in log rotation for container logs.
    • Configure max-size and max-file in your container runtime settings:
      apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
    - name: example-container
      image: example-image
      resources:
        limits:
          cpu: 100m
          memory: 128Mi
      terminationMessagePolicy: File
      terminationMessagePath: /dev/termination-log

  3. Centralize Log Collection

    • Implement a centralized logging solution (e.g., ELK stack, Loki, or Fluentd).
    • Use DaemonSets to ensure log collectors run on every node.
    • Example Fluentd configuration:
      apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
spec:
  selector:
    matchLabels:
      name: fluentd
  template:
    metadata:
      labels:
        name: fluentd
    spec:
      containers:
        - name: fluentd
          image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch

  4. Utilize Kubernetes Namespaces

    • Organize your logs by using Kubernetes namespaces.
    • Create separate namespaces for different environments or teams.
    • Use namespace labels in your logging configuration:
      apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    environment: prod

  5. Implement Proper Log Levels

    • Use appropriate log levels (DEBUG, INFO, WARN, ERROR) consistently across your applications.
    • Configure log levels dynamically using ConfigMaps:
      apiVersion: v1
kind: ConfigMap
metadata:
  name: logger-config
data:
  log_level: 'INFO'

  6. Set Resource Limits for Logging

    • Prevent logging from consuming excessive resources by setting limits.
    • Example:
      apiVersion: v1
kind: Pod
metadata:
  name: logging-pod
spec:
  containers:
    - name: logger
      image: logger-image
      resources:
        limits:
          cpu: '200m'
          memory: '128Mi'

  7. Implement Log Sampling

    • For high-volume applications, consider sampling logs to reduce storage and processing overhead.
    • Use tools like Fluentd's sampling filter:
      <filter **>
  @type sampling
  interval 10
  sample_unit lines
</filter>

  8. Use Contextual Logging

    • Include relevant context in your logs, such as request IDs, user IDs, or transaction IDs.
    • Implement correlation IDs for tracing requests across microservices.

  9. Monitor and Alert on Logging System

    • Set up monitoring for your logging infrastructure.
    • Create alerts for log collection failures or abnormal log volumes.

  10. Implement Log Retention Policies

    • Define and enforce log retention periods based on compliance and operational needs.
    • Use tools like Elasticsearch Curator for managing log indices:
      actions:
  1:
    action: delete_indices
    description: 'Delete indices older than 30 days'
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
      - filtertype: age
        source: creation_date
        direction: older
        unit: days
        unit_count: 30

  11. Secure Your Logs

    • Encrypt logs in transit and at rest.
    • Implement access controls to restrict log access based on roles.
    • Use Kubernetes RBAC for controlling access to log resources:
      apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: log-reader
rules:
  - apiGroups: ['']
    resources: ['pods', 'pods/log']
    verbs: ['get', 'list']

By implementing these best practices, you can create a robust and efficient logging system in your Kubernetes environment. Remember to regularly review and adjust your logging strategy as your application and infrastructure evolve.

Integrating kubectl logs with Other Tools

Using watch for Real-time Monitoring

watch -n 5 'kubectl logs <pod-name> --tail=20'

Saving Logs to a File

kubectl logs <pod-name> > pod_logs.txt

Collecting Kubernetes logs in Uptrace

Uptraceopen in new window is an open-source APM tool that supports collecting and analyzing logs from Kubernetes clusters. Here's how you can set it up:

  1. Deploy Uptrace in your Kubernetes cluster or set it up externally.
  2. Configure a log collection agent like Fluent Bit:
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        5
        Daemon       Off
        Log_Level    info
        Parsers_File parsers.conf

    [INPUT]
        Name             tail
        Path             /var/log/containers/*.log
        Parser           docker
        Tag              kube.*
        Refresh_Interval 5
        Mem_Buf_Limit    5MB
        Skip_Long_Lines  On

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Merge_Log           On
        K8S-Logging.Parser  On

    [OUTPUT]
        Name          http
        Match         *
        Host          uptrace-host
        Port          14318
        URI           /api/v1/logs
        Format        json
        Json_date_key timestamp
  1. Deploy the log collector and configure Uptrace to receive and process the logs.
  2. Use the Uptrace UI to search, filter, and analyze your Kubernetes logs.

Conclusion

Mastering kubectl logs and related logging techniques is essential for effectively managing and troubleshooting Kubernetes applications. By understanding these tools and best practices, you'll be well-equipped to handle logging challenges in your Kubernetes clusters.

Remember, while kubectl logs is powerful for debugging and quick checks, consider implementing a more robust logging solution for production environments to handle log aggregation, searching, and long-term storage.

As you continue to work with Kubernetes, keep exploring new logging techniques and tools. The field of observability is constantly evolving, and staying up-to-date will help you maintain and improve your applications more effectively.

FAQ

  1. How can I view logs from all containers in a pod? Use the --all-containers flag: kubectl logs <pod-name> --all-containers

  2. How do I tail logs in real-time? Use the -f or --follow flag: kubectl logs -f <pod-name>

  3. Can I see logs from a previous instance of a crashed pod? Yes, use the --previous flag: kubectl logs --previous <pod-name>

  4. How can I filter logs for a specific time range? Use the --since flag: kubectl logs <pod-name> --since=1h

  5. Is it possible to get logs from multiple pods at once? Yes, you can use label selectors: kubectl logs -l app=myapp

  6. How can I save logs to a file? Redirect the output to a file: kubectl logs <pod-name> > pod_logs.txt

  7. Can I get logs from a specific container in a multi-container pod? Yes, use the -c flag: kubectl logs <pod-name> -c <container-name>

  8. How do I view logs with timestamps? Use the --timestamps flag: kubectl logs <pod-name> --timestamps

You may also be interested in:

Last Updated: