Monitoring Linux and managing logs

Ankita Lunawat's photo
Ankita Lunawat
·

4 min read

Monitoring Linux and managing logs are essential for maintaining system performance, security, and stability, and this guide provides comprehensive instructions on how to do so.

1. Monitoring Linux Systems

A. Key Metrics to Monitor

  • CPU Usage: Monitors processor load to identify processes using too many resources.

  • Memory Usage: Monitors the usage of RAM and swap memory.

  • Disk Usage and I/O: Ensures there is enough free space and monitors the read/write performance.

  • Network Usage: Tracks bandwidth, packet loss, and network errors to monitor network usage.

  • Processes: Monitors active processes, their resource usage, and identifies zombie processes.

  • System Uptime: Checks the duration of the system's uptime.

B. Tools for Monitoring

1. Built-in Tools

  • top and htop: Use tools like top and htop to monitor real-time system processes and resource usage.

  • vmstat: Displays statistics for CPU, memory, and I/O..

  • iostat: Provides disk and I/O statistics using iostat.

  • netstat or ss: Monitors network connections and sockets using netstat or ss.

  • free: Shows memory usage and swap details using the free command.

  • uptime: The uptime command shows how long the system has been running and the average load.

2. Advanced Monitoring Tools

  • Nagios: Nagios is an open-source tool used for monitoring infrastructure and networks.

  • Zabbix: Zabbix monitors servers, networks, and applications with powerful visualization.

  • Prometheus and Grafana: Real-time monitoring and alerting are achieved with customizable dashboards.

  • Glances: Glances is a cross-platform tool for system monitoring with a web-based interface.

  • Netdata: Lightweight monitoring with a user-friendly web interface.

3. Cloud-Based Monitoring Tools

  • AWS CloudWatch, Datadog, New Relic, and Dynatrace are used for monitoring in hybrid or cloud-native environments.

2. Log Management in Linux

A. Importance of Log Management

Logs are crucial for debugging, analyzing performance, and ensuring security by providing valuable insights.

  • System Events: Boot logs, kernel messages, and hardware activity are examples of system events.

  • Application Logs: Errors, access logs, and application-specific information are crucial for debugging, analyzing performance, and ensuring security..

  • Security Logs: Unauthorized access attempts, user activity, and auditing are important components of security logs..

B. Common Linux Log Files

  1. System Logs:

    • /var/log/syslog (General system logs in Debian-based systems).

    • /var/log/messages (General system logs in Red Hat-based systems).

  2. Kernel Logs:

    • /var/log/kern.log: Logs related to kernel events are stored .

  3. Authentication Logs:

    • /var/log/auth.log: User login activity is recorded in /var/log/auth.log on Debian-based systems..

    • /var/log/secure: Authentication activity is recorded in /var/log/secure for Red Hat-based systems..

  4. Application Logs:

    • /var/log/httpd/ or /var/log/nginx/: Web server logs.

    • /var/log/mysql/: Database logs.

  5. Cron Logs:

    • /var/log/cron: Cron job execution logs.

  6. Boot Logs:

    • /var/log/boot.log: Boot process details.

C. Log Management Tools

1. Manual Log Analysis

  • Use text-processing commands.

      bashCopyEdittail -f /var/log/syslog
  grep "ERROR" /var/log/syslog
  less /var/log/auth.log

2. Centralized Log Management Tools

  • Syslog Services:

    • rsyslog: Handles system logging with customizable settings for log rotation and forwarding..

    • journald: Manages logs for systems using systemd..

  • Log Aggregation Tools:

    • Elastic Stack (ELK): Elastic Stack (ELK) integrates Elasticsearch, Logstash, and Kibana to aggregate, parse, and visualize logs.

    • Graylog: Centralized log management allows for powerful querying capabilities..

    • Fluentd: Fluentd is a tool for collecting and forwarding logs.

    • Splunk: Enterprise-grade log analysis and monitoring is provided by Splunk..

3. Cloud-Based Log Management

  • AWS CloudWatch Logs, Azure Monitor, and Google Cloud Logging are used for managing logs in the cloud.

D. Best Practices for Log Management

  1. Enable Log Rotation:

    • Use logrotate to manage disk space by preventing logs from taking up too much space, as shown in the example configuration (/etc/logrotate.d/app).

        plaintextCopyEdit/var/log/app/*.log {
      daily
      rotate 7
      compress
      missingok
      notifempty
      create 0640 root root
  }

  2. Secure Logs:

    • Restrict access to log files with permissions and encrypt sensitive logs during storage or transmission.

  3. Centralize Logs:

    • Forward logs to a central server for easier management and analysis, such as using rsyslog for forwarding.

        bashCopyEdit*.* @@logserver.example.com:514

  4. Monitor Logs:

    • Set up alerts for specific patterns or anomalies, like failed login attempts.

  5. Archive Logs:

    • Store logs for long-term analysis or compliance by using backups or cloud storage.

3. Combining Monitoring and Logging

  • Integrated Tools: Tools such as Datadog, Splunk, and ELK Stack integrate monitoring and log analysis to provide a complete view of system health.

  • Alerting Systems: Use tools like Prometheus Alertmanager or Nagios to send alerts based on log events or monitoring metrics.

Combining effective monitoring with strong log management ensures a stable, secure, and high-performing Linux environment.

Linuxmanagement