MySQL Memory Consumption: Systemd Leak Case Study

Symptoms​Monitoring alerts indicated low free memory (<10%) on a database server. Running top sorted by memory usage revealed:

‍

[root@mysql-slaver ~]# top
...
KiB Mem : 32780028 total, 905684 free, 19957900 used, 11916444 buff/cache
KiB Swap: 0 total, 0 free, 0 used.

‍

The/run directory consumed 8.6GB​ of memory via tmpfs, despite minimal active processes. Key observations:

• MySQL (48.2% MEM) and Redis (9.9% MEM) dominated usage.
• Hidden files in /run/systemd/users totaled ​300,000+ entries, dating back years.

​Root Cause Analysis​

1. ​Memory Mapped to tmpfs:
   • /run uses tmpfs (in-memory filesystem), causing high buff/cache usage.
   • df -h confirmed /run occupied 8.6GB.
2. ​Systemd Session Leaks:
   • /run/systemd/users stored 300k+ orphaned session files from cron jobs.
   • loginctl list-sessions revealed ​2,131 inactive user sessions​ (mostly root).

​Key Findings​

• ​Inactive Sessions Persist: Crond-generated sessions (2018–2023) remained unclosed, consuming memory.
• ​Lsof Validation

lsof -p $(pidof dbus-daemon) | grep sessions | wc -l  
2126

‍

• Showed 2,126 open file descriptors linked to expired sessions.

​Solutions Implemented​

1. ​Short-Term Mitigation:
   • Reduced MySQL’s innodb_buffer_pool_size to free OS memory.
   • Scheduled machine reboot after Redis migration to clear tmpfs artifacts.
2. ​Long-Term Cleanup:
   • Manually purged old sessions via loginctl kill-session <ID>.
   • Configured automated session cleanup using systemd-tmpfiles:

# /etc/tmpfiles.d/custom.conf
d /run/systemd/users 0755 root root 10d

‍

Lessons Learned​

• ​Monitor tmpfs Usage: Regularly check /run and /dev/shm for abnormal growth.
• ​Audit Cron Jobs: Disable unnecessary periodic tasks to prevent session accumulation.
• ​Leverage systemd Tools: Use systemd-tmpfiles-clean.timer for automated cleanup.

​Optimized Configuration Snippet

# MySQL Memory Tuning Example
[mysqld]
innodb_buffer_pool_size = 16G  # Adjust based on workload

‍

By addressing both application-level memory allocation and OS-level artifacts, we achieved stable performance and prevented future leaks. Let me know if you need deeper insights!