I’ve been trying to debug some memory problems on a ReadyNAS 1100. It has munin-node running, and I see the ‘app’ memory slowly raise something like 50-100MB a day. What’s odd is that Munin reports that it’s using 230MB of ram for ‘apps’ while memstat only reports 118224k (118MB or so), making it difficult to track down where the memory is going.

‘free’ and ‘/proc/meminfo’ only report the amount of free memory, and the amount of memory in buffers and cache other other little kernel bits. There’s no clear value for memory used. Munin calculates the used memory by subtracting other bits from memory total. I can’t find a lot of information about meminfo beyond this sort of descriptive bits about what each value means. It seems to be that if the memory is allocated, but not to buffers or cache or other small things, we assume it’s used by applications but that doesn’t pan out with tools that I can find to tell me how much memory an application is using.

The description here of the difference between VSZ (virtual size) and RSS (resident set size) is useful for looking at ‘ps aux’ output, but there’s nothing there that is using a ton of memory and feels like it’s count is pretty close to that generated by ‘memstat’.

The smugmug discussion about swappiness is interesting, as that was originally my problem because running out of memory with vm.swappiness set to 0 got the OOM killer going buck wild.  This discussion has recently made it to the lkml.

I’ll probably post to the lkml if I don’t figure something out this afternoon, as I’ve been staring at a lot of numbers lately.