Bad start for a day!
Land at office in the morning only to see my X killed as my workstation had run out of swap space. A quick “top -o size” reveals no specific hoggers but spamd and its children were taking a lot of CPU. Hmm, mailq shows 3000+ mails in there.
Killed fetchmail before further damage happens and poked around in the mail logs. Spamd was taking a long time to do the network based tests for each of these mails. The 3000+ mails itself was a result of a script going bad. But the surprising part was the damage that spamd ended up doing to my box.
I am assuming it was spamd which ate up all the RAM and the FreeBSD VM decided to kill X for starters. Since the logs get rotated at 100KB, I had only a few hours worth of logs to rely on. They had nothing new to say about when the problem actually happened.
To prevent such things in the future, I have moved the spamc recipe in .procmailrc way down to the bottom and added addresses from which I receive a huge number of alerts to the whitelist in SpamAssassin user_prefs.
Have to now STFW to see how to increase Sendmail 8.13.x deliveries to use available spamd threads. During the queue bursts, I increased spamd count to 8 but I hardly saw 2 of them being in a run state at any point of time. The other 6 threads were in select state.
