Compiling mail/mutt-devel with WITH_MUTT_MAILDIR_HEADER_CACHE speeds up mailbox reading by a huge factor. I can now move between mailboxes at ~3 secs per switch.
Cool!
[buffy] ~> mutt -v | grep CAC
+HAVE_ICONV -ICONV_NONTRANS +HAVE_LIBIDN +HAVE_GETSID +HAVE_GETADDRINFO +USE_HCACHE