Author: Daniel Mettler

Discovered a strange behavior (some might call it a bug ;) of MT’s notification feature. MT doesn’t seem to ensure that 8-bit mail headers are properly encoded (“7-bit clean”), as the following extract of an amavisd-new error message shows:

INVALID CHARACTERS IN HEADER

Non-encoded 8-bit data (char FC hex) in message header ‘Subject’
Subject: …o ‘Registration opened for OSCOM 4 in Z\374rich’\n
^
This nondelivery report was generated by the amavisd-new program
at host melon. Our internal reference code for your message
is 09847-03.

WHAT IS AN INVALID CHARACTER IN MAIL HEADER?

The RFC 2822 standard specifies rules for forming internet messages.
It does not allow the use of characters with codes above 127 to be used
directly (non-encoded) in mail header (it also prohibits NUL and bare CR).

If characters (e.g. with diacritics) from ISO Latin or other alphabets
need to be included in the header, these characters need to be properly
encoded according to RFC 2047. This encoding is often done transparently
by mail reader (MUA), but if automatic encoding is not available (e.g.
by some older MUA) it is the user’s responsibility to avoid the use
of such characters in mail header, or to encode them manually. Typically
the offending header fields in this category are ‘Subject’, ‘Organization’,
and comment fields in e-mail addresses of the ‘From’, ‘To’ and ‘Cc’.

Sometimes such invalid header fields are inserted automatically
by some MUA, MTA, content checker, or other mail handling service.
If this is the case, that service needs to be fixed or properly configured.
Typically the offending header fields in this category are ‘Date’,
‘Received’, ‘X-Mailer’, ‘X-Priority’, ‘X-Scanned’, etc.

If you don’t know how to fix or avoid the problem, please report it
to _your_ postmaster or system manager.

Such messages hence aren’t standard-compliant and might get rejected by some picky mail hosts (which is the correct behavior). This particularly limits the use of MT in non-US countries (where 8-bit characters are much more common)

Other things I found out (mostly trivial things, but eventually nice to know as they’re not always intuitive):

* Gentoo doesn’t seem to ship with the Mail::Sendmail Perl module. If you use Gentoo, don’t set “MailTransfer smtp” in mt.cfg as it probably won’t work. Instead either set “MailTransfer sendmail” or simply comment it out (as “sendmail” is the default). You don’t need to change or enable “SendMailPath /usr/sbin/sendmail” as this is the default setting which is fine for Gentoo.

* You only need to add your e-mail address to MT’s “Notification List” if you’d like to receive a notification when new entries are posted. Usually, this feature only makes sense for readers who don’t use an aggregator already.

* If you just like to receive notifications when new comments are posted, you only need to

1) tell MT to send e-mail messages through sendmail (see above)
2) Enable “Email New Comments” in your weblog’s “Preferences” screen (at the bottom of the page under “Comment Configuration”)
3) enter your e-mail address in your user profile (click on your username to get there)
4) Eventually make sure you only use 7-bit characters for the subjects of your entries and the name of your blog (as long as this issue isn’t fixed)

Finally, e-mail notification works for me :) IOW: I’m ready to receive all the comment spam twice ;) Probably time to install MT-Blacklist soon. [1]

[Addendum: useful links (Movable Type User Manual: TROUBLESHOOTING):
I never receive email notifications for comments
My webserver doesn’t have sendmail]

[1] Update: I’ve just received my first comment spam as an e-mail message too, but guess what, spamassassin has correctly qualified it as being spam :) Soon, MT-Blacklist will be added to this blog.

As I didn’t have time to redesign the whole site for use with WordPress, I simply upgraded this blog to MovableType 3.01D. It went smoothly although there is still much space for improvements regarding the deployment and the upgrading path of MT (it still largely relies on manual file handling etc. -> if I were SixApart, I’d sponsor people to better streamline MT’s deployment with current operating system distributions – the lower the entry barrier, the steeper the adaptation curve – dead simple. Once you’re in, migrating to another blog soft usually requires considerable efforts and costs – see above. Well, yeah, there’s the TypePad business, but I tend to favor offensive strategies..). As a (positive) side-effect of manually upgrading MT, I could clean up the directory structure a bit.

Surprisingly, MT 3.01 (MT 3.1 allegedly even more) brings quite some improvements. The most obvious being the brushed-up admin interface which offers better usability and among others – finally – HTML shortcut buttons for Mozilla Firefox users too :) Spam protection seems to be better now: Anonymous comments can be moderated, there’s optional TypePad authentication and comments can be deleted effortlessly through the admin UI.

As a conclusion, there’s still the major drawback of a not-so-clever architecture (rebuilding static entries instead of generating them dynamically). Nevertheless I like the improvements MT 3.01D brings as much that I’m thinking about keeping it (instead of migrating to WP). Well, for now, I keep it – experience doesn’t hurt.

this was the most massive blogspam attack i ever experienced: within the last two days, this site was hit by 87 blogspam comments! which equals 87 * 3 + (87 \ 5) * 2 = 312 clicks just to get rid of them again through moveabletype’s admin interface[1]. sheesh.

for spammers, blogspamming is even easier than e-mail spamming as so far, none of the currently available blog apps offers a convincing way to deal with this problem (and it’s more effective too – people trust google’s page ranking more than a filthy e-mail spam message).

* ip based blocking doesn’t work as ips are spoofed anyway (yes, i verified it)
* content-based blocking only works in few cases – the comments i had two delete were full of intentional misspellings to circumvent any blacklist-based filtering approach.
* disabling direct links: might work in the longterm, but at the moment, spammers obviously don’t care (my site doesn’t allow direct links and yet it was spammed. they don’t even seem to have checked the site prior to spamming). reason: so far, too few blogs use such a feature – and those few don’t count when mass-hammering thousands (or millions) of blogs.
* renaming comment-scripts. this site is proof enough that this approach is not effective either.
* requiring posters to decipher distorted signs/numbers/words. might work but can be circumvented as machine character recognition improves. use questions/phrases instead? mostly annoying for real humans, not computers.
* disabling anonymous comments or requiring users to register at a central registry. might work as long as there aren’t any spammer scripts to create fake accounts prior to spamming. a matter of time only. and a blog is supposed to encourage a spontaneous public discussion, right? a classical trade-off between free access and control (analogous to e-mail spamming).
* moderate all posts or those with more than # links. doesn’t really fit the idea of a low-barrier communication media (i smell censorship). takes too much time in general and particularly for things like filtering spam you don’t want to spend much time for. basically just a human spam-filter, not a wise approach. prevents any spontaneous discussion.
* distributed (almost) real-time blacklisting based on comment fingerprints/hashes. one measure that might work (unless they use randomly generated characters). it’s what works best against e-mail spam (based on my experiences as a user of spamassassin). note the word “distributed”. distributed problems tend to require distributed solutions as else one usually runs into scalability troubles pretty soon.

nevertheless i’m optimistic that sooner or later {e-mail|blog}spamming will be a thing of the past. it’s on everybody’s radar now :>

eventually i should give mt 3.01d or 3.1 a try. or migrate to wordpress which doesn’t seem to be a high-profile target atm..

anyway.. happy “1st of august” (swiss national holiday ;)

[1] instead, using sql queries through phpmyadmin or the mysql console is suggested. not feasible for joe average bloggers however.

signed fingerprints.