Delete duplicate e-mail messages

If you need to delete duplicate e-mail messages on an IMAP server, take a look at this useful IMAP de-duplicator script:

IMAP de-duplicator – IMAPdedup

As IMAPdedup is a command line interface tool (a python script), it’s particularly useful for:

  • automated deletion of duplicates (as it can be called from other scripts)
  • extraordinarily big mailboxes or if you have many subfolders (as there’s no intervention by the user required)
  • if you have console/shell access to the IMAP server (as you can then run the script on the server itself, speeding the de-duplication process further up)

I also found that it deals relatively well with failures (e.g. when a folder is read-only and hence messages can’t be deleted): It simply reports them on the screen and carries on.

Here’s a quick’n’dirty bash script to de-dup the inbox and all subfolders of the specified account:

#!/bin/sh
# Delete all duplicate messages in all folders of said account.
# Note that we connect through SSL (-x) to the default port.

SERVER="my.server.com"
USER="mylogin"
PASS="mypass"

for folder in `imapdedup.py -s $SERVER -x -u $USER -w $PASS -l`;
do
 imapdedup.py -s $SERVER -x -u $USER -w $PASS $folder
done

If you only have to de-duplicate messages in a small folder, you could also use the following de-duplication add-on for Mozilla Thunderbird:

Remove Duplicate Messages Add-on for Thunderbird

Note however that the ‘Remove Duplicate Messages’ add-on is intended for interactive use only, not for batch processing. I also noticed that it fails at cleaning big mail folders (e.g. containing 50’000 messages).

 

Moving Zimbra Collaboration Server to a new IP address

Here’s a quick overview how to migrate a ZCS mail server (based on Ubuntu) to a new IP address:

0) Not covered here: Adjusting DNS entries. Make sure you lower the TTLs of the relevant DNS entries a couple of days in advance in order to minimize downtime for clients (e.g. set a TTL of 300 for a 5 minute downtime).

1) Set the new IP address in:
* The relevant DNS entries
* /etc/network/interfaces
* /etc/hosts
* If ZCS runs in a container/VM, don’t forget to adjust its IP address too.

2) If the new IP address is part of a new subnet, make sure to add this new subnet to ZCS’s trusted_networks, otherwise, sending (relaying) messages through ZCS from Zimbra Desktop (or any other mail client) won’t work[1]. This can be set using ZCS’s web admin interface (i.e. https://mail.myserver.com:7071/zimbraAdmin/):
Navigate to “Server settings”, then open the “MTA” tab and set something analogous to the following in “MTA Trusted Networks”:
127.0.0.0/8 w.x.y.z/26

3) Restart networking and the ZCS services (it’s important, as this adjusts the trusted_network setting in ZCS’s amavisd too):
# /etc/init.d/zimbra stop
# /etc/init.d/networking restart
# /etc/init.d/zimbra start

Alternatively, just reboot the server, particularly if it runs in a VM.

Voilà!

Note: The need for the adjustments in step 2) might come as a surprise. Authenticated messages to be relayed through ZCS apparently seem to originate from the external IP address, not localhost/127.0.0.1.

[1] A typical postfix error message might look like:
Delivery Failure Notification: Invalid address: somebody . com.zimbra.cs.mailbox.MailSender$SafeSendFailedException: MESSAGE_NOT_DELIVERED; chained exception is: com.zimbra.cs.mailclient.smtp.InvalidRecipientException: RCPT failed: Invalid recipient somebody@somedomain.com: 554 5.7.1 : Relay access denied

iTerm2 – Mac OS Terminal Replacement

iTerm2, the successor of iTerm, seems to be quite a bit better then the default Mac OS X terminal app:

iTerm2 – Mac OS Terminal Replacement.

If only every Mac OS X app (i.e. Quartz) would also support copy on select, middle button paste and the other X11-like features! [1] Further, I’d love to see a terminal app that disallows pasting (cmd-v) from the keyboard-controlled clipboard completely as this is potentially a very dangerous thing.

[1] Note: It’s possible to emulate X11’s behaviour to some degree using BetterTouchTool. It’s still not the same though as X11 distinguishes between the mouse-controlled buffer and the keyboard-controlled buffer and doesn’t just “paste from the clipboard”. For reference, see:

Notes on tracing code execution in Django and Python « SaltyCrane Blog

Eliot from the SaltyCrane blog wrote a nice Django management command that allows to easily trace a Django runserver simply by executing ./manage.py trace runserver. Works great!

Django trace tool, django-trace is [..] a Django management command that uses sys.settrace with other Django management commands. https://github.com/saltycrane/django-trace.

via Notes on tracing code execution in Django and Python « SaltyCrane Blog.

Zimbra Collaboration Server: Troubles after updating to ZCS 7.1.1 – here’s the solution.

Generally, Zimbra‘s update scripts for ZCS run smoothly. However, after updating ZCS from 7.0.1 to 7.1.1 on Ubuntu 10.04.2 LTS a couple of days ago, I noticed that most of the server’s services crashed within 1-2 hours after starting the (virtual) server.

To make a long debugging story short, here’s a summary of the problem:

The update script doesn’t properly remove old entries in /etc/rsyslog.conf when creating a new dedicated rsyslog configuration file for zimbra (/etc/rsyslog.d/60-zimbra.conf). This makes rsyslog log its own logging due to double rule entries – making /var/log/zimbra-stats.log grow at 2 MB/s. Like this, zimbra effectively DOSes itself as the server runs out of free disk space in no time (my above-average 12 GB of free space were filled within about 1 hour 40 minutes).

And here’s the solution:

1. Stop rsyslog to stop it from filling up your disk with nonsense: ‘/etc/init.d/rsyslog stop’ or ‘service rsyslog stop’
2. If necessary (i.e. if the updated server has been running for several minutes already), reclaim your free disk space by deleting/emptying big log files in /var/log, e.g. zimbra-stats.log, zimbra.log, mail.info, mail.log, mail.err, mail.warn etc.
3. Edit /etc/rsyslog.conf and remove the entries (likely at the end of the file) that look similar to these:
local0.* @mail.yourserver.com
local1.* @mail.yourserver.com
auth.* @mail.yourserver.com
local0.* -/var/log/zimbra.log
local1.* -/var/log/zimbra-stats.log
auth.* -/var/log/zimbra.log
mail.* @mail.yourserver.com
mail.* -/var/log/zimbra.log

(these are now in /etc/rsyslog.d/60-zimbra.conf)
4. Restart rsyslog: ‘service rsyslog restart’
5. Restart ZCS if it doesn’t run properly anymore: /etc/init.d/zimbra restart (you may even have to reboot the whole box if that doesn’t work)

Less is more: colordiff and more or less

In the Unix/Linux/Mac OS X world, less is more. Literally, in that ‘less‘ fully emulates ‘more‘, and figuratively, as it provides useful additional functionality like backwards scrolling. So, you really want to use ‘less’ instead of ‘more’ for paging another command’s output, e.g.

cat a_long_document.txt|less

When used to page the output of colordiff however, ‘less’ displays a mess instead of properly displaying colored output like ‘more’.
The trick is to use ‘less’ with either the -r or -R option (which both repaint the screen), i.e.

colordiff -u file_old.py file_new.py|less -r

or

colordiff -u file_old.py file_new.py|less -R

(try which one works better with your system and terminal)

tiny tiny rss: A great web-based feed reader!

I’ve just installed tiny tiny rss (tt-rss), an open source web-based news reader/aggregator for Atom, RDF and RSS feeds. Configuring it as the default news reader in Firefox is very easy (just click on the according link at the bottom of the preferences page) and a convenient solution.

The installation is pretty straightforward too, but here are a couple of hints for installing it on a Gentoo box:

1. Download the tt-rss-1.3.3.ebuild file and all other files and directories from http://overlays.gentoo.org/svn/proj/sunrise/reviewed/www-apps/tt-rss/ and place it in the www-apps/tt-rss directory (create it) in your local Portage overlay (usually /usr/local/portage).

2. Rename the file to tt-rss-1.3.4.ebuild (= the most recent version at the time of writing, released on Oct 21, 2009), execute ‘ebuild tt-rss-1.3.4.ebuild digest’, set the flags you need (e.g. for mysql and vhosts) and emerge the ebuild.

3. Follow the post-install instructions on the screen (bascially the official tt-rss installation notes)

If you intend to use the default, single-process update daemon, you can use the following init files I created (loosely based on Pierre-Yves Landure’s init script):

/etc/init.d/tt-rss:

#!/sbin/runscript
# Copyright 1999-2009 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

depend() {
    use net
    use mysql
}

#
# Function that starts the daemon/service
#
start() {
    ebegin "Starting $NAME daemon"
    start-stop-daemon --start --quiet --make-pidfile --background --chdir $DAEMON_DIR --pidfile $PIDFILE --exec $DAEMON -- $DAEMON_ARGS
    eend $?
}

#
# Function that stops the daemon/service
#
stop() {
    ebegin "Stopping $NAME daemon"
    start-stop-daemon --stop --quiet --make-pidfile --retry=TERM/1/KILL/5 --pidfile $PIDFILE --name $NAME
    eend $?
}

(replace “mysql” by “postgresql” if you use postgresql)

/etc/conf.d/tt-rss:

# Defaults for the Tiny Tiny RSS update daemon init.d script

# Location of your Tiny Tiny RSS installation.
TTRSS_PATH="/var/www/localhost/htdocs/admin/tt-rss"

DAEMON_SCRIPT="update_daemon.php"

DAEMON=/usr/bin/php
DAEMON_ARGS="$TTRSS_PATH/$DAEMON_SCRIPT"
DAEMON_DIR="$TTRSS_PATH"
PIDFILE=/var/run/tt-rss.pid
NAME=tt-rss

(make sure TTRSS_PATH points to your tt-rss installation)

4. Note that for using the default update method, PHP needs to be compiled with pcntl support. If required, set the pcntl flag and remerge PHP.

5. Have fun!

Kimai – Open Source Time Tracking Tool

So far, I’ve always used “good old” spreadsheets for time tracking on projects. Custom ones I pimped up with some nifty formulae, but still just spreadsheets. Advantage: I can easily adjust them to any special needs anytime – be it the inclusion or exclusion of specific work or just a customization of the sheet’s design or layout. The price for this flexibility is the generally higher effort to track the time “manually” rather than using a specialized time tracking tool – which makes time tracking a tedious task.

Of course I’ve evaluated many proprietary and open source time tracking tools over the years, but so far, none of them managed to fully convince me.

Today, I’ve just stumbled over Kimai – an open source, web-based time tracking tool written in PHP. And so far, Kimai looks promising. Installation is dead easy – just make sure you’ve compiled PDO support into PHP (Gentooers: enable the PDO flag for dev-lang/php and remerge php), else the nice web-based installation wizard will abort without printing any error message.

Once you’ve logged in, you’ll be presented a very clean, intuitive GUI where you can setup customers, projects and tasks. On the top-right there’s a big push-button to start/stop/pause the time tracking.

During my quick evaluation, I haven’t found the functionality yet to export the timesheets, but as far as I know, such functionality will be provided by extensions that can be installed. Let’s see. [Addition 20091009: There’s a stats extension quick-hack for Kimai 0.8.x that can be used to list and print selected reports. To use it, simply download it, extract it in the extensions folder and navigate to {Kimai install folder}/extensions/stats/]

Here’s a screenshot of Kimai 0.8.1.890:

Kimai 0.8.1.890 Screenshot
Kimai 0.8.1.890 Screenshot

With the currently still very limited feature-set, Kimai doesn’t compete with full-grown project management solutions (I’ve recently seen a quick demo of a very sophisticated and cool, Django-based project management solution I’m not allowed to tell any details about yet). But it looks like a promising start. I hope the Kimai project will gain momentum, grow and mature as there’s definitely a need for open source time tracking tools – particularly web-based ones.

P.S. I haven’t had the time yet to audit Kimai’s source code, but if the orderly, clean GUI is any indication, it can’t be too bad.