Gentoo – A changelog by Daniel Mettler

September 6, 2009September 6, 2009

Gentoo: Mailman 2.1.11 incompatibility with Python 2.6

Mailman 2.1.11 (or earlier) isn’t compatible to Python 2.6. So if you upgrade your box to Python 2.6 without upgrading to Mailman 2.1.12 at the same time, you’ll run into troubles. And if you’re unlucky (e.g. when running some low-volume mailing lists), you might not even notice it.

One sign of such troubles is that messages sent to your Mailman mailing lists aren’t processed anymore. They simply seem to get “swallowed” by your Mailman server: Messages don’t reach the list, don’t get forwarded to subscribers and there’s no bounce or failure notice. For low-traffic lists, this might go unnoticed for several days or even weeks.

Another sign (that can easily slip through under certain, quite common circumstances) is that as long as you’re running Mailman 2.1.11 with Python 2.6, cron will, every 5 minutes, send an error message to mailman@myhost.com with a subject similar to

Cron <mailman@myhost> /usr/bin/python -S /usr/lib64/mailman/cron/gate_news

and a content like

/usr/lib64/mailman/Mailman/Utils.py:32: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
import sha
Traceback (most recent call last):
File “/usr/lib64/mailman/cron/gate_news”, line 44, in <module>
from Mailman import MailList
File “/usr/lib64/mailman/Mailman/MailList.py”, line 51, in <module>
from Mailman.Archiver import Archiver
File “/usr/lib64/mailman/Mailman/Archiver/__init__.py”, line 17, in <module>
from Archiver import *
File “/usr/lib64/mailman/Mailman/Archiver/Archiver.py”, line 32, in <module>
from Mailman import Mailbox
File “/usr/lib64/mailman/Mailman/Mailbox.py”, line 21, in <module>
import mailbox
File “/usr/lib64/python2.6/mailbox.py”, line 19, in <module>
import email.message
ImportError: No module named message

That by itself wouldn’t be a big problem and probably get noticed quickly if you forwarded mailman@yourhost.com to your admin’s e-mail address. The problem arises if you happen to host a mailing list named “mailman” (which is often set up as a mailing list for testing and debugging). In that (not uncommon) case, all these cron error messages are forwarded to the “mailman” mailing list. This makes about 288 messages per day that are queued in /var/lib/mailman/qfiles/in and won’t be delivered (due to the compatibility problem with Python 2.6).

Once you upgrade to Mailman 2.1.12 (and restart /etc/init.d/mailman), all these messages in /var/lib/mailman/qfiles/in will be queued by your MTA (e.g. postfix). This may seem like a strange loop problem in Mailman (or your MTA configuration), which is quite irritating at first.

Here’s how to solve the problem (you may need to adjust these steps to your settings/system paths):

1) Temporarily stop the Mailman service

/etc/init.d/mailman stop

2) Delete all the queued messages to/for mailman-owner@myhost.mydomain.com in your MTA’s mail queue. For postfix as MTA, the following script may be helpful: mailq by Dan Mick.

3) Delete all the cron-generated error messages in /var/lib/mailman/qfiles/in.

In order to determine these error messages, use Mailman’s show_qfiles command to view the message content, e.g.

/usr/lib64/mailman/bin/show_qfiles /var/lib/mailman/qfiles/in/longnumber.longnumber.pck

The best way to identify these messages is by filtering them according to their file size. Usually they have sizes around 1600 bytes. E.g. for a file size of 1635 bytes, try sth similar to:

cd /var/lib/mailman/qfiles/in

find ./ -size 1635c -exec rm {} \;

4) Once you’ve deleted all these cron error messages in Mailman’s in-queue, you can restart Mailman (/etc/init.d/mailman start). Mailman will then start delivering the remaining valid files in its in-queue. Your MTA/postfix queue looks normal again (i.e. there’s no more overflowing)!

5) Finally, you need to manually “discard all messages marked Defer” for the ‘mailman’ mailing list using the web admin interface (usually on https://myhost.mydomain.com/mailman/admindb/mailman or similar). Before discarding these messages, make sure you don’t discard any valid messages.

That’s it!

Thanks to Mark Sapiro AKA msapiro from #mailman @ irc.freenode.net for very useful hints and help!

P.S. Some other helpful resources in case of Mailman problems:

How to find running Mailman instances (qrunners)

How to integrate postfix and Mailman

September 5, 2009

Gentoo, Django and PyFacebook: Updated Ebuild

If you happen to use PyFacebook on Gentoo you probably noticed that djangofb.py didn’t create/copy any skeleton files for the Facebook application to be created (for an example how it should work, take a look at the PyFacebook tutorial).

In order to fix this, just download my update of the pyfacebook-9999.ebuild. This updated ebuild reflects PyFacebook’s move to GIT for versioning (from SVN), hence ensuring that you always use the latest available (‘official’) sources with latest fixes (among others, some changes to /pyfacebook/setup.py to fix the djangofb problem). I also recommend to watch the forks of PyFacebook for recent changes.

September 5, 2009

Gentoo: How to fix a broken Python installation

If your Python environment seems to be buggy or broken after a recent Python upgrade, it probably is. To fix it, simply execute the following command on your Gentoo box:

# python-updater -v

(note that this process takes quite some time to complete)

Additionally, you might want to make Gentoo check the dependency tree and rebuild broken packages (related to Python or other packages):

# revdep-rebuild

P.S. This is more or less a repost of an earlier post about python-updater.

August 14, 2009

Gentoo: Ebuild for CakePHP 1.2.4.8284

I’ve just posted a new ebuild for CakePHP 1.2.4.8284:

http://bugs.gentoo.org/show_bug.cgi?id=221299

It’s only a version bump, but once again, the naming format of the archive on the servers has changed, requiring a minor adjustment in the ebuild.

April 26, 2009April 26, 2009

Gentoo Anti-Spam: iXhash Ebuild

As spam levels increased quite a bit lately, I made a Gentoo Ebuild for the iXhash SpamAssassin plugin by Bert Ungerer / Heise iX, Germany.

So far, it’s available on bugs.gentoo.org only. In order to install it, download

iXhash-1.5.5.ebuild

and put it in your local Portage Overlay.

YMMV, but it performs really well on my server (particularly for German spam which is difficult to catch without). Don’t forget to adjust the according penalty scores to your liking, though!

April 21, 2009

Gentoo: Greylisting for Postfix using Postgrey

Finally, I’ve had to enable greylisting for this domain due to ever increasing levels of spam. Fortunately, setting it up is very easy. For Gentoo and Postfix, here’s a nice walkthrough:

postfix greylisting on gentoo (postgrey) | andreas d.’s.

Let’s hope that this, in conjunction with other anti-spam measures, will do it for a while.

Non-spammers shouldn’t experience any problems due to this change, but if you do, please contact me using Skype (mettlerd), phone or SMS.

March 9, 2009March 8, 2009

Gentoo ebuild for Lx-Office ERP 2.6.0 beta 1

Finally, I’ve created Gentoo ebuilds for Lx-Office ERP 2.6.0 beta 1 and its dependencies. Lx-Office is a fork of the server-based open source accounting solution SQL-Ledger and customized for the German market (and to some extent, the Swiss and Austrian markets).

A screenshot of Lx-Office ERP 2.6.0 beta 1 showing the XUL menu:

And here’s a flash video of Lx-Office ERP 2.6.0 beta 1 showing the XUL menu in action.

Lx-System (the company backing Lx-Office ERP) and LINET Services host a public demo of Lx-Office ERP 2.4.3 (user: demo, password: demo).

To install Lx-Office ERP 2.6.0 beta 1 on Gentoo, follow these steps:

1) Set up a local portage overlay (e.g. at /usr/local/portage), if you haven’t done so already.

# mkdir -p /usr/local/portage

In /etc/make.conf, set

PORTDIR_OVERLAY=/usr/local/portage

2) Download lx-office-erp-2.6.0_beta_p1-r1_plus_dependencies.tgz and extract it to your local portage overlay

# cd /usr/local/portage

# tar xzvf lx-office-erp-2.6.0_beta_p1-r1_plus_dependencies.tgz

3) In /etc/portage/package.keywords, add the line

www-apps/lx-office-erp ~amd64

(or ‘www-apps/lx-office-erp ~x86‘, depending on the architecture of your machine)

4) In /etc/portage/package.use, add the line

www-apps/lx-office-erp vhosts

5) Install Lx-Office ERP on your system by executing

# emerge -av lx-office-erp

Depending on your current portage settings and installed ebuilds, you may need to unmask additional ebuilds.

6) Use webapp-config to link your Lx-Office ERP installation to a specific host, e.g. by executing

# webapp-config -I -h localhost -d lx-erp lx-office-erp 2.6.0_beta_p1-r1

7) Follow the steps displayed on the screen to setup and configure Lx-Office ERP. Some of these steps might be automated in a later release of the ebuild.

8) Have fun using Lx-Office ERP on Gentoo! :)

(These ebuilds are sponsored by my company Printscreen GmbH, dedicated to the developers of Lx-Office ERP and Gentoo and released for free use under the terms and conditions of the GNU GPLv2 license.)

January 31, 2009

Gentoo: TLS/SSL and name-based Apache virtual hosts using mod_ssl

With the introduction of the TLS SNI extension (transport layer security server name indication), name-based virtual hosts (i.e. virtual hosts sharing the same IP address) can now use distinct SSL certificates.

Here’s how to configure TLS SNI on Gentoo, using Apache:

1) DON’T try to set up TLS SNI using mod_gnutls. When I tried it on January 30, 2009, mod_gnutls still seemed to be too experimental and unstable for reliable TLS SNI (I observed some non-deterministic behaviour).

2) It does work fine though with a reasonably current version of OpenSSL and a patched Apache mod_ssl. And here, Gentoo shines once again: With Gentoo, you don’t need to manually patch Apache or OpenSSL! All you have to do is add the “sni” use flag to Apache in /etc/portage/package.use:

www-servers/apache sni

and remerge Apache (’emerge -auDNv apache’).

Note that Gentoo’s OpenSSL is already SNI-enabled by default (since openssl-0.9.8g-r2.ebuild) – remerge/update OpenSSL if required.

The configuration of SNI-enabled name-based virtual hosts in Apache is transparent, i.e. works the same way as for SSL-enabled, non-SNI name-based virtual hosts (of course you’ll want to use different certificates for every virtual host and specify them, as explained on Kaspar Brand’s TLS SNI test site).

Have fun!

December 30, 2008

Gentoo: MOTD printed twice

With some recent changes, Gentoo now seems to print the message of the day (MOTD) twice, when logging in using SSH:

Last login: Mon Dec 29 04:45:22 UTC 2008 from box on ssh

(message of the day)

Last login: Mon Dec 29 22:55:09 2008 from box

(message of the day)

I tried the following solution

Looks like the answer was in the /etc/pam.d/login and /etc/pam.d/system-login files. I had to comment out the pam_motd.so lines.

as described on http://stackoverflow.com/questions/190912/etcmotd-printing-twice-on-gentoo-linux. With these quick fixes, the MOTD doesn’t get printed twice anymore.

In addition, I also had to comment out the

session optional pam_lastlog.so

lines in /etc/pam.d/login and /etc/pam.d/system-login in order to prevent the “last login” line from being printed twice.

November 1, 2008

Best practice tips for Gentoo sysadmins

Currently, there are some critical ebuild dependency issues in Gentoo’s portage tree that might seriously hurt your box. Symptoms: When updating your system, portage might display an error message similar to this one:

[ebuild     U ] sys-fs/e2fsprogs-1.41.2 [1.40.9] USE=”nls (-static%)” 4,263 kB
[ebuild N    ] sys-libs/e2fsprogs-libs-1.41.2 USE=”nls” 479 kB
[blocks B     ] sys-libs/ss (is blocking sys-libs/e2fsprogs-libs-1.41.2)
[blocks B     ] <sys-fs/e2fsprogs-1.41 (is blocking sys-libs/e2fsprogs-libs-1.41.2)
[blocks B     ] sys-libs/com_err (is blocking sys-libs/e2fsprogs-libs-1.41.2)
[blocks B     ] sys-libs/e2fsprogs-libs (is blocking sys-libs/ss-1.40.9, sys-libs/com_err-1.40.9)

The important thing: DON’T unmerge ss or com_err, as it will break wget and other essential parts of your system! Portage thus won’t be able to download e2fsprogs-libs-1.41.2 which is required to replace the removed ss and com_err libraries (which are part of e2fsprogs-libs starting with v1.41.2).

Solution: Either wait until this issue gets resolved by the Gentoo core dev team or read through the following posts and Gentoo bug reports:

If you really know what you do, you might want to try this suggested quickfix (Important disclaimer: Looks reasonable and fine as a quick workaround to me, but I haven’t tried it yet. You apply it at your own risk, as usual! Note that this workaround doesn’t solve the real problem.)

As serious issues like these are quite common in Gentoo, here are some best practice tips for Gentoo sysadmins that help prevent some of the potential problems:

First of all, try to use stable ebuilds only. If this is not possible for some reason, try to minimize the number of unstable ebuilds (~amd64 etc.) on your system.
It’s rather tempting, but DON’T setup a cronjob to do automatic emerges! Portage only catches the most evident issues, but emerging new ebuilds is never without risks (not updating your system is risky too, however). The best approach would be testing any updates on a test box first before installing them on a production system. The second best approach is probably doing a monitored, manual update in small, incremental steps with immediate testing afterwards. This helps isolating problems, should they occur (it’s difficult to isolate a problem that was detected after an automatic update of hundreds of ebuilds).
Automate ’emerge –sync’ by putting it in your daily crontab in order to refresh your portage tree regularly. That’s neither particularly safe nor unsafe, but it guarantees that you don’t emerge that weeks-or-months-old broken ebuild that has been fixed in the meantime.
Regularly fetch new source packages by setting up a cronjob for ’emerge -uDN –fetchonly world’ (or -f). Like this, portage uses some additional hard disk space for the package sources (always make sure you have enough free space and properly setup partitions/volumes!). It makes sense though as one day, you’ll use most of these source packages anyway and having a source package locally can be very helpful in a situation like the one described in this post. IOW: If you aren’t able to download anything anymore due to a severely broken system, chances are, that you can still solve the problem on localhost, if you have source packages at hand.
Append “buildpkg” to the FEATURES variable in /etc/make.conf. Like this, portage will additionally create binary packages in /usr/portage/packages/All when emerging new ebuilds. This will require some spare free space on your hard disk again, but having a prebuilt, binary package at hand can be very helpful if there are any problems with the gcc toolchain or any other compiler chain needed. If you don’t like to enable this feature permanently, you can use the -b or –buildpkg option when executing emerge.
If you haven’t used the “buildpkg” feature so far, you can create binary packages of all the installed ebuilds on your system using the “quickpkg” utility and my quickpkg_all bash script.
Keep old, compiled kernel images in /boot and listed in your /boot/grub/menu.lst. Booting a new manually configured and compiled kernel is always a bit of an adventure (unless it was tested on an identical box before), and it’s good to keep previous kernels that are known to work. Even if it doesn’t work perfectly, it can take you to a console login prompt at least.
Instead of doing things the regular “remove old packages first, then install new packages” way, get used to the Gentoo way of doing things: “install new packages first, then remove old packages (if at all)”. Avoids serious problems that can occur when accidentally deinstalling an old, seemingly no longer used package that other important packages depend on and don’t work without.
When merging new configuration files, use dispatch-conf instead of etc-update. dispatch-conf uses CVS to create backups of old config files (which can be a helpful source of information in some situations). See the value of the “archive-dir” setting in /etc/dispatch-conf.conf.
Add files and directories to CONFIG_PROTECT, if in doubt. It’s better to have one ._cfg0000_XXX file too much than an important configuration file accidentally overwritten by portage.
Use emerge’s -D option for improved (deep) dependency checking.
Do regularly use revdep-rebuild to check for broken dependencies and to remerge the according ebuilds.
And of course, create automated, incremental backups of your systems regularly. You’ll sleep better, believe me ;)
Monitor your systems for errors. I do it with some custom bash scripts I wrote, but there are many full-fledged monitoring solutions for general purpose health monitoring.
As a fallback for (some) DNS problems with DHCP-based systems, I regularly send a heartbeat of a DHCP system to a box in another network, revealing the DHCP system’s last known/assigned IP address.
Not limited to Gentoo sysadmins: Having a (hardware) remote console accessible via a different IP address is worth a lot in case there are serious troubles with the operating system or the hardware.

Feel free to add other helpful best practice tips for Gentoo sysadmins!