Proxmox VE 3.4 and ZFS: How to create an ISO to use lz4 compression by default

EDIT 20150607: Meanwhile, this fix has been included by Proxmox – the current official Proxmox VE 3.4 Installer ISO finally enables lz4 compression by default at installation time. I thus recommend using an official Proxmox VE ISO image instead of creating an ISO image yourself, although creating your own ISO might still be interesting from an educational point of view or to include fixes that haven’t made it into the official Proxmox ISO yet.

Proxmox VE is a truly great, Debian based Linux distribution to host both KVM virtual machines and OpenVZ containers, using a RedHat based kernel (numlock.ch runs as a KVM guest on Proxmox VE, BTW ;).

With the recent release of Proxmox VE 3.4, the whole package got even better, now also allowing to install Proxmox on top of ZFS (which is the best enterprise file system available to date), specifically its native port for Linux, ZFS on Linux (ZoL).

Unfortunately, Proxmox VE 3.4 doesn’t allow to use lz4 as a compression algorithm at installation time yet: Using lz4 would be strongly recommended however instead of using ZoL’s default lzjb (or much worse: gzip). There are some known workarounds for this, but those are rather tedious, either involving extensive file copying or a manual setup of the ZFS pool and file systems.

The most elegant solution at the moment is creating a bootable ISO image (based on the original bootable Proxmox VE 3.4 ISO image) with a patched /usr/bin/proxinstall script that uses “compression=lz4” instead of “compression=on” by default.

And here’s how to create the patched ISO:

1. Loop-mount the original ISO (can only be mounted read-only):

# mount -o loop /path/to/proxmox-ve_3.4-3f2d890e-1.iso /mnt/cdrom

2. Copy /usr/bin/proxinstall from the mounted ISO image to a read-writeable directory (create directories as necessary):

# cp -a /mnt/cdrom/usr/bin/proxinstall /mnt/cdrom_patched/usr/bin/proxinstall

3. Edit the /mnt/cdrom_patched/usr/bin/proxinstall script as follows:

# diff -u ../cdrom/usr/bin/proxinstall usr/bin/proxinstall 
--- ../cdrom/usr/bin/proxinstall	2015-02-12 17:52:50.000000000 +0100
+++ usr/bin/proxinstall	2015-03-17 21:50:07.662031284 +0100
@@ -592,7 +592,7 @@
     # disable atime during insatll
     syscmd ("zfs set atime=off $zfspoolname") == 0 ||
 	die "unable to set zfs properties\n";
-    syscmd ("zfs set compression=on $zfspoolname") == 0 ||
+    syscmd ("zfs set compression=lz4 $zfspoolname") == 0 ||
 	die "unable to set zfs properties\n";
 }

4. Use the complicated but very powerful tool xorriso (install it, if necessary) to create a bootable ISO image based on the original ISO, but with /usr/bin/proxinstall “overwritten” with the edited proxinstall script:

# xorriso -boot_image grub patch -indev proxmox-ve_3.4-3f2d890e-1.iso -overwrite on -outdev proxmox-ve_3.4-3f2d890e-1-with-lz4-patch.iso -blank as_needed -pathspecs on -add /usr/bin/proxinstall=/mnt/cdrom_patched/usr/bin/proxinstall -- -commit

5. Use this patched ISO to install Proxmox VE 3.4 as usual

After rebooting the installed Proxmox VE 3.4 host/server, use the following command to verify that lz4 has been used by default:

# zfs get compression
NAME              PROPERTY     VALUE     SOURCE
rpool             compression  lz4       local
rpool/ROOT        compression  lz4       inherited from rpool
rpool/ROOT/pve-1  compression  lz4       inherited from rpool
rpool/swap        compression  lz4       inherited from rpool


For your convenience, here’s a patched ISO of Proxmox VE 3.4 for download (Note: Use it at your own risk!)

proxmox-ve_3.4-3f2d890e-1-with-lz4-patch.iso_.gz (571 MB, md5sum: 2abba5445133c011aadb1808237202b0)

Download it, gunzip it and rename it to proxmox-ve_3.4-3f2d890e-1-with-lz4-patch.iso to get rid of the silly underscore (“_”) WordPress appended when I uploaded the file. The resulting ISO image will be 725 MB, md5sum: e09e5d250d16fa182129c72be88a5aa2.

EDIT 20150607: Meanwhile, this fix has been included by Proxmox – the current official Proxmox VE 3.4 Installer ISO finally enables lz4 compression by default at installation time. I thus recommend using an official Proxmox VE ISO image instead of creating an ISO yourself.

Have fun!

Microsoft and Skype’s privacy settings

I don’t know exactly when these settings were introduced in Skype, but it’s obvious that they haven’t been there in this form before Microsoft’s acquisition of Skype.

The irritating thing is that one of my Skype accounts had the “Allow Microsoft to use your Skype profile information to show you more relevant ads in this application” setting enabled and I can’t remember (nor imagine) I ever knowingly allowed ad targeting.

skype_privacy

The other one is “Allow Microsoft to collect information on my behalf to make the service better” which they decided to hide in the “advanced” settings tab, although it’s clearly very privacy relevant (just read the disclaimer).

skype_advanced

So, make sure to revise all your settings in Skype, particularly those in the privacy and advanced tabs.

Homebrew package for t-prot: TOFU protection for e-mail

In a current LUGS mailing list thread, mowgli pointed to the t-prot script as a nice tool to “fix” the so-called (and very common) TOFU style message quoting and some other annoyances (like commercial webmail provider signatures, overly excited !!!! statements, etc.) in e-mail messages. t-prot was originally developed for mutt, but can also be used with other mail clients.

As there wasn’t any t-prot package for my favourite Mac OS X package manager Homebrew yet (apparently only for MacPorts), I quickly created one myself [Edited on Feb 24, 2015: Updated for t-prot version 3.3]:

  1. Create the file /usr/local/Library/Formula/t-prot.rb and add the following (fantastically trivial) content:
    require "formula"
    
    class TProt < Formula
      homepage "http://www.escape.de/~tolot/mutt/"
      url "http://www.escape.de/~tolot/mutt/t-prot/downloads/t-prot-3.3.tar.gz"
      sha1 "9be24629abc598b24e887f47e9a62feddc3ed4ad"
    
      def install
        bin.install 't-prot'
        man1.install 't-prot.1'
      end
    
    end
    
  2. Install t-prot (run the following Homebrew command in a terminal window):
    brew install t-prot
  3. Add the following line to your .muttrc (or check ‘man t-prot’ or the t-prot web page for options and examples):
    set display_filter="t-prot -cemt -M=mutt"

Guaranteeing memory safety in Rust | Air Mozilla

Nicholas Matsakis of Mozilla on how Rust guarantees type soundness, memory safety, and data-race freedom while still offering control similar to C/C++:

(via Guaranteeing memory safety in Rust | Air Mozilla.)

Rust, with its funny name, reminds more of a mutt than an academic programming language. It’s that pragmatism however, that has a lot of potential to help solving well-known “real world” reliability problems in software engineering. It would definitely make a worthwhile addition to other programming languages taught in CS classes.

Mozilla has already started building Servo, a web browser engine based on Rust. While still far from usable (Disclaimer: I’ve just compiled and quickly tested the current master branch sources on Mac OS X Mavericks), it’s the right strategic move and I wonder when other browser projects, like Chromium, will start shifting to safer programming languages too.

Modern web browsers are very complex and accordingly vulnerable pieces of software and yet, with the rise of the browser as an app platform, we depend more and more on their reliability, safety and security.

Ironically, my Chrome browser irrecoverably froze while I was drafting this blog post. Although chrome doesn’t rust, some Rust would actually be a good thing for Chrome, in the long term.

Using IRC on the go

Running the console IRC client weechat (not to confuse with the wechat messenger) in a screen or tmux session on a Linux (or Mac OS X) server and accessing it trough mosh is a great way to use IRC on the go, i.e. with a unsteady Internet connection.

P.S. Thanks to Devaux and rmeyer for the hints

Zimbra ZCS: OpenDKIM not running

After upgrading Debian, OpenDKIM sometimes stops running or fails to properly restart. If you don’t actively monitor your ZCS server, the only way to detect this is either through a growing mail queue of unsent messages or through a note in the ZCS server admin panel.

Usually, manually restarting either OpenDKIM itself or the whole ZCS server once “fixes” the problem:

# service zimbra restart

or

# /etc/init.d/zimbra restart

In other cases, fixing the permissions may help:

# /opt/zimbra/libexec/zmfixperms -verbose -extended

(source)

Some helpful Git resources

A friend recently told the following joke:

“The idea that git can be used offline is an illusion – you still need connectivity for googling which arguments to pass to what command.”

That’s an exaggeration, of course, but as always, there’s a grain of truth in it. So here we go: