Paragon CampTune – a handy tool to resize the Boot Camp partition

Recently I ran out of space on a Boot Camp partition with Windows 10 Pro. So I looked around for ways to make more space for Windows by shrinking the macOS partition and enlarging the Windows partition. Apple doesn’t officially support this in Boot Camp without reinstalling Windows, and doing these operations by hand, e.g. with the help of GNU Parted, is time consuming and tedious.

Luckily, I stumbled over Paragon CampTune, a commercial macOS utility (ready for 10.14 Mojave) that automates these tedious tasks and allows to resize the macOS and Boot Camp partitions on the fly, without having to reinstall Windows or macOS.

It finally worked wonderfully, the only irritating thing was that the tool showed a bland error at the first start of the repartitioning process: “Object not found”. After restarting the process with slightly different partition sizes, it could be successfully completed.

I can thus recommend this handy utility as it can save hours of work for a few bucks (ca. 22 USD).

Beware: iOS 12 may have deleted your voice memos! How to recover them using iTunes and Time Machine.

Apple’s hardware and software ecosystem generally provides a fantastic UX by tight integration, relatively good usability (compared to most competitors) and good services and support.

Sometimes, users may be bitten by terrible hw and sw bugs nonetheless. The worst Apple software bug I’ve personally experienced so far is that I recently noticed that all my many voice memos on my iPhone were gone! The exact reason why and how this happened hasn’t been fully investigated yet, but the observed issue is evidently linked to the recent iOS 12 update release (and potentially also to later iOS 12.x releases, as neither the problem nor any fix have been mentioned in any of the iOS 12.x update changelogs to date). The data loss seems to have to do with the introduction of the revamped voice memo app (that is now also available on the iPad), perhaps due to a bug in the iCloud synchronisation.

So, lots of invaluable voice memos I recorded for beloved ones, recordings of important conversations and thoughts, all gone! Nothing to be delighted about, to say the least!

I thus contacted Apple support and they were very kind and keen to determine the problem and find a mitigation or solution (I was on the phone with them for roughly 45 minutes, involving 2nd level support too). They didn’t find any such issue mentioned in their support database though and apparently, none of the supporters I talked to ever heard of anything like that before, despite it being mentioned all over twitter). For understandable legal reasons, as they never recommend “random” 3rd party tools as a matter of principle, they couldn’t recommend the solution detailed in this blog post, but suggested a more general solution, involving only Apple tools and services (see at the very bottom of this post; if you don’t mind the risk and backing up your iPhone to iCloud, you could try this alternatively).

After a quick assessment, I decided to go with another solution as all in all, it seemed more transparent and promising, less time consuming and with an acceptable risk for me.

Idea and big picture: Extract the lost voice memos from an old iPhone backup in iTunes, ideally the latest backup before the release/installation of iOS 12, i.e. before Sept 17, 2018.

Disclaimer 1: Although the following tips worked fine for me, I can’t guarantee they’ll work for you too – you follow these steps at your own risk. If in doubt, I recommend backing up all your data redundantly on various media before.

Disclaimer 2: Let me tell you this first: If you don’t have any backups of your iPhone, don’t use iTunes for iPhone backups and don’t have Time Machine backups of those backups in iTunes, you’re likely out of luck. At least I didn’t find any method to restore the disappeared voice memos on the iPhone itself – once iOS 12.x is installed, those memos apparently weren’t anywhere on the device itself anymore. Considering iOS 12’s and iPhone’s more and more restrictive data protection measures, it would likely also be difficult for professional data recovery services to recover any lost voice memos from the device (they would need to be technically on par with secret services and forensic experts -> seldom and accordingly expensive).

So here are the detailed steps to follow:

  1. As precaution, create a current Time Machine backup of the Mac that holds the iTunes backups.
  2. Restore from Time Machine to the Mac the latest iPhone backup before iOS 12 was installed on your iPhone. Usually, this should be the latest Time Machine backup before Sept 17, 2018:
    1. Determine the relevant iPhone backup in iTunes
      1. Open iTunes
      2. In the iTunes preferences, navigate to the “Devices” pane
      3. Figure out the relevant backup of your iPhone
      4. Right-click on the entry and choose “Show in Finder”
    2. Enter Time Machine (by clicking on the according icon in the macOS menu bar and by choosing “Enter Time Machine”) and restore the latest version of the selected folder that was created before Sept 17, 2018, from your Time Machine backup. Note that restoring that folder can take anything from  10 minutes up to an hour or two, depending on the iPhone’s storage size, your network’s bandwidth etc.
  3. If your iPhone backups in iTunes aren’t encrypted, you can try using a free open source tool like the Open Backup Extractor (download page for the binary) to extract your voice memos from the backup in iTunes. Note: You use this tool at your own risk, like any tool and suggestion here (I didn’t review or audit it).
  4. If your iPhone backups in iTunes are encrypted: In this case, the above free, open source tool doesn’t work, as accessing encrypted backups is not supported by it yet (in v1.1 at the time of writing).
    I’ve only found commercial tools that can deal with encrypted backups.
    The tool I used successfully (and hence purchased for roughly 40-50 bucks) is iMazing (v2 at the time of writing) by DigiDNA. I chose this tool as it seemed to be the most professional and most trustworthy one of the many iPhone recovery tools I found online. It’s by a Swiss company; up-to-date (v2.7.2 was released on Oct 3, 2018); the binary files are properly signed; the web, GitHub, social media pages I visited looked professional and didn’t hint a scam.
    Note: This is in harsh contrast to many shady data recovery tools out there, some of them obviously being scams,  not working correctly, being trojans, crypto lockers, data sniffers or similar.
    Either way: You use any 3rd party tool at your own risk! Mind that you’re basically giving the tool full access to both your Mac and iPhone!

    1. Download the iMazing demo app to your Mac
    2. As a precautionary measure, you may want to disable wifi networking and disconnect any Ethernet network cable (like that you could at least prevent live data sniffing and sending by such a tool -> not delayed data sniffing and sending though)
    3. Install the iMazing demo app
    4. If you run macOS 10.14 (Mojave) or later, give iMazing.app full disk access:
      1. In System Preferences -> Security & Privacy -> Privacy -> Full Disk Access click the lock icon to make changes, then click the “+” button and add ‘iMazing.app’ to the list of apps with full disk access (this is required as access to the iTunes ‘Backup’ folder is restricted in macOS 10.14 and later)
    5. Run the iMazing demo app and extract the voice memos of the pre-iOS-12 iTunes backup of your iPhone:
      1. Run the iMazing demo app
      2. Click “later” when prompted to buy a license (we want to test whether the tool works before making a purchase, right?)
      3. In the list on the left, instead of accessing the iPhone directly, select the iTunes backup we previously restored from the Time Machine backup
      4. When prompted by iMazing, enter the encryption password to let the app decrypt the encrypted backup
      5. Go to the “Voice Memos” app icon in the list and select all the listed voice memos you’d like to extract from the backup.
      6. If you want to extract 3 voice memos only, you can do this with the demo app. If you want to extract more than 3 voice memos, you need to purchase and unlock/activate the full version of iMazing, which I did. If you try to extract more than 3 voice memos, a handy assistant will be displayed to guide you through purchasing and activating the full version of iMazing, which is pretty straightforward.
      7. Extract the voice memos to a local folder. Voilà, here you have your dear voice memos again (as .m4a files, playable with e.g. the Quick Time Player or VLC)! Not in the voice memo app on your iPhone anymore, but that’s usually less of a concern anyway (I for one won’t trust the new, apparently rushed voice memo app and will always export future voice memos to an external medium right after recording one).
  5. If you don’t need anything else from the old iPhone backup in iTunes (some other users experienced also other data loss when upgrading to iOS 12, so check if that applies to you too), you can now restore the latest iPhone backup in iTunes from the Time Machine snapshot we created in step 1.
  6. If you don’t need iMazing anymore you can remove its “full disk access” privilege and uninstall iMazing (e.g. using any of the uninstaller apps in the Apple Store or AppCleaner). I also removed the “iMazing.Versions” folder in ~/Library/Application\ Support/MobileSync/Backup that iMazing created for its own purpose.
  7. To lower the risk of losing voice memos and other data on your iPhone/iPad in the future, consider the following tips:
    1.  Create iTunes (or iCloud) backups of your iPhone/iPad regularly. If you use iTunes to back up your iPhone/iPad, make sure to also create regular Time Machine backups of your Mac holding your iTunes backups. It’s best to automate both tasks by ticking the according checkboxes in iTunes and Time Machine.
    2. In the settings of the “voice memos” app on your iPhone/iPad, tell the app to “never” remove user-deleted voice memos (instead of removing them after “30 days”, which seems to be the somewhat unfortunate default in the new voice memo app). AFAIK, this only applies to already manually deleted voice memos though.
      BTW, I’d also recommend disabling the location-dependent naming of voice memos as this is a pretty silly feature for most users.
    3. Consider disabling “automatic updates” in your iPhone/iPad settings under “General” -> “Software updates”. Like that you can wait a couple of days or weeks before installing newly released iOS versions and check the feedback of other users on social media like Twitter first, and maybe catch potential big glitches like that. Actually, Apple’s beta/developer release staging mechanism is supposed to catch those glitches before they can reach the general public in an official release, but apparently, that mechanism hasn’t worked as well yet as it should have.
      Note though that on the other hand, by not installing new releases automatically, you’ll potentially expose your iPhone/iPad to additional security risks due to a bigger time window with missing security patches. So, this advice is a two-edged sword. Decide for yourself!

Interesting observations:

  • I was a bit astonished to see that iTunes itself apparently doesn’t create incremental backups, but only saves the very latest state/snapshot of any device. It thus apparently fully relies on Time Machine’s incremental backup feature if you want to access earlier backups than just the latest snapshot, i.e. you need additional Time Machine backups of your iTunes backups to accomplish this. It looks like iTunes also doesn’t use macOS versioning or APFS’s snapshot feature.
  • Note that iMazing creates a new folder named “iMazing.Versions” in ~/Library/Application\ Support/MobileSync/Backup for its own purpose, which is astonishing and slightly annoying.

Finally, here’s what Apple support suggested doing, instead of the above method:

  1. Backup your current iPhone to iCloud. If you don’t have enough iCloud storage to store the content of your iPhone, purchase a suitable amount of storage before.
  2. Reset your iPhone, deleting all content
  3. Restore your phone from an old (pre-iOS-12) backup in iTunes that still contains your voice memos (note: they didn’t mention that one has to restore that backup from Time Machine first, in most cases)
  4. When starting your iPhone, connect it to iCloud. In particular, let the voice memo app synchronise with iCloud.
  5. Switch off and switch on again the iCloud synchronisation in the settings of the voice memo app in order to make it synchronise the old voice memos to iCloud
  6. Restore the backup from iCloud

According to the Apple support, this should intelligently merge old and new data, so that you end up with iOS 12 and all the new and old data on it, without losing any, including the old, previously vanished voice memos.

If you don’t mind the potential risk of a failed data merge and don’t object backing up your iPhone to iCloud, you could alternatively try this.

Either way, I hope these tips are helpful. Good luck!

 

 

 

 

GitLab 10.5 and later: Solution for error “Validation failed for domain” with Let’s Encrypt

GitLab 10.5 introduced built-in support for Let’s Encrypt.

Unfortunately, if you follow the official GitLab instructions how to enable Let’s Encrypt support, you may encounter the following error when rebuilding GitLab:

Running handlers:
There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[yourhost.yourdomain.com] (letsencrypt::http_authorization line 3) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) had an error: RuntimeError: [yourhost.yourdomain.com] Validation failed for domain yourhost.yourdomain.com

Running handlers complete
Chef Client failed. 11 resources updated in 11 seconds

Warnings:

Let’s Encrypt is enabled, but external_url is using http

The last line is rather misleading, as the domain validation can apparently also fail if one sets  external_url = “https://yourhost.yourdomain.com”

As a workaround, add the following two additional lines to /etc/gitlab/gitlab.rb (hat tip to Kai Mindermann and Thomas Jost for the hints):

nginx['redirect_http_to_https_port'] = 80
nginx['redirect_http_to_https'] = true

 

So, all in all, you need to set in /etc/gitlab/gitlab.rb:

external_url 'https://yourhost.yourdomain.com'

and add the following lines (adjust the notification e-mail address):

letsencrypt['enable'] = true
letsencrypt['contact_emails'] = ['gitlab-notifications@yourdomain.com'] # optional
nginx['redirect_http_to_https_port'] = 80
nginx['redirect_http_to_https'] = true

Make sure that your firewall doesn’t block access to ports 22 (SSH), 80 (HTTP), 443 (HTTPS).

After that, reconfigure GitLab (in a shell):

# gitlab-ctl reconfigure

That’s it! You can now register/login at https://yourhost.yourdomain.com.

The ‘All-in-One WP Migration’ plugin is all you need to migrate your WordPress blog

This blog has just been migrated to a newer and (much) faster host node running Proxmox 5 with ZFS.

Therefore, I was looking for the best method to migrate a WordPress blog to another server (and/or database and/or directory and/or URL).

Do you remember the times when migrating a WordPress blog was rather tedious, involving many manual steps, despite (other) handy tools like WP-CLI? Apparently, this is no longer needed, as all you need is the following plugin:

All-in-One WP Migration

This plugin’s export and import functionality takes care of all the required configuration and path adjustments, allowing you to easily migrate a blog with up to 512 MB data. The steps are thus:

  1. Install the All-in-One WP Migration plugin on your current WordPress site
  2. Use the plugin to export all your data, plugins, themes, configuration etc., e.g. as a downloadable file.
    Note: In particularly tricky cases you can also manually replace certain strings in the db or exclude specific data and files.
  3. Setup a new vanilla WordPress installation at another location (server, directory). You’ll need a database and the WP installation files for this.
  4. Install the All-in-One WP Migration plugin on your new WordPress site
  5. Use the plugin on your new site to import the previously exported data from your old site

It’s hard to believe, but that’s really it!

Solution for: Proxmox backup error due to iothread=1

If you see the following error when trying to backup a KVM VM image on Proxmox:

ERROR: Backup of VM 100 failed – disk ‘scsi0’ ‘zfsvols:vm-100-disk-1’ (iothread=on) can’t use backup feature currently. Please set backup=no for this drive at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 77. INFO: Backup job finished with errors TASK ERROR: job errors

edit /etc/pve/qemu-server/100.conf, look for a line similar to

scsi0: zfsvols:vm-100-disk-1,iothread=1,size=70G

and change it to

scsi0: zfsvols:vm-100-disk-1,iothread=0,size=70G

Afterwards, it’s possible to backup the VM.

How to check filesystems in a qcow2 image

A useful post how to fsck (check and fix) a filesystem in a qcow2 image (as typically used for KVM VMs, e.g. in Proxmox):

How to recover a qcow2 file using fsck

On Proxmox or Debian, one does the following:

Attention:

  • Make sure the according VM isn’t running, i.e. the partition not mounted
  • Adjust the commands below to match your system, use the correct qcow2 image, use the correct fsck-variant, fsck the correct filesystem, note that -p tries to automatically fix errors!
# modprobe nbd max_part=8
# qemu-nbd --connect=/dev/nbd0 /var/lib/vz/images/100/vm-100-disk-1.qcow2
# fdisk -l /dev/ndb0
/dev/nbd0p1            2048     7813119     3905536   82  Linux swap / Solaris
/dev/nbd0p2   *     7813120   119537663    55862272   83  Linux
# fsck.ext4 /dev/nbd0p2
# fsck.ext4 -p /dev/nbd0p2
# qemu-nbd --disconnect /dev/nbd0

Like this, one doesn’t need to boot the VM using a boot ISO/CDROM and can fix the filesystem right from the host node.

 

Combining the Eisenhower Matrix and Kanban on Trello

I like the Eisenhower Matrix a lot: In everyday life, the simple concepts often tend to be the most practical ones, and in the end, usually also the most useful and powerful ones. Remember: “Everything should be made as simple as possible, but not simpler” (although that’s also a reminder to not over-simplify things).

Here’s an idea how to combine the concept of 2-dimensional triage (Eisenhower Matrix) with the idea of JIT workflows (JIT processes from supply chain to production to delivery) in Kanban / Kanban in software development, using Trello:

eisenhower_matrix_combined_with_kanban_on_trello

The top Kanban workflow would thus be:

1. INBOX: If urgent & important: DO!

2. If urgent & not important: DELEGATE!

3. If not urgent & important: DECIDE+PLAN!

4. Rest: Not urgent & not important: ELIMINATE!

(& := logical AND)

This would guarantee, that first, all incoming tasks/requests (e-mails, phone calls, mail, visitors, time-triggered events, other events, ..) would get proper (highest) attention and get collected in the INBOX list. They would then be triaged (qualified) and, if appropriate, distributed/moved to the other lists accordingly. If nothing else matches, they would end up in the low priority box (not urgent AND not important), i.e. the equivalent of a “trash”.

In this order, it’s a failsafe workflow, where all incoming tasks are treated as highest priority tasks at first by default. In some seldomn special cases, usually just temporarily, one could also imagine that reversing the workflow would make more sense (e.g. in a situation of a foreseeable flood of low-prio incoming requests, when you only have time to pick the high-priority ones out of these and leave almost all tasks in the “trash” by default -> this is not failsafe, of course!)

Integrating a particular product’s Kanban workflow would require adding further “sub-lists” for each of the steps of the product lifecycle. Or one could also think of using a separate board for each of the 4 states of the Eisenhower matrix, with lists representing the Kanban states/stages connected through Kanban JIT processes.

Note that each team member can (and should) have her/his own Eisenhower sub-matrix in addition, also combined with her/his own Kanban sub-workflow.


Side note: One could also extend the above board and make it more fine-grained by subdividing each list into the following 5 GTD (Getting Things Done) sub-workflows:

capture, clarify, organize, reflect, and engage

These would ideally be represented by sub-lists, which don’t exist in Trello (yet?). Currently, one would thus have to improvise a bit and e.g. use cards for them, using comments as “sub-cards” – or add further lists, e.g. “1.1 capture”, “1.2 clarify”, “1.3 organize”, “1.4 reflect”, “1.5. engage”.

One could also treat the above combination of the Eisenhower Matrix and Kanban as an application of the 5 GTD workflows and do without making those explicit. At the moment, I favor this view and, for simplicity, don’t recommend visualizing these GTD “meta-workflows”.


 

Disclaimer: The ideas proposed here are “work in progress”, likewise this post. This is just one example of many imaginable ones. I’ll probably edit this post or clarify, correct or extend it in follow-up posts.

#32c3 presentations to watch (note to self)

Overview of recorded presentations:

https://media.ccc.de/b/congress/2015

My list of particularly interesting presentations (mostly for myself – disclaimer: I haven’t watched all of these presentations yet as I didn’t make it to Hamburg this year):

zpool: Symbol `spa_feature_table’ has different size in shared object, consider re-linking

If you see the following error message when executing ‘zpool status’ after “upgrading” Proxmox to the (currently) latest version with ZoL 0.6.4 instead of 0.6.3:

zpool: Symbol `spa_feature_table' has different size in shared object, consider re-linking

Explanation:

“Re-linking” doesn’t have a special meaning in ZFS/ZoL terminology, it rather just describes the usual linking step after compiling some source code.

Solution:

The most likely cause is that some new, updated and required packages have not been installed (e.g. with ‘uname -a’, you’ll see that the old kernel is still running, even after rebooting). Usually, this happens if one executes “apt-get update && apt-get upgrade” (as usual) rather than “apt-get update && apt-get dist-upgrade“. So, in order to solve this problem, do:

# apt-get update && apt-get dist-upgrade
# reboot

In this case, ZoL 0.6.4 also includes a couple of new features which can be enabled for all local ZFS pools by executing:

# zpool upgrade -a
This system supports ZFS pool feature flags.

Enabled the following features on 'rpool':
 spacemap_histogram
 enabled_txg
 hole_birth
 extensible_dataset
 embedded_data
 bookmarks

Proxmox VE 3.4 and ZFS: How to create an ISO to use lz4 compression by default

EDIT 20150607: Meanwhile, this fix has been included by Proxmox – the current official Proxmox VE 3.4 Installer ISO finally enables lz4 compression by default at installation time. I thus recommend using an official Proxmox VE ISO image instead of creating an ISO image yourself, although creating your own ISO might still be interesting from an educational point of view or to include fixes that haven’t made it into the official Proxmox ISO yet.

Proxmox VE is a truly great, Debian based Linux distribution to host both KVM virtual machines and OpenVZ containers, using a RedHat based kernel (numlock.ch runs as a KVM guest on Proxmox VE, BTW ;).

With the recent release of Proxmox VE 3.4, the whole package got even better, now also allowing to install Proxmox on top of ZFS (which is the best enterprise file system available to date), specifically its native port for Linux, ZFS on Linux (ZoL).

Unfortunately, Proxmox VE 3.4 doesn’t allow to use lz4 as a compression algorithm at installation time yet: Using lz4 would be strongly recommended however instead of using ZoL’s default lzjb (or much worse: gzip). There are some known workarounds for this, but those are rather tedious, either involving extensive file copying or a manual setup of the ZFS pool and file systems.

The most elegant solution at the moment is creating a bootable ISO image (based on the original bootable Proxmox VE 3.4 ISO image) with a patched /usr/bin/proxinstall script that uses “compression=lz4” instead of “compression=on” by default.

And here’s how to create the patched ISO:

1. Loop-mount the original ISO (can only be mounted read-only):

# mount -o loop /path/to/proxmox-ve_3.4-3f2d890e-1.iso /mnt/cdrom

2. Copy /usr/bin/proxinstall from the mounted ISO image to a read-writeable directory (create directories as necessary):

# cp -a /mnt/cdrom/usr/bin/proxinstall /mnt/cdrom_patched/usr/bin/proxinstall

3. Edit the /mnt/cdrom_patched/usr/bin/proxinstall script as follows:

# diff -u ../cdrom/usr/bin/proxinstall usr/bin/proxinstall 
--- ../cdrom/usr/bin/proxinstall	2015-02-12 17:52:50.000000000 +0100
+++ usr/bin/proxinstall	2015-03-17 21:50:07.662031284 +0100
@@ -592,7 +592,7 @@
     # disable atime during insatll
     syscmd ("zfs set atime=off $zfspoolname") == 0 ||
 	die "unable to set zfs properties\n";
-    syscmd ("zfs set compression=on $zfspoolname") == 0 ||
+    syscmd ("zfs set compression=lz4 $zfspoolname") == 0 ||
 	die "unable to set zfs properties\n";
 }

4. Use the complicated but very powerful tool xorriso (install it, if necessary) to create a bootable ISO image based on the original ISO, but with /usr/bin/proxinstall “overwritten” with the edited proxinstall script:

# xorriso -boot_image grub patch -indev proxmox-ve_3.4-3f2d890e-1.iso -overwrite on -outdev proxmox-ve_3.4-3f2d890e-1-with-lz4-patch.iso -blank as_needed -pathspecs on -add /usr/bin/proxinstall=/mnt/cdrom_patched/usr/bin/proxinstall -- -commit

5. Use this patched ISO to install Proxmox VE 3.4 as usual

After rebooting the installed Proxmox VE 3.4 host/server, use the following command to verify that lz4 has been used by default:

# zfs get compression
NAME              PROPERTY     VALUE     SOURCE
rpool             compression  lz4       local
rpool/ROOT        compression  lz4       inherited from rpool
rpool/ROOT/pve-1  compression  lz4       inherited from rpool
rpool/swap        compression  lz4       inherited from rpool


For your convenience, here’s a patched ISO of Proxmox VE 3.4 for download (Note: Use it at your own risk!)

proxmox-ve_3.4-3f2d890e-1-with-lz4-patch.iso_.gz (571 MB, md5sum: 2abba5445133c011aadb1808237202b0)

Download it, gunzip it and rename it to proxmox-ve_3.4-3f2d890e-1-with-lz4-patch.iso to get rid of the silly underscore (“_”) WordPress appended when I uploaded the file. The resulting ISO image will be 725 MB, md5sum: e09e5d250d16fa182129c72be88a5aa2.

EDIT 20150607: Meanwhile, this fix has been included by Proxmox – the current official Proxmox VE 3.4 Installer ISO finally enables lz4 compression by default at installation time. I thus recommend using an official Proxmox VE ISO image instead of creating an ISO yourself.

Have fun!