Johan Ekenberg

Computer Programmer and Double Bass Player

Mosh+luit: Mobile Shell on ISO-8859-1

Mosh + luit solves the UTF-8 / ISO-8859-1 problem.

Mosh (mobile shell) is the perfect choice for roaming ssh sessions. Login at home, work on the train, wake the laptop from sleep when arriving at work and find the ssh session still active. It feels almost like magic. I use it all the time. There is however one snag: Mosh requires UTF-8 on both client and server. For modern clients, this is not a problem, but servers can be another story. I regularly work with legacy servers still on ISO-8859-1. Accessing them with UTF-8 is awkward: we Swedes need our beloved ‘åäö’, and UTF-8 <-> ISO-8859-1 will mess up non-ascii characters.

Enter luit. Luit runs as a filter between an application and the terminal, translating between UTF-8 and the application encoding. That means luit can transparently filter the entire terminal session if setup as a filter for the login shell. On the server, I put this as the last line of my .bash_profile (adjust locale environment variables according to your needs):

Last line of .bash_profile
1
LC_ALL='' LC_CTYPE=sv_SE LANG='' exec luit

Warning: Don’t put it in .bashrc, and make sure your .bashrc and .bash_profile are not symlinked to the same file! exec luit starts an interactive shell which invokes .bashrc which does exec luit which starts an interactive shell… Your login will never complete, and the server will fill up with thousands of luit processes, an effective fork bomb.

Luit might already be installed on your server. If not, it should be available through your packet manager. On any Debian derived system (Ubuntu etc): sudo apt-get install x11-utils

Pbtrim - Ltrim Your Pasteboard

I spend a fair amount of time on Stack Overflow, following interesting topics and answering the occasional question. Writing answers on Stack Overflow usually means writing and testing code, using Emacs and Mac Terminal. When finished, copy-paste code into the form on Stack Overflow and apply formatting using Markdown.

Since I normally use Emacs in terminal mode (emacs -nw), I have problems with the copy-paste: Marking lines with the mouse and copying them puts lots of spaces at the end of each line. It’s a well known problem with screened terminal applications (using ncurses or similar), copying gets you the entire physical line on screen, whitespace and all.

What I needed was a way to ltrim() each line in the pasteboard (Mac:ish for clipboard) after copying. It turns out this was easy with the help of a few utilities:

  • The Mac OSX commands pbcopy and pbpaste handle copy and paste from the commandline.
  • BetterTouchTool is a great (free) tool for custom keyboard shortcuts and gestures.
  • A small bash script to do the actual trimming, I call it pbtrim
pbtrim
1
2
3
4
5
6
    #!/bin/bash

    # pbpaste: send pasteboard contents to sed
    # sed: remove all terminating whitespace
    # pbcopy: copy result back to pasteboard
    pbpaste | sed  's/ *$//' | pbcopy

Save pbtrim somewhere on your $PATH and make it executable.

Open BetterTouchTool. Create a new keyboard shortcut for application Terminal. I use Cmd-Shift-C. Make it Trigger Other Keyboard Shortcut Cmd-C. Now you have an alternate shortcut for the copy command. Next Attach Additional Action, choose Controlling Other Applications then Execute Terminal Command: pbtrim. Done. Cmd-Shift-C will first trigger a normal copy, then run pbtrim to remove terminating whitespace. Ready to paste into Stackoverflow!

Winter Is Here

These guys are having a snow war!

See the other one up the hill?

You can’t hit me, I’m protected!

Linux Time Machine With Rsync

Edit: The script in this article has been superseeded by the Linux Time Machine project on GitHub

Time Machine is Apple’s backup solution for Mac computers. It makes incremental backups and provides a gui (the star field) for browsing and restoring data from arbitrary points in time. I use it for my Macs and it is nice, although the star field is limited since it provides no terminal access.

Ok, but my important computers run Linux and I want something similar for them. No need for the gui, but incremental backups that are easy to access. The magic bullet is to use rsync with the --link-dest= option. This makes rsync use hard links for every file that is unchanged between backup versions. A hard link takes virtually no space, so the backup will not use more space than necessary for all files + whatever changes occur between versions.

Here’s how I did it, together with a small script to get you started.

You need

  • rsync. It is probably already installed on your Linux machine. Else, install it through your package manager.
  • Somewhere to put the backups. I use a NAS from QNAP, but it could be any kind of external storage. Please note that the target filesystem must support hard links. Microsoft FAT-32 does not.

Configuring the script

My backup storage is mounted over NFS:

1
2
$ grep qnap-backup /etc/fstab
qnap:/backup  /mnt/qnap-backup nfs auto,user 0 0

so $BACKUP_MOUNTPOINT will be /mnt/qnap-backup. You could also use ssh to backup over the network. After setting up ssh-keys for password-less connection, $BACKUP_MOUNTPOINT will be something like backup-hostname:/path/to/backup

$BACKUP_EXCLUDE is the path to a file specifying what to exclude from the backup. Basically it contains exclude-patterns, one per line. See man rsync for details. Mine contains entries like:

/home/johan/Videos/*
/tmp/*

Adjust $BACKUP_MOUNTPOINT and $BACKUP_EXCLUDE. Schedule it to run (as root) periodically. I run my every night at 3 am. This is what it looked like after a few nights (piano is my hostname):

piano:/mnt/qnap-backup$ ls -1
piano-backup-2013-02-20
piano-backup-2013-02-21
piano-backup-2013-02-22
piano-backup-2013-02-23
piano-backup-2013-02-24
piano-backup-2013-02-25
piano-backup-2013-02-26
piano-backup-current

The last entry is a symlink to the most current backup. This is important, since every new backup is compared to the current one, and only new or changed files are actually stored on the filesystem. Everything else is hard-linked, meaning that it takes virtually no extra space.

do_incremental_backup.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#!/bin/sh

BACKUP_MOUNTPOINT="/mnt/qnap-backup"
BACKUP_NAME="`hostname`-backup"
BACKUP_EXCLUDE="~/local/etc/backup_exclude.conf"

current_user=`id -u`

if [ "$current_user" != "0" ]; then
    echo "You need to be root (sudo) to run system backups"
    exit 1
fi

VERBOSE_ARGS=""
if [ "x$1" = "x-v" ]; then
    VERBOSE_ARGS="--progress -v"
fi

TODAY=`date +"%Y-%m-%d"`

CURRENT_BACKUP=$BACKUP_MOUNTPOINT/${BACKUP_NAME}-current
NEW_BACKUP=$BACKUP_MOUNTPOINT/${BACKUP_NAME}-$TODAY

mkdir -p $NEW_BACKUP

if [ ! -d $NEW_BACKUP ]; then
    echo "No such directory: $NEW_BACKUP"
    exit 1
fi

rsync $VERBOSE_ARGS -a --delete --relative --one-file-system \
	--numeric-ids --exclude-from=$BACKUP_EXCLUDE \
	--link-dest=$CURRENT_BACKUP/ / $NEW_BACKUP/

# new symlink to current
[ -h $CURRENT_BACKUP ] && rm -f $CURRENT_BACKUP
ln -s $NEW_BACKUP $CURRENT_BACKUP

Going further

This was just to get you started. The script should probably be extended to do things like:

  • Erase old backups over a certain age.
  • Check that the backup storage is not full.
  • Log success or failure somewhere.
  • Alert you if there is a problem.

Here is an excellent article, explaining the rsync approach in more depth

Book Review: Execute

Book: Execute by Josh Long / Drew Wilson

This is a shallow book, both historically and contextually. Historically because the writers live in “Internet time”, meaning that everything further back than a decade is lost in a prehistoric blur. Contextually, because their message is only applicable to a very limited number of people (male urban internet designers between 20 and 30). Still, if you are in the web-business and want to get more productive, there are a few gems here.

Inspiration: A powerful energy source

After adrenaline for survival, inspiration is the most powerful source of energy a creative person can have. A main theme of the book is the need to stay inspired as much as possible, and how that can be accomplished.

80-100 rule

Always do your work at a 80-100% intensity. If you cannot maintain that level of focus, stop. Switch to another task that inspires you more, or get what ever recreation, rest, food, etc needed to get back to the 80-100 zone.

The ultimate goal is to live inspired, all the time. As for myself, I’d say I spend on average 1% of my days being really inspired. So, a modest increase to 2% would be a 100% gain.

SPV: Smallest Possible Version

Whatever you make, make it the smallest possible version. Spend time removing features, sliming and simplifying. Simplicity is everything. Focus on cutting away anything not strictly necessary. The leaner your product, the narrower your focus, the more likely you will actually deliver.

“It’s the best for ___”

This is SPV put another way. Make it clear what single purpose your product is best for. Don’t invent a slightly improved copy of some general product - invent a lean solution to a singular problem.

Build for yourself

Building stuff for yourself, you will not fail. Building for others is so much harder, and taxing on inspiration.

Planning, stalling or executing?

To keep your self in check, keep asking yourself: Am I planning, stalling or executing? Planning is guessing. Don’t get lost planning.