You know how when someone accidentally deletes their files or their hard drive crashes or some other apocalyptic event occurs, the first thing people ask is "where is your backup"? Of course, we've all seen it (*ahem* been there :/ ). It's a bit unintuitive, because backups have no equivalent in the real world. If you drive your car into a lake, there's no way to get it back. But making backups is the single best way to prevent losing your stuff. So do it!

Don't backup "my files"

But don't just backup "my documents, my pictures, my whatever". If you computer crashes and you have a backup of "my files", then sure, it's not a total loss. It's better than nothing. But it's not what you actually need. You need the whole thing

This "my files" nonsense is born out of the fact that the delightful company that produced your operating system doesn't want you to be able to make a backup of it. Because if you did, you could make trivial copies of the operating system, and they don't like that idea. Have you ever asked yourself why in 30 years, through all manner of viruses, blue screens of death and hardware crashes Microsoft has never given you or sold you a full system backup program? It's not because they never thought of it. (Or because no one asked for it).

Making full backups is easy

If you've ever installed Gentoo manually (ie. not with one of the automated installers)... yes, the demographic for this one is not immense. But then that's why we're here, to spread the happy message! :happy: Anyway, if you have, then you know immensely easy (this is astonishing especially if you have a Windows background) it is to make a full system backup. In the course of a Gentoo install (and yes, I'm about to reveal the big secret here...*drumroll*), you boot from the livecd, you mount your root partition, you download a tar of a minimal Gentoo filesystem that has your basic /bin/ls and so on, and then you just.... untar it. That's it. No magic, no voodoo, no secret foobared locations on the filesystem that can't be written to, just extract the archive and you're done!

To put it bluntly, this is all you have to do:

tar zcvf backup.tar.gz /

And to restore the backup:

tar zxpvf backup.tar.gz -C /mnt/root_partition

Put that in perspective to the Windows world where a whole industry has sprung up to solve problems that Microsoft deliberately introduced with their *ahem* novel engineering. Idiotic programs like Norton Ghost that you have to get your hands on just to do the same simple thing that you can do with tar on a decent operating system.

Making it more convenient

Granted, you could just use the above tar command, but you may want something a little more convenient. For starters, you may want to skip some files on your file system. The method I use is inspired by a script posted on the gentoo forums a long time ago. I used that script for years without really understanding it, but a while back I decided to rewrite it to suit me better.

Besides just tarring the files it also writes a log file that you can grep to see if some particular file of interest is in the backup, it timestamps the backup with the current date/time and it keeps track of how many backups you want to keep.

Backups are made in a special backup_dir location. This directory is supposed to hold lists of files (recipes, if you like) you want to backup. For example, a simple recipe could be called full.lst:

/
--exclude=/backup/*.tgz*
--exclude=/proc/*
--exclude=/sys/*
--exclude=/tmp/*
--one-file-system

The syntax for the file is that of tar, and it's a list of things to backup. / means the full file system will be included. But certain directories are excluded, /backup because we don't want to include our old backup files in new backups, /proc and /sys because they are virtual file systems and don't contain "real" files, and we don't care about /tmp. Finally, we say --one-file-system, which prevents mounted disks, cds and things like that to be included.

And here is the script that makes this possible. Run it, it will produce a backup file that is compressed. Try to get it below 4.3gb and write it on a dvd+rw, now you have a backup system. :party:

#!/bin/bash
#
# Author: Martin Matusiak <numerodix@gmail.com>
# Licensed under the GNU Public License, version 3.

backup_dir=/backup
num_backups=1


verbose="$@"
lists=$backup_dir/*.lst
ext=tgz
date_params="%Y-%m-%d-%H%M"
nice_val="nice -n20"

# colors
wh="\e[1;37m"
pl="\e[m"
ye="\e[1;33m"
cy="\e[1;36m"
re="\e[1;31m"

if [[ "$verbose" && "$verbose" != "-v" ]]; then
	echo "Usage:  $0 [-v]"
	exit 1
fi

if [ ! -d $backup_dir ]; then
	echo -e "${re}Backup dir $backup_dir does not exist.${pl}"; exit 1
fi


for list in $(ls $lists); do
	name=$(basename $list .lst)
	file_root=$backup_dir/$name.$(date +$date_params)
	
	stdout="1> /dev/null"
	stderr="2> $file_root.$ext.err"
	if [ "$verbose" ]; then
		stdout=""
	fi

	cmd="cat $list | $nice_val xargs tar zlcfv \
		$file_root.$ext $stderr | tee $file_root.$ext.log $stdout"

	trap 'echo -e "${re}Received exit signal${pl}"; exit 1' INT TERM

	echo " * Running \`$name\` job..."
	if [ "$verbose" ]; then echo -e ${ye}$cmd${pl}; fi
	echo -en $cy; bash -c "$cmd" ; echo -en $pl
	status_code=$?

	if [ $status_code -gt 0 ]; then
		# Dump error log
		echo -en $re ; cat $file_root.$ext.err
		echo -en $pl ; echo "Tar exit code: $status_code"
	else
		# Kill error file
		rm $file_root.$ext.err
	fi

	# Evict old backups we don't want to keep
	num=$num_backups
	for evict in $(ls -t $backup_dir/$name.*.$ext); do
		if [ $num -le 0 ]; then 
			rm -f "$evict"
		else
			num=$(($num-1))
		fi
	done

	# Report number of files in backup
	echo -n "$(wc -l < $file_root.$ext.log) files"
	echo ", $(ls -l $file_root.$ext | awk '{ print $5 }') bytes"

done

Worse is better

I've been thinking about how to handle backups most effectively, and it occurs to me that backups are a case of "worse is better". The thing is you could make a really nice and easy application to make backups, but .tar.gz is still the optimal format to store them in. Wherever you are, you have tar and gzip available, and restoring backups usually happens under somewhat constricted conditions, sometimes without network access. So you want to avoid introducing dependencies, it's safer to make do with the tools that are there already.

So it may not be the most elegant system, but it's damn reliable.

Limitations (NEW)

Basically what I'm saying is that if you have no backup system then using tar is a pretty decent system. At the very least it has worked well for me the last 5 years. That isn't to say you shouldn't use a different method if you have different needs.

What about scaling? Well, I think this works quite well up to backups of say 4gb or so. My root partition is using 12gb of space at the moment. The purpose of this method is to back up your working system with all the configuration, applications and so on. Not to back up your mp3 collection, I would exclude that (not least because it's pointless to compress mp3 files and other formats that already are well compressed).

What about the bootloader? (NEW)

Some people have asked how this backup method concerns the bootloader. The answer is that it does backup the files that belong to the bootloader (in /boot). It does not backup the actual boot sector of your hard drive (which isn't represented as a file). So if, for example, you want to restore the backup on another computer (which I've done lots of times), you'll still need to use grub/lilo to update the boot sector.

UPDATE: Apologies to the indignant Windows users. I pretty much wrote this in rant mode, without doing any research and what I wrote about Windows is just from my own experience. I would have been more thorough if I had known this would make the frontpage of digg and draw so many hits.

Hah, Google. What idiots! First they announce their creed is "don't be evil" and now they have to live up to it, because every little report to the contrary has the tubes flowing with indignation. How dumb was that?

One marketing company knows better. They would never fall into a trap like that. Appluz propaganda-fu is so strong that they copied Microsoft's business tactics and yet people still like them. Now that's evil you can't buy!

Business people love the Apple example. Love it to death. How much you wish you had come up with it! Sell overpriced junk to drones with an identity crisis who desperately want to be "special" and get rich doing it. Lex Luthor couldn't have done it better.

Apple have achieved the exact opposite of what Microsoft always wanted to. Beloved Redmond with their upgrade cycle from hell have done well for themselves. But iSpecial people run to the store to buy the latest junk voluntarily, in fact they run like it's the 100m sprint in the Olympics. As special as we all are, there is no doubt the muppets with the latest white plastic are the most special of all.

As with any empire, there are cracks in the concrete here and there. The people on fuckingnda.com have something to say about the experience of writing code for a device without even being able to talk to each other. This same week Apple is facing a class action suit for failing to pay overtime and imposing evil working hours on their employees (sounds like a page from EA Games's book of evil). And least surprising of all, a talk on Apple security at Black Hat got canceled by the marketing department (if there ever was any doubt that the marketing department runs the company).

Yes, the Jobs empire of iDRM is showing healthy levels of evil indeed.

I'm extremely pleased that there are so many talks published online these days. They compose a rich buffet of interesting topics for anyone who can reach google video/youtube. I see it as a big step forward in our global communication that you can not only read what people have to say, but also hear it directly from them. It also raises the value of all these conferences that organize the talks dramatically. Not only their ability of attracting speakers the following year, but also their function in our society as "broadcasters" of knowledge.

Unfortunately, some conferences, even big ones, don't publish the talks. Instead, they sometimes publish the slides. That's too bad, but they have their reasons I suppose. I guess people consider this a second best, you can't see the talk but well you have the slides, so you have the "content". Well, that depends on the slides. I've always been of the opinion that the slides should be able to speak for themselves in case someone didn't see the talk. But speakers have different styles.

OSCON 2008 only published slides, let's see some examples. (Heads up, that page is "sponsored by Microsoft", whatever that means. Perhaps they wrote the html.)

Some use them as a "visual aid" for their talk, and consider the whole thing as more of a performance art. Now, I'm not down on talks like this, because the talks are sometimes quite good. It's just that the slides are fairly worthless on their own. The Mac/Rails crowd seems to be all over this minimalistic slides idea: ruby web frameworks, capistrano. There's very little content in these slides, some of them are just single words on a page. A particularly egregious example is a talk on revision control. What is the point of publishing slides like this?

One guy who both gives good talks and writes good slides is Alex Martelli. Compare his slides on code reviews. Granted, if you don't understand what they mean, there isn't much to do about it, but I think this is pretty much what you can hope to achieve with slides alone. They offer real value even without commentary.

Now, there is a real dilemma about how to write slides effectively. I have agonized over this for a while and I don't have an answer yet. And I'm guilty of writing bad slides. I gave a talk a while back which was quite well received, but the slides are not good: dynamic python. The content is there, but there is no context to it.

Well well, what have we here? The omelet is well underway in the pan, but we've dropped some eggs on the floor, that's too bad. KDE likes to try things, and that's really cool. I'm pleased as long as they do not stop trying until they produce a better, or at least an equally good, outcome. (Which is my experience with KDE so far.)

There is another slight regression here. The click area for the plus (+) sign beside each directory (the one that expands/collapses it) appears smaller now, after the icon has been changed. It's smaller now, and it was already quite small before.

Ubuntu Bug: #254039

KDE Bug: 168379

I'm not submitting it specifically to kde bugzilla given how Shuttlesworth raves about Launchpad's synchronization capabilities to various bugzillas every chance he gets. Hopefully that means they have a link set up with KDE as well. The regression was spotted in Ubuntu anyway.