The tool addresses three different issues:

• Cloning/pulling multiple repos in one step.
• Keeping repo clones in sync across machines.
• Better handling of local tracking branches.

Listing repos

Let’s start with a basic situation. I’ve cloned some of my repos on github:

$ ls -F
galleryforge/  italian-course/  re/  spiderfetch/

Download this code: re_ls_repos.txt

I run re list to scan the current path recursively and discover all the repos that exist:

$ re list
[galleryforge:git]
    origin.url = git@github.com:numerodix/galleryforge.git
[italian-course:git]
    origin.url = git@github.com:numerodix/italian-course.git
[re:git]
    origin.url = git@github.com:numerodix/re.git
[spiderfetch:git]
    origin.url = git@github.com:numerodix/spiderfetch.git
> Run with -u to update .reconfig

Download this code: re_list.txt

It creates a configuration file called .reconfig that contains the output you see there. By default it doesn’t overwrite the config, just shows you the result of the detection action. Pass -u to update it.

This file format is similar to .git/config. Every block is a repo, and :git is a tag saying “this is a git repo”. (By design re is vcs agnostic, but in practice I only ever use git and the only backend right now is for git. It probably smells a lot of git in any case.)

Every line inside a block represents a remote (git terminology). By default there is only one. If you add add a remote in the repo and re-run re list it will detect it. But it will assume that origin is the canonical remote (more on why this matters later).

Pulling repos

Now let’s say I want to pull all those repos to sync them with github. I use (you guessed it) re pull:

$ re pull
> Fetching galleryforge
> Fetching italian-course
> Fetching re
> Fetching spiderfetch
> Merging galleryforge
> Merging italian-course
> Merging re
> Merging spiderfetch
-> Setting up local tracking branch ruby-legacy
-> Setting up local tracking branch sqlite-try
-> Setting up local tracking branch db-subclass
-> Setting up local tracking branch next

Download this code: re_pull.txt

As you can see it does fetching and merging in separate steps. Fetching is where all the network traffic happens, merging is local, which is why I think it’s nice to separate them. (But there are more reasons to avoid git pull.)

What it also does is set up local tracking branches against the canonical remote. The canonical remote is the one listed first in .reconfig. So it doesn’t matter what it’s called, but it’s a good idea to make it origin, because that’s what re list will assume when you use it to update .reconfig after you add/remove repos.

It handles local tracking branches only against one remote, because if both origin and sourceforge have a branch called stable then it’s not clear which one of those the local branch stable is supposed to track. I find this convention quite handy, but your mileage may vary.

If I later remove the branch ruby-legacy from github and run re pull, it’s going to detect that I have a local tracking branch that is pointing at something that doesn’t exist anymore:

$ re pull spiderfetch
> Fetching spiderfetch
> Merging spiderfetch
-> Stale local tracking branch ruby-legacy, remove? [yN]

Download this code: re_pull_stale_local_tracking.txt

Scaling beyond a single machine

Now, re helps you manage multiple repos, but it also helps you keep your repos synced across machines. .reconfig is a kind of spec for what you want your repo-hosting directory to contain, so you can just ship it to a different machine, re pull and it will clone all the repos over there, set up local tracking branches, all the same stuff.

In fact, why not keep .reconfig itself in a repo, which again you can push to a central location and from which you can pull onto all your machines:

$ re list
[.:git]
    origin.url = user@host:~/repohost.git
[galleryforge:git]
    origin.url = git@github.com:numerodix/galleryforge.git
[italian-course:git]
    origin.url = git@github.com:numerodix/italian-course.git
[re:git]
    origin.url = git@github.com:numerodix/re.git
[spiderfetch:git]
    origin.url = git@github.com:numerodix/spiderfetch.git
> Run with -u to update .reconfig

Download this code: re_list_reconfig.txt

It does not manage .gitignore, so you have to do that yourself.

Advanced uses

Those are the basics of re, but the thing to realize is that it doesn’t limit you to a situation like the one we’ve seen in the examples so far, with a single directory that contains repos. You can have repos at any level of depth, you can have .reconfigs at different levels too, and you can then use a single re pull -r to recursively pull absolutely everything in one step.

Get it from github:

• re

Simple substitutions

The simplest use is just a straight search and replace. All the files in the current directory will be tried to see if they match the search string.

$ nametrans.py "apple" "orange"
 * I like apple.jpg    -> I like orange.jpg
 * pineapple.jpg       -> pineorange.jpg
 * The best apples.jpg -> The best oranges.jpg

Download this code: nametrans_applelist

There are also a number of options that simply common tasks. Options can be combined and the order in which they are set does not matter.

Ignore case

Matching against strings with different case is easy.

$ nametrans.py -i "pine" "wood"
 * pineapple.jpg -> woodapple.jpg
 * Pinetree.jpg  -> woodtree.jpg

Download this code: nametrans_ignorecase

Literal

The search string is actually a regular expression. If you use characters that have a special meaning in regular expressions then set the literal option and it will do a standard search and replace. (If you don’t know what regular expressions are, just use this option always and you’ll be fine.)

$ nametrans.py --lit "(1)" "1"
 * funny picture (1).jpg -> funny picture 1.jpg

Download this code: nametrans_literal

Root

If you prefer the spelling “oranje” instead of “orange” you can replace the G with a J. This will also match the extension “.jpg”, however. So in a case like this set the root option to consider only the root of the filename for matching.

$ nametrans.py --root "g" "j"
 * I like orange.jpg    -> I like oranje.jpg
 * pineorange.jpg       -> pineoranje.jpg
 * The best oranges.jpg -> The best oranjes.jpg

Download this code: nametrans_root

Hygienic uses

Short of specific cases of transforms, there are some general options that have to do with maintaining consistency in filenames that can apply to many scenarios.

Neat

The neat option tries to make filenames neater by capitalizing words and removing characters that are typically junk. It also does some simple sanity checks like removing spaces or underscores at the ends of the name.

$ nametrans.py --neat
 * _funny___picture_(1).jpg -> Funny - Picture (1).jpg
 * i like apple.jpg         -> I Like Apple.jpg
 * i like peach.jpg         -> I Like Peach.jpg
 * pineapple.jpg            -> Pineapple.jpg
 * the best apples.jpg      -> The Best Apples.jpg

Download this code: nametrans_neat

Lower

If you prefer lowercase, here is the option for you.

$ nametrans.py --lower
 * Funny - Picture (1).jpg -> funny - picture (1).jpg
 * I Like Apple.jpg        -> i like apple.jpg
 * I Like Peach.JPG        -> i like peach.jpg
 * Pineapple.jpg           -> pineapple.jpg
 * The Best Apples.jpg     -> the best apples.jpg

Download this code: nametrans_lower

If you want the result of neat and then lowercase, just set them both. (If you like underscores instead of spaces, also set --under.)

Non-flat uses

Presuming the files are named consistently you can throw them into separate directories by changing some character into the path separator.

Note: On Windows, the path separator is \ and you may have to write it as “\\\\”.

$ nametrans.py " - " "/"
 * france - nice - seaside.jpg -> france/nice/seaside.jpg
 * italy - rome.jpg            -> italy/rome.jpg

Download this code: nametrans_prefixasdir

The inverse operation is to flatten the entire directory tree so that all the files are put in the current directory. The empty directories are removed.

$ nametrans.py --flatten
 * france/nice/seaside.jpg -> france - nice - seaside.jpg
 * italy/rome.jpg          -> italy - rome.jpg

Download this code: nametrans_flatten

In general, the recursive option will take all files found recursively and make them available for substitutions. It can be combined with other options to do the same thing recursively as would otherwise happen in a single directory.

$ nametrans.py -r --neat
 * france/nice/seaside.jpg -> France/Nice/Seaside.jpg
 * italy/rome.jpg          -> Italy/Rome.jpg

Download this code: nametrans_recursive

In recursive mode the whole path will be matched against. You can make sure the matching only happens against the file part of the path with --files or only the directory part with --dirs.

Special uses

Directory name

Sometimes filenames carry no useful information and serve only to maintain them in a specific order. The typical case is pictures from your camera that have meaningless sequential names, often with gaps in the sequence where you have deleted some pictures that didn’t turn out well. In this case you might want to just use the name of the directory to rename all the files sequentially.

$ nametrans.py -r --dirname
 * rome/DSC00001.jpg -> rome/rome 1.jpg
 * rome/DSC00007.jpg -> rome/rome 2.jpg
 * rome/DSC00037.jpg -> rome/rome 3.jpg
 * rome/DSC00039.jpg -> rome/rome 4.jpg

Download this code: nametrans_dirname

Rename sequentially

Still in the area of sequential names, at times the numbers have either too few leading zeros to be sorted correctly or too many unnecessary zeros. With this option you can specify how many leading zeros you want (and if you don’t say how many, it will find out on its own). Based on an old piece of code that has been integrated.

$ nametrans.py -r --renseq 1:3
 * rome/1.jpg   -> rome/001.jpg
 * rome/7.jpg   -> rome/007.jpg
 * rome/14.jpg  -> rome/014.jpg
 * rome/18.jpg  -> rome/018.jpg
 * rome/123.jpg -> rome/123.jpg

Download this code: nametrans_renseq

The argument required here means field:width, so in a name like:

series14_angle3_shot045.jpg

the number 045 can be shortened to 45 with “3:2″ (third field from the beginning) or “-1:2″ (first field from the end).

Get it from sourceforge:

• nametrans

I finally got around to organizing these types of bits that have no specific place of their own into a new github repository, appropriately named “pybits”. It holds the pretty printer and this rewritten ansicolor module, and it’ll probably grow with the ages.

But to business. Anyone spitting out ansi escapes who has figured out the system knows it’s trivial to make a color chart. So to keep the tradition going, here’s proof that ansicolor is able to enumerate the colors:

Notice that section at the bottom about highlighting colors. As you might be able to deduce by sheer logic, black and white are not great colors for highlighting something in a terminal, because they are typically used respectively as the background and foreground of the term (or vice versa). (The colors of a term can actually be anything, but black and white are the common ones. Ideally, code should detect this at runtime, but I don’t know of a way to check for this. Besides, lots of programs [eg. portage] do make this assumption also.) So the highlighting colors are supposed to be useful for when you want to output a wall of text and mark something in the middle of it, so the user can spot it.

Suppose you are (as I have been in the past) developing a regular expression and you can’t get it right on the first try (yeah, unbelievable, I know). Well, what you do is highlight the string so you can see how the matching worked out:

Regular expressions tend to get hairy (yes way) so it helps to compare their results when you’re trying to unify two half-working variants into one. Adding a second regex will show the matches from both. Where they overlap the styling is bold:

Think of the green highlighting as a layer of paint on the wall. You then paint a layer of yellow on top, but you don’t cover exactly the same area. So where the green wasn’t painted over it’s still green. Where the yellow covered it, the paint is thicker. And where the yellow didn’t overlap the green it’s just plain yellow.

Adding a third regex potentially produces segments highlighted three layers thick, so there the color becomes reverse.

And then bold and reverse.

ansicolor doesn’t support background colors, but that’s a product of my use so far, I’ve never needed it. I don’t think they improve readability.

You will find this cutting edge technology in the repo:

• pybits

So I thought that I would build a recursive iterable that I can give to pprint. Here’s an example:

class Node(object):
    classatt = 'hidden'
    def __init__(self, name):
        self.name = name
 
a, b, c, d = Node('A'), Node('B'), Node('C'), Node('D')
a.refs = [b, d]
b.refs = [c]
c.refs = [a]
d.refs = [c]

Download this code: pretty_printing_object.py

This will give you:

{'__type__': '<Node {id0}>',
 'name': "'A'",
 'refs': [{'__type__': '<Node {id1}>',
           'name': "'B'",
           'refs': [{'__type__': '<Node {id2}>',
                     'name': "'C'",
                     'refs': ['dup <Node {id0}>']}]},
          {'__type__': '<Node {id3}>',
           'name': "'D'",
           'refs': [{'__type__': '<Node {id2}>',
                     'name': "'C'",
                     'refs': ['dup <Node {id0}>']}]}]}

Download this code: pretty_printing_output.py

There are two things being shown here:

• node C is reachable through ABC and ADC.
• A takes part in two cycles: ABCA and ADCA.

It would be nice to have a way to see this from the output. So aside from the object attributes themselves there is also a __type__ attribute which tells you the type that you’re looking at. And it has a marker of the form {id1}, where id1 is an identifier for this object, so that you can see where it pops up in a different part of the graph.

Now, suppose we follow A to B to C and then to A. We are now seeing A for the second time. Instead of printing the object again we print a duplicate marker: dup <Node {id0}>. The identifier is supposed to be vim * friendly, so if you pipe the output to vim, put the cursor over it and hit * (also might want to do set hlsearch) then you’ll see it light up all the other instances of it in the graph.

Well, that’s all for now. It’s definitely not the last word in pretty printing, but it’s useful already.

I thought maybe github’s gists would be appropriate for something like this:

• pp module

Flat structure

Let’s check out one right now:

./frame.py
./master.py
./mystring.py
./page.py
./sentence.py
./user.py

Download this code: python_codebase_structure_flat.txt

And here’s the import relationship between them:

Easy, straightforward. I can execute any one of the files by itself to make sure the syntax is correct or to run an “if __main__” style unit test on it.

Tree structure

But suppose the codebase is expanding and I decide I have to get a bit more structured? I devise a directory structure like this:

./media/book/__init__.py
./media/book/page.py
./media/book/sentence.py
./media/__init__.py
./media/master.py
./media/movie/frame.py
./media/movie/__init__.py
./media/mystring.py
./user.py

Download this code: python_codebase_structure_tree.txt

The same files, but now with __init__.py files all over the codebase to tell python to treat each directory as a package. And now my import statements have to be changed too, let’s see master:

# from:
import mystring
import page
# to:
import media.mystring
import media.book.page

Download this code: python_codebase_structure_master.py

Nice one. Okay, let’s see how this works now:

$ python user.py
page says hello!
sentence says hello!
frame says hello!
mystring says hello!
master says hello!

Download this code: python_codebase_structure_run_user.txt

user imports page and then master. The first 4 lines are due to page, which imports three modules, and finally we see master arriving at the scene. All the files it imports have already been imported, so python doesn’t redo those. Everything is in order.

As you can see, imports between modules in the tree work out just fine, page finds both the local sentence and the distant frame.

But if we run master it’s a different story:

$ python media/master.py
master says hello!
Traceback (most recent call last):
  File "media/master.py", line 3, in <module>
    import media.mystring
ImportError: No module named media.mystring

Download this code: python_codebase_structure_run_master.txt

And it doesn’t actually matter if we run master from media/ or run media/master from ., it’s the same result. And it’s the same story with page, which is deeper in the tree.

These modules, which used to be executable standalone, no longer are.

A hackish solution

So we need something. The nature of the problem is that once we traverse into media/, python no longer can see that there is a package called media, because it’s not found anywhere on sys.path. What if we could tell it?

The problem pops up when the module is being executed directly, in fact when __name__ == ‘__main__‘. So this is the case in which we need to do something differently.

Here’s the idea. We put a file in the root directory of the codebase, a file we can find that marks where the root is. Then, whenever we need to find the root, we traverse up the tree until we find it. The file is called .codebase_root. And for our special when-executed logic, we use a file called __path__ that we import conditionally. Here’s what it looks like:

import os
import sys
 
def find_codebase(mypath, codebase_rootfile):
    root, branch = mypath, 'nonempty'
    while branch:
        if os.path.exists(os.path.join(root, codebase_rootfile)):
            codebase_root = os.path.dirname(root)
            return codebase_root
        root, branch = os.path.split(root)
 
def main(codebase_rootfile):
    thisfile = os.path.abspath(sys.modules[__name__].__file__)
    mypath = os.path.dirname(thisfile)
    codebase_root = find_codebase(mypath, codebase_rootfile)
 
    if codebase_root:
        if codebase_root not in sys.path:
            sys.path.insert(0, codebase_root)
 
codebase_rootfile = '.codebase_root'
main(codebase_rootfile)

Download this code: python_codebase_structure_fix.py

So now, when we find ourselves in a module that’s somewhere inside the media/ package, we have this bit of special handling:

print "master says hello!"
 
if __name__ == '__main__':
    import __path__
import media.mystring
import media.book.page

Download this code: python_codebase_structure_master_new.py

Unfortunately, importing __path__ unconditionally breaks the case where the file is not being executed directly and I haven’t been able to figure out why, so it has to be done like this.

You end up with a tree looking as you can see in the screenshot.

I’ve pushed the example to Github so by all means have a look:

• python-codebase-structure

We pass the test, all the modules are executable standalone again. But I can’t say that it’s awesome to have to do it like this.

I realized I haven’t really explained anything yet, so let’s look at a suitably contrived example.

A network manager

Suppose you’re writing a network manager type of application (I actually tried that once). You might have a class called NetworkIface. And the class has an attribute ip. So how does ip get its value? Well, it can be set statically, or via dhcp. In the latter case there is a method dhcp_request, which requests an ip address and assigns to ip.

# <./main.py>
class NetworkIface(object):
    def __init__(self):
        self.ip = None
 
    def dhcp_request(self):
        self.ip = (10,0,0,131) # XXX magic goes here
 
 
if __name__ == '__main__':
    iface = NetworkIface()
    iface.ip = (10,0,0,1)
    iface.ip = (10,0,0,2)
    iface.dhcp_request()

Download this code: main.py

Now suppose you are in the course of writing this application, and you need to do some debugging. It would be nice to know a few things about NetworkIface:

1. The dhcp server seems to be assigning ip addresses to clients in a (possibly) erroneous manner. We’d like to keep a list of all the ips we’ve been assigned.
2. Sometimes the time between making a dhcp request and getting a response seems longer than reasonable. We’d like to time the execution of the dhcp_request method.
3. Some users are reporting strange failures that we can’t seem to reproduce. We would like to do exhaustive logging, ie. every method entry and exit, with parameters.

Now, this kind of debugging logic, however we realize it, is not really something we want in the release version of the application. It doesn’t belong. It belongs in debug builds, and we’re probably not going to need it permanently.

Here we will demo how to achieve the first point and omit the other two for brevity.

Where AOP comes in

Common to these issues is the fact that they all have to do with information gathering. But that’s not necessarily the only thing we might want to do. We might want to tweak the behavior of dhcp_request for the purpose of debugging. For instance, if it took too long to get an ip, we could set one statically after some seconds. Again, that would be a temporary piece of logic not meant to be in the release version.

Now, AOP says “don’t change your code, you’ll only make a mess of it”. Instead you can write that piece of code you need to write, but quite separately from your codebase. This you call an aspect, with the intention that it captures some aspect of behavior you want to inject into your code. And then, during compilation from source code to bytecode (or object code) you inject the aspect code where you want it to go. Compiler? Yes, AOP comes with a special compiler, which makes injection very toggable. Want vanilla code? Use the regular compiler. Want aspected code? Use the AOP compiler.

How does the compiler know where to inject the aspect code? AOP defines strategic injection points called join points. Exactly what these are depends on the programming language, but typically there is a join point preceding a method body, a join point preceding a method call, a method return and so on. (As we shall see, in aopy we are being more Pythonic.) Join points are defined by the AOP framework. But how do you tell it to inject there? With point cuts. A point cut is a matching string (ie. regular expression) which is matched against every join point and determines if injection happens there.

Back to you, John

Enough chatter, the code is getting cold! As it happens, Python has first rate facilities for writing AOP-ish code. We already have language features that can modify or add behavior to existing code:

• Properties let us micromanage assignment to/reading from instance variables.
• Decorators let us wrap function execution with additional logic, or even replace the original function with another.
• Metaclasses can do just about anything to a class by rebinding the class namespace arbitrarily.

We will use these language constructs as units of code injection, called advice in AOP. This way we can reuse all the decorators and metaclasses we already have and we can do AOP much the way we write code already. Let’s see the aspects then.

A caching aspect

The first thing we wanted was to cache the values of ip. For this we have a pair of functions which will become methods in NetworkIface and make ip a property.

# <aspects/cache.py>
class Cache():
    def __init__(self):
        self.values = set()
        self.value = None
cache = Cache()
 
def get(self):
    return cache.value
 
def set(self, value):
    if value:
        print "c New value: %s" % str(value)
    if any(cache.values):
        prev = ", ".join([str(val) for val in cache.values])
        print "c  Previous values: %s" % prev
    if value:
        cache.values = cache.values.union([value])
    cache.value = value

Download this code: cache.py

Cache is the helper class that will store all the values.

A spec

Aspects are defined in specification files which provide the actual link between the codebase and the aspect code.

# <./spec.py>
import aopy
 
import aspects.cache
 
caching_aspect = aopy.Aspect()
caching_aspect.add_property('main:NetworkIface/ip',
    fget=aspects.cache.get, fset=aspects.cache.set)
 
__all__ = ['caching_aspect']

Download this code: spec.py

We start by importing the aopy library and the aspect code we’ve written. Then we create an Aspect instance and call add_property to add a property advice to this aspect. The first argument is the point cut, ie. the matching string which defines what this property is to be applied to. Here we say “in a module called main, in a class called NetworkIface, find a member called ip“. The other two arguments provide the two functions we wish to use in this property.

Compiling

To compile the aspect into the codebase we run the compiler, giving the spec file. And we give it a module (or a path) that indicates the codebase.

$ aopyc -t spec.py main.py
Transforming module /home/alex/uu/colloq/aopy/code/main.py
Pattern matched: main:NetworkIface/ip on main:NetworkIface/ip

Download this code: main.compile

The compiler will examine all the modules in the codebase (in this case only main.py) and attempt code injection in each one. Whenever a point cut matches, injection happens. The transformed module is then compiled to bytecode and written to disk (as main.pyc).

main.pyc now looks like this:

# <./main.py> transformed
import sys ### <-- injected
for path in ('.'): ### <-- injected
    if path not in sys.path: ### <-- injected
        sys.path.append(path) ### <-- injected
 
import aspects.cache as cache ### <-- injected
 
class NetworkIface(object):
    def __init__(self):
        self.ip = None
 
    def dhcp_request(self):
        self.ip = (10,0,0,131) # XXX magic goes here

    ip = property(fget=cache.get, fset=cache.set) ### <-- injected
 
 
if __name__ == '__main__':
    iface = NetworkIface()
    iface.ip = (10,0,0,1)
    iface.ip = (10,0,0,2)
    iface.dhcp_request()

Download this code: main_t.py

Injected lines are marked. First we find some import statements that are meant to ensure that the codebase can find the aspect code on disk. Then we import the actual aspect module that holds our advice. And finally we can ascertain that NetworkIface has gained a property, with get and set methods pulled in from our aspect code.

Running aspected

When we now run main.pyc we get a message every time ip gets a new value. We also get a printout of all the previous values.

c New value: (10, 0, 0, 1)
c New value: (10, 0, 0, 2)
c  Previous values: (10, 0, 0, 1)
c New value: (10, 0, 0, 131)
c  Previous values: (10, 0, 0, 1), (10, 0, 0, 2)

Download this code: main.output

And the yet the codebase has not been touched, if we execute main.py instead we find the original code.

Here the show endeth

And that wraps up a hasty introduction to AOP with aopy. There is a lot more to be said, both about AOP in Python and aopy in particular. Interested parties are kindly directed to these two papers:

1. Strategies for aspect oriented programming in Python
2. aopy: A program transformation-based aspect oriented framework for Python

If you prefer reading code rather than English (variable names are still in English though, sorry about that), here is the repo for your pleasure:

• aopy repo

And if you still have no idea what AOP is and think the whole thing is bogus then you can watch this google talk (and who doesn’t love a google talk!) by mr. AOP himself.

For example, look at that handsome guy there on the right. Is that a gorgeous graph, or what? (Incidentally, if you aren’t yet a fan of graphviz, it’s time to become one, it’s superb!)

Now take a gander at the code snippet below, that’s the python representation of it.

class Node(object):
    def __init__(self, name):
        self.name = name
 
a, b, c, d = Node('a'), Node('b'), Node('c'), Node('d')
a.refs = [b, d]
b.refs = [c]
c.refs = [a]
d.refs = []

Download this code: python_graphtraversal_model.py

But it’s not there just to be stared at, so let’s do something with it! One thing we could do is go over the graph and print out the nodes. But let’s do one better, let’s also show how deep into the graph we are by indenting the output! What we want is this:

a
__b
____c
__d

Download this code: python_graphtraversal_spec.py

We don’t have the nice arrows here, but you can still make out the shape of the graph.

First iteration

def traverse_naive(node, depth):
    print('__'*depth + node.name)
    for ref in node.refs:
        traverse_naive(ref, depth+1)

Download this code: python_graphtraversal_naive.py

Very simple. Print the name of the node at the current level of depth, and then recurse down the outgoing edges. But when we run this something bad happens:

>>> traverse_naive(a, 0)
a
__b
____c
______a
________b
..
RuntimeError: maximum recursion depth exceeded

Download this code: python_graphtraversal_naive_output.py

There’s no sign of d, instead we keep going round and round along the path a-b-c-a. Woops.

Preempting cycles

Traditionally, graph traversal algorithms have set properties on nodes in the graph to indicate to themselves that a particular node had been seen before. For example, we could set node.been_here_before = True every time we enter a node, and then we could check for this property to make sure we don’t re-enter the node later.

But this is not awesome, because we then have to change the graph as we traverse it. What if we want to traverse it again later, do we then need another traversal algorithm to remove all the markers or what?

There is another way to do this, however. We can use a data structure completely outside the graph in which we keep track of what we’ve seen so far (imagine a guy walking around a warehouse with a clipboard, he then doesn’t have to mark any of the merchandise!).

So instead of checking the value of node.been_here_before, we’re going to check the value of cache[node].

Now, there are two main strategies for where to do this check. We can do it right before the recursive call, or right after.

def traverse_check_before(node, depth, cache):
    cache[node] = None
 
    print('__'*depth + node.name)
    for ref in node.refs:
        # we are about to recurse, first check if the node we want
        # to recurse on is in the cache
        if ref in cache:
            print('Already in cache: %s' % ref.name)
            continue
        # recurse if we reach this point
        traverse_cached(ref, depth+1, cache)
 
 
def traverse_check_after(node, depth, cache):
    # we have just recursed, return if this node is already in
    # the cache
    if node in cache:
        print('Already in cache: %s' % node.name)
        return
    cache[node] = None
 
    print('__'*depth + node.name)
    for ref in node.refs:
        # recurse without exception
        traverse_cached(ref, depth+1, cache)

Download this code: python_graphtraversal_two.py

Here it doesn’t matter which one we use, so we’re just going to use the second option.

Second iteration

At this point you might be thinking that everything is going swimmingly, but in fact a small problem has crept up. It cannot have escaped your attention that we’ve snuck another parameter into that function definition. This means that we have to call the function like this:

>>> traverse_check_after(a, 0, {})
a
__b
____c
Already in cache: a
__d

Download this code: python_graphtraversal_param.py

The function works, but we really, really don’t want to have to do it like this. There is no reason the caller has to know about the cache, let alone that it is a dictionary. And if we ever decide to change the cache mechanism, we have to update all the client code.

So what can we do? We can’t set up the cache in the function body, because the function is recursive!

This is where you might get that look in your eye that you get when you’re being clever. Hah, but what about this:

def traverse_cached(node, depth, cache={}):
    if node in cache:
        print('Already in cache: %s' % node.name)
        return
    cache[node] = None
 
    print('__'*depth + node.name)
    for ref in node.refs:
        traverse_cached(ref, depth+1, cache=cache)

Download this code: python_graphtraversal_cached.py

Brilliant! We don’t have to set up the cache neither inside the function body nor at the call site. This cache has to be initialized somewhere between the call and the function being executed, but somehow we’ve found a magical in-between, haven’t we!

Keyword parameters really are a tremendous boon, but unfortunately they will not save our skin this time. Do you know what happens if we do this?

>>> traverse_cached(a, 0)
>>> traverse_cached(a, 0)

Download this code: python_graphtraversal_cached_call.py

Read and weep:

>>> traverse_cached(a, 0)
a
__b
____c
Already in cache: a
__d
>>> traverse_cached(a, 0)
Already in cache: a

Download this code: python_graphtraversal_cached_output.py

Keyword parameters

Keyword parameters don’t do what you thought they did. You thought traverse_cached would get a new dictionary every time it was called without a cache argument. But that’s not what happens. (But isn’t python magical???)

In fact, it works like this. The cache keyword parameter gets initialized once and for all when the function is compiled. Since the value it is set to is an empty dictionary, the dict object is initially empty. But every time you call the function, it’s the same dictionary that gets passed in!

Third iteration

It’s come to this, I’m afraid. The problem is unsurmountable. We can’t not pass the cache parameter. But we definitely don’t want to do it at the call site. Which leaves no other option than…

def traverse_cached_fixed(node, depth):
    def rec(node, depth, cache):
        if node in cache:
            print('Already in cache: %s' % node.name)
            return
        cache[node] = None
 
        print('__'*depth + node.name)
        for ref in node.refs:
            rec(ref, depth+1, cache)
    rec(node, depth, {})

Download this code: python_graphtraversal_fixed.py

I really hate using an inner function just for this. It makes it messier, you have to route the arguments through an extra function call. But that’s the price you pay, I’m afraid.

Postscript

By now you must be fuming at the fact that I’ve come all this way while pretending that the function call didn’t have another parameter that was completely pointless and indeed deserving of the very same criticism.

Well, the difference is that you can set depth=0 as a keyword parameter in the function definition, so that you don’t have to pass it. And this doesn’t break anything. But why??

The answer is found when poking around python. A function is in fact a mutable object. Have a look:

def f(a=0):
    a = 1
 
>>> print f.func_defaults
(0,)

Download this code: python_graphtraversal_keywords1.py

func_defaults is the tuple that stores the values of the keyword parameters which have a default value. The value we see here is the integer 0, and it will not change no matter how many times we execute the function.

But collection types (and your own types too) are different:

def g(a=[]):
    a.append(1)
    print a
 
>>> print g.func_defaults
([],)
>>> g()
[1]
>>> print g.func_defaults
([1],)
>>> g()
[1, 1]

Download this code: python_graphtraversal_keywords2.py

Despite different, it actually works the same way (sorta). The binding of the keyword default is constant. But with an integer, the parameter is bound to the constant 0, whereas with “heavier” objects, it is bound to the object, but the object’s inner properties can very well mutate!!!

Well, bash --version tells you about bash, ls --version tells you about coreutils and so on, you can keep going with that. uname will tell you about the platform and the kernel, useful stuff. What about the distro? Distros are diverse, sometimes it helps a lot to know what kind of environment you’re in. Well, strangely enough, there’s no standard answer to that.

They all do their own thing, if they do at all. Some distros use lsb_release to dispense this information. Others have files in /etc that you can check for, if you know what they are supposed to be called. So I decided to try and detect this. I’ve checked a bunch of livecds and it works on those distros that identify themselves somehow.

# Author: Martin Matusiak <numerodix@gmail.com>
# Licensed under the GNU Public License, version 3
#
# <desc> Detect OS (platform and version) of local machine </desc>
#
# <usage>
# source this file in bash, then run `osdetect`
# </usage>
 
 
_concat() {
	local s="$1";shift;
	while [ "$1" ]; do
		s="$s $1"; shift
	done
	echo "$s" | sed "s|^[ \\t]*||g" | sed "s|[ \\t]*$||g"
}
 
_glob() {
	local file=
	local glob=
	local lst=
	while [ -z "$file" ] && [ "$1" ]; do
		glob="$1";shift;
		lst=$(ls $glob 2>/dev/null | grep -v /etc/lsb-release)
		if [ "$lst" ]; then
			file=$(echo "$lst" | head -n1)
		fi
	done
	echo "$file"
}
 
osdetect() {
	# ref: http://linuxmafia.com/faq/Admin/release-files.html
 
	local os=
	local release=
	local machine=
	if ! which uname &>/dev/null; then
		echo -e "${cred}No uname on system${creset}" >&2
		os="N/A"
	else
		os=$(uname -s)
		release=$(uname -r)
		machine=$(uname -m)
	fi
	if [ "$os" = "SunOS" ]; then
		os="Solaris"
		machine=$(uname -p)
	fi
	local platform="$(_concat "$os" "$release" "$machine")"
 
	# prefer lsb_release
	if which lsb_release &>/dev/null; then
		local id="$(_concat "$(lsb_release -i | sed "s|.*:||g")")"
		local rel="$(_concat "$(lsb_release -r | sed "s|.*:||g")")"
		local code="$(_concat "$(lsb_release -c | sed "s|.*:||g")")"
	elif [ -f /etc/lsb-release ]; then
		local id="$(_concat "$(grep DISTRIB_ID /etc/lsb-release | sed "s|.*=||g")")"
		local rel="$(_concat "$(grep DISTRIB_RELEASE /etc/lsb-release | sed "s|.*=||g")")"
		local code="$(_concat "$(grep DISTRIB_CODENAME /etc/lsb-release | sed "s|.*=||g")")"
 
	# find a file or another
	else
		local vfile=$(_glob "/etc/*-rel*" "/etc/*_ver*" "/etc/*-version")
		[ "$vfile" ] && local id=$(cat "$vfile")
 
		# distro specific
		[ "$vfile" = /etc/debian_version ] && [ "$id" ] && id="Debian $id"
	fi
 
	[ "$id" = "n/a" ] && id=
	[ "$rel" = "n/a" ] && rel=
	[ "$code" = "n/a" ] && code=
 
	local version="$(_concat "$id" "$rel" "$code")"
	[ ! -z "$version" ] && version=" ~ ${cyellow}$version${creset}"
 
	echo -e "Platform: ${ccyan}${platform}${creset}${version}"
}

Download this code: osdetect.sh

Let’s use findpkgs as an example here. The function is defined in a separate file and the file is source’d. But this means that every time I start a new shell session the whole function has to be read and loaded into memory. That might not be convenient if there are a number of those. networktest, for instance, defines four function and is considerably longer.

So let’s “compile to bytecode” again:

findpkgs ()
{
    [ -f ~/.myshell/findpkgs.sh ] && . ~/.myshell/findpkgs.sh;
    findpkgs $@
}

Download this code: lazyfuncionload_loadfunc

When the function is defined this way, the script actually hasn’t been sourced yet (and that’s precisely the idea), but it will be the minute we call the function. This, naturally, will rebind the name findpkgs, and then we call the function again, with the given arguments, but this time giving them to the actual function.

Okay, so that was easy. But what if you have a bunch of functions loaded like that? It’s gonna be kinda messy copy-pasting that boilerplate code over and over. So let’s not write that code, let’s generate it:

lazyimport() {
# generate code for lazy import of function
	function="$1";shift;
	script="$1";shift;
 
	dec="$function() {
		[ -f ~/.myshell/$script.sh ] && . ~/.myshell/$script.sh
		$function \\$@
	}"
	eval "$dec"
}

Download this code: lazyfuncionload_lazyimport

Don’t worry, it’s the same thing as before, just wrapped in quotes. And now we may import all the functions we want in the namespace by calling this import function with the name of the function we want to “byte compile” and the script where it is found:

## findpkgs.sh
 
lazyimport findpkgs findpkgs
 
## networktest.sh
 
lazyimport havenet networktest
lazyimport haveinet networktest
lazyimport wifi networktest
lazyimport wifiscan networktest
 
## servalive.sh
 
lazyimport servalive servalive

Download this code: lazyfuncionload_functions

So let’s imagine a hypothetical execution. It goes like this:

1. Start a new bash shell.
2. Source import.sh where all this code is.
3. On each call to lazyimport a function declaration is generated, and eval’d. The function we want is now bound to its name in the shell.
4. On the first call to the function, the generated code for the function is executed, which actually sources the script, which rebinds the name of the function to the actual code that belongs to it.
5. The function is executed with arguments.
6. On subsequent executions the function is already “compiled”, ie. bound to its proper code.

So what happens, you may wonder, in cases like the above with networktest, where several mock functions are generated, all of which will source the same script? Well nothing, whichever of them is called first will source the script and overwrite all the bindings, remember? It only takes one call to whichever function and all of them become rebound to the real thing. So all is well.

But what about files that are changing? Files get transfered all the time, but network connections are not always reliable. Have you ever been in the middle of a transfer wondering if it just stopped dead, wondering if it’s crawling along very slowly, too slow, almost, to notice? Or how about downloading something where the host doesn’t transmit the size of the file, so you’re just downloading not knowing how much there is left?

These things happen, not everyday, but from time to time they do. And it’s annoying. A file manager doesn’t really do a great job of showing you what’s happening. Of course you can stare at the directory and reload the display to see if the file size is changing. (Or the time stamp? But that’s not very convenient to check against the current time to measure how long it was since the last change.) Or maybe the file manager displays those updates dynamically? But it’s still somewhat lacking.

Basically, what you want to know is:

1. Is the file being written to right now?
2. How long since the last modification?

And you want those on a second-by-second basis, ideally. Something like this perhaps?

Here you have the files in this directory sorted by modification time (mtime). One file is actively being written to, you can see the last mtime was 0 seconds ago at the time of the last sampling. Sampling happens every second, so in that interval 133kb were written to the file and the mtime was updated. The other file has not been changed for the last 7 minutes.

The nice thing about this display is that whether you run the monitor while the file is being transfered or you start it after it’s already finished, you see what is happening, and if nothing is, you see when the last action took place.

#!/usr/bin/env python
#
# Author: Martin Matusiak <numerodix@gmail.com>
# Licensed under the GNU Public License, version 3.
#
# <desc> Watch directory for changes to files being written </desc>
 
import os
import sys
import time
 
 
class Formatter(object):
    size_units = [' b', 'kb', 'mb', 'gb', 'tb', 'pb', 'eb', 'zb', 'yb']
    time_units = ['sec', 'min', 'hou', 'day', 'mon', 'yea']
 
    @classmethod
    def simplify_time(cls, tm):
        unit = 0
        if tm > 59:
            unit += 1
            tm = float(tm) / 60
            if tm > 59:
                unit += 1
                tm = float(tm) / 60
                if tm > 23:
                    unit += 1
                    tm = float(tm) / 24
                    if tm > 29:
                        unit += 1
                        tm = float(tm) / 30
                        if tm > 11:
                            unit += 1
                            tm = float(tm) / 12
        return int(round(tm)), cls.time_units[unit]
 
    @classmethod
    def simplify_filesize(cls, size):
        unit = 0
        while size > 1023:
            unit += 1
            size = float(size) / 1024
        return int(round(size)), cls.size_units[unit]
 
    @classmethod
    def mtime(cls, reftime, mtime):
        delta = int(reftime - mtime)
        tm, unit = cls.simplify_time(delta)
        delta_s = "%s%s" % (tm, unit)
        return delta_s
 
    @classmethod
    def filesize(cls, size):
        size, unit = cls.simplify_filesize(size)
        size_s = "%s%s" % (size, unit)
        return size_s
 
    @classmethod
    def filesizedelta(cls, size):
        size, unit = cls.simplify_filesize(size)
        sign = size > 0 and "+" or ""
        size_s = "%s%s%s" % (sign, size, unit)
        return size_s
 
    @classmethod
    def bold(cls, s):
        """Display in bold"""
        term = os.environ.get("TERM")
        if term and term != "dumb":
            return "\\033[1m%s\\033[0m" % s
        return s
 
class File(object):
    sample_limit = 60  # don't hold more than x samples
 
    def __init__(self, file):
        self.file = file
        self.mtimes = []
 
    def get_name(self):
        return self.file
 
    def get_last_mtime(self):
        tm, sz = self.mtimes[-1]
        return tm
 
    def get_last_size(self):
        tm, sz = self.mtimes[-1]
        return sz
 
    def get_last_delta(self):
        size_last = self.get_last_size()
        try:
            mtime_beforelast, size_beforelast = self.mtimes[-2]
            return size_last - size_beforelast
        except IndexError:
            return 0
 
    def prune_samples(self):
        """Remove samples older than x samples back"""
        if len(self.mtimes) % self.sample_limit == 0:
            self.mtimes = self.mtimes[-self.sample_limit:]
 
    def sample(self, mtime, size):
        """Sample file status"""
        # Don't keep too many samples
        self.prune_samples()
        # Update time and size
        self.mtimes.append((mtime, size))
 
class Directory(object):
    def __init__(self, path):
        self.path = path
        self.files = {}
 
    def prune_files(self):
        """Remove indexed files no longer on disk (by deletion/rename)"""
        for f in self.files.values():
            name = f.get_name()
            file = os.path.join(self.path, name)
            if not os.path.exists(file):
                del(self.files[name])
 
    def scan_files(self):
        # remove duds first
        self.prune_files()
        # find items, grab only files
        items = os.listdir(self.path)
        items = filter(lambda f: os.path.isfile(os.path.join(self.path, f)),
                       items)
        # stat files, building/updating index
        for f in items:
            st = os.stat(os.path.join(self.path, f))
            if not self.files.get(f):
                self.files[f] = File(f)
            self.files[f].sample(st.st_mtime, st.st_size)
 
    def display_line(self, name, time_now, tm, size, sizedelta):
        time_fmt = Formatter.mtime(time_now, tm)
        size_fmt = Formatter.filesize(size)
        sizedelta_fmt = Formatter.filesizedelta(sizedelta)
        line = "%6.6s   %5.5s   %6.6s   %s" % (time_fmt, size_fmt,
                                               sizedelta_fmt, name)
        if time_now - tm < 6:
            line = Formatter.bold(line)
        return line
 
    def sort_by_name(self, files):
        return sorted(self.files.values(), key=lambda x: x.get_name())
 
    def sort_by_mtime(self, files):
        return sorted(self.files.values(),
                      key=lambda x: (x.get_last_mtime(),x.get_name()))
 
    def display(self):
        time_now = time.time()
        files = self.sort_by_mtime(self.files.values())
        print("\\nmtime>   size>   delta>   name>")
        for f in files:
            line = self.display_line(f.get_name(),
                                     time_now, f.get_last_mtime(),
                                     f.get_last_size(), f.get_last_delta())
            print(line)
 
 
def main(path):
    directory = Directory(path)
    while True:
        try:
            directory.scan_files()
            directory.display()
            time.sleep(1)
        except KeyboardInterrupt:
            print("\\rUser terminated")
            return
 
 
if __name__ == '__main__':
    try:
        path = sys.argv[1]
    except IndexError:
        print("Usage:  %s /path" % sys.argv[0])
        sys.exit(1)
    main(path)
 

Download this code: peek.py