tadhg.com » coding

Metaweb Acquired by Google

Tadhg — Sat, 17 Jul 2010 06:57:46 +0000

The official announcements are out, and my former employer Metaweb no longer exists. I’m happy that the ideas, and most of the people, have found a home, but it feels strange that the company is no longer a distinct entity. In many ways it makes a lot of sense for Google to end up owning them, and I hope the former-Metaweb-now-Google employees prosper.

Many of them are good friends of mine, and that in itself is significant. I was there for under two years; the proportion of people that I get on well with or admire (or both) there was very high. For the most part they did an excellent job of hiring, and of keeping the company culture coherent and inspiring.

My time there was important professionally as well as rewarding personally. It’s where I first really dug into Python, and through learning Python and working there I feel as if I embarked on a process that has seen me gain quite a few levels in the Software Engineer class. The exposure there to extremely skilled and accomplished people certainly helped me a great deal.

Those aren’t the only reasons I wish them all well. I still have the classic geek’s desire for a more rationally structured world, and the classic geek’s optimism that thinking about and organizing things in better ways can help bring that rational structure closer. Those combine to make me want to see realized the three major possibilities (in my view) that the Metaweb project presents:

A canonical namespace/“topic DNS” for, well, everything, so that words can easily be annotated (and analyzed, and manipulated) with pointers that make their actual meaning clear.
A public repository of schemas for more or less everything, so that individual groups or organizations could simply use and build upon those, rather than separately creating their own—standardization in that arena would vastly improve the user experience for innumerable things online (especially if you’re as picky as I am) and, not inconsequentially from my perspective, also eliminate all the work that programmers do reinventing the wheel with independently-developed schemas for common things.
A public repository of all the data for all the things covered by the first two points. This is what most people probably think the key point of Metaweb/Freebase is, and while it’s certainly an amazing and lofty goal, it’s not more important than the first two, both of which could be truly revolutionary if done on a large enough scale (and adopted by sufficient numbers).

So, too all those moving to Google: best of luck, and I hope you succeed at those three goals!

To all those who have contributed, in ways large or small, to the project formerly known as Metaweb: well done, you’ve helped make something of potentially great importance, and I have no doubt that at some point the ideas you’ve helped develop will be extremely influential, and possibly ubiquitous. To even get this far towards such an outcome is no small achievement.

Tags: coding, database, organization, personal, python, semantic-web, work

Unit Testing as Game

Tadhg — Fri, 16 Jul 2010 06:23:43 +0000

One of the great things about unit testing is that you can get into a game-like mode where you make incremental but measurable progress—“flow”, basically—but what if it’s just not similar enough to a game for you?

Install Unit Testing Achievements, a Python package that works with nose, unittest, and Django. Somewhat crazy, definitely hilarious. Some of the achievements themselves are excellent, such as My God, It’s Full of Dots: The suite has at least 2,001 passing tests.

Sadly not yet working is another one I like, Heisenbug: Make a passing suite fail without changing anything.

Tags: coding, flow, humor, psychology, python

8-Bit Cities

Tadhg — Tue, 13 Jul 2010 04:35:14 +0000

As a geek of my generation, I cannot help but find an eight-bit map of San Francisco (there are maps of other cities too) a wonderful thing. It became even cooler when I found out it was programmed in Python.

Tags: coding, culture, tech

iPad First Impressions: Consumption Machine

Tadhg — Mon, 24 May 2010 01:52:52 +0000

I got an iPad for work on Friday, and have been playing around with it. I would not have bought one for myself, and have grave misgivings about the device, primarily due to its highly proprietary, locked-down, walled-garden approach.

That being said, I think it’s an extremely slick, well-designed device, and may represent the first steps towards a new phase in accessing computer and/or internet artifacts.

Before I get to that, I want to note that I hated having to synch it to iTunes before I could use it. For a “next-generation” device, one that supposedly breaks with older computing paradigms, that seemed rather pathetic. Also, I can’t stand iTunes, and the iPad setup was the only time I’d started it in the years I’ve had my Macbook Pro.

Once that was out of the way, however, it was all slick and smooth thereafter. Using the device: easy. Installing applications: easy. Browsing the web: easy.

The interface is, as promised, extremely intuitive. It works as you expect it to work. It’s also just fun to use, somehow.

So what’s the problem? Well, all of the criticisms about its closed model remain true. That it’s not easy to tinker with, that applications have to be approved by Apple, that getting data on and off it has to be managed by iTunes or some other application—these are all really bad things, in my opinion. That ground has been covered before, and if you’re interested you can read Cory Doctorow’s take, which I largely agree with.

What struck me even more than those issues, however, is just how geared towards consumption the iPad is. You add functionality via one mechanism: the App Store. While you can get some things for free there, the name makes clear that it’s about buying things. You can get content through the iTunes store. Many of the other apps, like the amazon.com application (or the Kindle application), are focused on making it easy to buy things.

Beyond that, the activities the device is suited for are consumption activities. Reading, browsing, buying, and watching videos are all really easy. I haven’t tried games on it yet, but I suspect that while games on it will be fun, they won’t (yet, at least) be as deep as games on dedicated platforms or on fully general-purpose computers. It does, however, support “casual games” quite well.

Up until this point, all general-purpose computing devices have been geared towards productivity of some kind. Even if they ended up being used as gaming boxes or as entertainment and netsurfing devices, they were designed for production of some kind, usually writing, programming, or management. This is not the case with the iPad. Regardless of whether or not you could actually edit video, or create music, or write efficiently, on it, its primary purpose is consumption. As such, I think it represents a significant break in the history of computing and the internet as the first mass-market, general-purpose device that is geared towards consuming rather than both consuming and producing.

This raises a question, one that is critical to assessing the validity of many of the criticisms of it (such as Doctorow’s): have computers been combined production/consumption machines up until now merely due to technological limitations, or is that combination an essential part of what a computer is? I think that what Jobs has done, deliberately and effectively, is tried to separate those two parts and get rid of the irritating bits that support production so that consumption can be made accessible and easy to more and more people.

Furthermore, Apple recognized that what made this attempted break with computing history possible was the touch interface. Crucially, the touch interface removes a layer of abstraction. Now there’s no cursor to move around with a mouse, and no commands to invoke via unknown (or clumsily announced) keystrokes. No abstraction, and hence almost no need for training. In many ways, the iPad is a device aimed squarely at the people who can’t tell the difference between an article about Facebook’s login page and Facebook’s login page itself.

This is in many ways an inevitable step. URLs aren’t the best-designed things in the world (having the hierarchy go specific-to-general for domains and general-to-specific for resources on a domain, for example, is clearly screwy), and turning sites (accessible by URLs) into applications (accessible like other applications, by an easy launch mechanism) is probably inevitable too. But there’s clearly a loss with this, as abstraction in computing generally means power. That is, being able to manipulate resources with abstractions gives the user power, and abstracting things out before they reach the user takes power away from the user while making the smaller set of tasks they’re still able to do easier.

Many people will be in favor of this tradeoff; I’m not one of them, although I’d feel a lot better about it if it were a transient state that one could get past on a device (which I hope will be the case with the post-iPad tablets that emerge) rather than the state the device desires to keep its users in (which is the iPad’s approach).

Tags: coding, consciousness, culture, internet, tech, usability

sabbatical.close()

Tadhg — Mon, 17 May 2010 06:36:56 +0000

After a highly enjoyable, productive, and extended period, it’s time for me to return to the world of paid work.

I’m quite happy with the things I’ve done during my time off. Many of them are important only to me, but then, it’s been my time off.

Here are some of the things I’ve done since leaving Metaweb that I consider significant:

CrossFit. I’m approaching the one year mark, and it’s hard to overstate the importance of that for me. This is the first time in my life I’ve managed to keep an exercise routine of any kind going for that long. Not coincidentally, I’m also in the best shape of my life. I now feel that some kind of regular, intense, exercise has a similar place in my life to my morning pages—that is, something I I regard as mandatory and more or less permanent.
Diet. Not just switching back to eating meat, but also radically cutting down on sugar and processed foods. This also has a lot to do with my being in good shape.
Document format. Until this year, I hadn’t found a text-based document format I liked, but I have now finally settled on reStructuredText. One of the effects of this is that I shifted to doing all of my writing in a text editor rather than doing some of it in word processors. Another effect is that all of my writing is now much more amenable to programmatic manipulation.
Programming my own environment. Following from the previous item, I spent quite a bit of time working on solutions to various small niggles in my workflows, so that far less of my time is consumed by setup for writing. Some examples of this include scripts to insert the template for blog posts into my editor, scripts to publish to my blog directly from my editor, scripts to do word counting precisely as I wish it to be done, and many other small utilities that remove minor annoyances that I previously encountered almost every day. In many respects this kind of thing is precisely what programming and computers are for: to make it easier to create, and to lessen the gap between a thought and expression of that thought.
Digital organization. This includes recovering my subversion repository, restoring my blog to proper working order, and various other steps that have made my digital domain much more manageable. This is bound up with the previous item too.
Switching to Vim. This was probably a long time coming, but nevertheless required a fair amount of work, not least in porting all my scripts for jEdit over to it. I’ve been extremely happy with Vim. I think that to most people this might sound trivial; how many people would regard switching word processors as an “achievement”? But a text editor is different from a word processor, and switching to Vim isn’t like switching between non-modal text editors. This, combined with moving all my writing to text editors and thus to Vim, is a profound change for me, and one that I’m very happy I took the time to get through.
WordPress plugins. I wrote some of these to improve my blog after I got it working again.
PyWordCount. My word counting script/Vim plugin is done, but I have to finish the documentation for it and make it public. It’s significant to me because it’ll be my first self-contained public Python project, and also because it’s an outgrowth of the document format switch, the Vim switch, and the focus on programming my own environment to a much greater degree.
Q’Rith. I ran my first roleplaying sessions in 15 years, and am currently running my first roleplaying campaign in 15 years, in a world that I’ve created and am quite happy with so far.
Tennis. I started playing it again, and don’t know why I ever stopped.

Those are the main highlights. Many of them are internal, or are things that only I notice, but I think the effect they’ve had on me is actually quite major, and in positive ways.

As for what’s next, I’ll write about that tomorrow.

Tags: coding, CrossFit, diet, document formats, exercise, food, games, goals, personal, python, reStructuredText, roleplaying, software, sports, T, tech, tennis, text editing, version-control, Vim, WordPress, writing

Rather Better Than a Cover Letter

Tadhg — Fri, 14 May 2010 01:55:12 +0000

It might sound like a cheesy stunt, but I have a hard time coming up with a more fitting way to land a job at an advertising agency.

Tags: advertising, coding, work

Improving a Python Word Counting Function

Tadhg — Thu, 29 Apr 2010 20:33:36 +0000

This post could be summarized as “regular expressions are a lot faster than naive for loops”.

I’ve been working on improving the script I use for live wordcount in Vim, partly for performance and partly so that I can package it up as a plugin and share it with other people. Along the way I’ve improved the speed of the script rather significantly, and will go through the key part of that change here.

The core part of the script, unsurprisingly, is the count_words() function. As is common, it treats groups of letters surrounded by spaces as “words”. New lines also mark the end of a word. So do a variety of other characters, such as em dashes—in fact, one of the motivations for writing my own wordcount was because I wanted em dashes to be recognized as word separators.

In the following code, self.WORD_SEPS and self.LINE_SEPS are lists containing characters that separate words and lines. self.REPEATER_SEPS contains characters like “-”, which in my opinion does not separate words when used singly (e.g. “twenty-three” is one word) but does when used consecutively; “–” is a simple way to write an em dash when you’re restricted to ASCII. self.IGNORE is a list of characters that cannot form words on their own but aren’t necessarily separators, either.

The code is quite naive in its approach. It goes through the text letter by letter, checks the letter to see whether or not it’s in one of those lists, adjusts the count accordingly, and also stores previous_char in order to be able to handle the repeater separators.

def count_words(self, text):
    words, lines = 0, 1
    word, previous_char = 0, None
    lineword_separators = self.LINE_SEPS + self.WORD_SEPS
    repeater_test = lambda c, pc: bool(
        c in self.REPEATER_SEPS and pc in self.REPEATER_SEPS)
    separator_test = lambda c, pc: bool(
        char in lineword_separators or repeater_test(c, pc))
    for char in text:
        if separator_test(char, previous_char):
            word = 0
            if char in (self.LINE_SEPS):
                lines = lines + 1
        elif char in (self.IGNORE):
            pass
        else:
            #it’s part of a word.
            if not word:
                words = words + 1
                word = 1
        previous_char = char

    return (words, lines)

With that code as the central function, the script takes about three seconds to process a text file that’s approximately 400K/68,000 words.

The new version eliminates the need for a separate list of repeater separators, is fewer lines of code, and is faster:

def count_words(self, text):

    def ors(l): return r"|".join([re.escape(c) for c in l])
    def retext(text, chars, sub):
        return re.compile(ors(chars)).sub(sub, text)

    lines = text and len(re.compile(ors(self.LINE_SEPS)).split(text)) or 0

    text = retext(text, self.WORD_SEPS + self.LINE_SEPS, u" ")
    text = retext(text.strip(), self.IGNORE, u"")
    words = text and len(re.compile(r"[ ]+").split(text)) or 0

    return (words, lines)

The basic approach here is different. Rather than going through the text and evaluating it character by character, instead it replaces all of the characters in the separator lists with spaces, removes all the characters in the ignore list, uses split() to get a list of all the words in the text, and then counts the length of that list.

The most awkward part of it is having to construct the regular expressions out of the lists of characters, which is what the ors() function does. I could have replaced the lists with regular expressions, but I want it to be easy for users to alter the lists to suit their own preferences, and so left them as plain lists.

(Note that I’m restricted to Python 2.3 here since this needs to run in MacVim; hence the old-style and and or instead of new-style ternary expressions.)

Using that code instead of the previous version, and operating on the same 400K/68,000-word file, the script takes about 0.12 seconds—a 25-fold speed increase. Since this is a “live” wordcount, one that has to recount the entire file whenever you do something other than typing or adding a single new line, that’s an important difference, and it makes it much more usable when working with large files.

I had no illusions that my original version was anything close to optimal when I first wrote it, but I didn’t realize that the speed gains from doing it better would be so significant. Python’s regular expression handling is done by a C module, and I suspect that C module is quite highly optimized indeed. So, if speed is important, let highly-optimized C do the work, instead of doing it in your own code.

Tags: coding, python, software, tech, text editing, Vim, writing

How to Solve It and Project Euler

Tadhg — Sat, 17 Apr 2010 03:35:41 +0000

Some tips on problem solving.

Some problems.

Tags: coding, math, psychology

Python Flatten/Concantenate Comparisons

Tadhg — Sat, 10 Apr 2010 07:31:32 +0000

I’m going to compare seven different ways in Python to make one list out of several lists containing different numbers of elements, something that strikes me as a common but not necessarily everyday operation.

The philosophy of Python is that generally there should be one obvious and reasonable way to do things. I really like this philosophy—except when the Python way isn’t the way I prefer, of course.

I seem to instinctively like functional programming approaches, or at least an approach of breaking things down into small functions, each of which takes up a small amount of space in terms of code and which does just one thing. More and more I find myself thinking that if I can express succinctly (in English) what a block of code should do[1], the block shouldn’t be more than a couple of lines. I think this drive pushes me away from the Pythonic way of doing things.

So we have a list of lists. We want to make the list of lists into one list. [[1], [2, 3]] should become [1, 2, 3]. While trying to find the “right” way to do this trivial task, I became a little obsessed, hence this post and its seven approaches.

The first one is the one I suspect Python’s Benevolent Dictator For Life would want me (and everyone else) to use:

def flatten_for_loop(l):
    fl = []
    for i in l:
        for j in i:
            fl.append(j)
    return fl

It does the job, certainly. But at six lines, it just bugs me. My instinct tells me that “concatenate the lists” shouldn’t take that many lines. It’s possible that my instinct is simply off, but it’s pretty insistent.

This was my next idea:

from operator import add
def flatten_reduce_add(l):
    return reduce(add, l)

Yes, the much-maligned reduce. This does seem like a completely natural fit for it, however. I like this solution, but I don’t like having to import add, and I don’t like using reduce when in future it’ll need to be imported as well.

def flatten_lc(l):
    return [i for j in l for i in j]

While I love list comprehensions, this one just doesn’t sit well. Even though I know what it does, I have trouble parsing it for some reason—and that’s really a sign that I shouldn’t use it, since someone who doesn’t know what it does would have a significantly harder time.

Before discovering better ways, I went on to create my own function to do it:

def listadd(l, new=[]): return listadd(l, new + l.pop(0)) if l else new
def flatten_custfunc(l, new=[]):
    return listadd(l[:])

This modifies the list, necessitating the creation of a copy, and overall it’s pretty inefficient; I didn’t like it much, which is why I kept looking.

def flatten_sum(l):
    return sum(l, [])

Possibly my favorite. The problem is that since I didn’t know that you could use sum on non-numbers—by giving it a value to start off with, which it normally defaults to 0—I suspect a lot of other people wouldn’t know it either, so it might cause confusion. I’m on the fence on this one—after all, the name gives it away, and so its intent should be quite clear. But there are cases where a reader might think that it was going to get the sum of all the numbers in the lists.

from itertools import chain
def flatten_chain(l):
    return [i for i in chain(*l)]

Another favorite. This one seems quite clear to me (in part because the name helps), but again it requires an import.

def flatten_map_extend(l):
    new = []
    map(new.extend, l)
    return new

Better than the original for-loop version, but it rubs me the wrong way—probably because I’ve seen solutions like the ones laid out above and their concision is exerting a dark influence.

Naturally, the next thing to do was to check the performance of the various options.

So I ran each a million times against a number of different inputs. The first input was a list of tuples and dicts:

[
    [(1, {’type’: ’type1’, ’name’: ’name1’})],
    [(2, {’type’: ’type2’, ’name’: ’name2’})],
    [(3, {’type’: ’type3’, ’name’: ’name3’})],
    [(4, {’type’: ’type4’, ’name’: ’name4’})],
    [(9, {’type’: ’type9’, ’name’: ’name9’}), (10, {’type’: ’type9’, ’name’: ’name9’}), (11, {’type’: ’type9’, ’name’: ’name9’}), (12, {’type’: ’type9’, ’name’: ’name9’})],
    [(13, {’type’: ’type9’, ’name’: ’name9’})],
    [(23, {’type’: ’type9’, ’name’: ’name9’})]
]

And these are the times it took for each function to run 1000000 times with that input:

flatten_map_extend	9.0762510299682617
flatten_lc	9.6673529148101807
flatten_reduce_add	9.8464798927307129
flatten_sum	10.240524053573608
flatten_chain	10.286052942276001
flatten_for_loop	11.377233028411865
flatten_custfunc 16.330123901367188

Next were lists of numbers:

[
    [0],
    [1],
    [2],
    range(3,100),
    range(100,200),
    range(200,300),
    range(300,400),
    range(400,500),
    range(500,600),
    range(600,700),
    range(700,800),
    range(800,900),
    range(900,1000),
]

flatten_map_extend	34.196415901184082
flatten_sum	61.231782913208008
flatten_reduce_add	65.624041080474854
flatten_custfunc	75.283849000930786
flatten_lc	96.376346826553345
flatten_chain	126.78751182556152
flatten_for_loop 213.21602821350098

Finally, lists of strings:

[
    ["abcde"] * 10,
    ["abcde"] * 10,
    ["abcde"] * 10,
    ["abcde"] * 10,
    ["abcde"] * 10,
    ["abcde"] * 10,
    ["abcde"] * 10,
    ["abcde"] * 10,
    ["abcde"] * 10,
    ["abcde"] * 10,
]

flatten_map_extend	11.517388820648193
flatten_reduce_add	15.214270114898682
flatten_sum	15.836833000183105
flatten_lc	20.376528024673462
flatten_chain	22.739732027053833
flatten_custfunc	23.072113990783691
flatten_for_loop 33.499195098876953

I’m running these tests on a 2.5 GHz Intel Core 2 Duo machine, 4GB RAM, OS X, Python 2.6.

I don’t know how informative they were, but using map and list.extend seems like the most performant option. Which is unfortunate from my point of view, since I don’t like the way that looks, but since I should remember to turn it into a function whenever I need it, I should be able to deal with that.

It must be said that neither performance nor code readability (for programmers other than myself) matter in the project I was working on when I decided to explore this question, but I do want to know the right way to do it; it appears that I found a reasonable answer, although I’m definitely going to be tempted to use sum instead some of the time…

[1]

Of course, I mean a succinct, specific, and complete explanation, rather than e.g. “it should solve my problem”, as going down that track would lead one to think that all programs should be one line long. They shouldn’t. But the line between concision and obscurity is subjective, and I’m trying to fine-tune my subjective take on that particular question of style.

Tags: coding, python

Public Projects, Finishing Things, and bitbucket

Tadhg — Tue, 06 Apr 2010 19:46:14 +0000

I’ve been writing Python fairly steadily for the past couple of years, and a significant amount of that has been for my own projects; Python is what I tend to use to scratch workflow-related itches. That’s great, but many of these projects reach a point short of “finished” when I stop working on them.

This is because I generally stop when they’re good enough for me to use. Which is fine, except that they hang around mentally, because I don’t feel that they’re fully done. So I’m going to make an effort to formally finish them by making them public, and pushing them to bitbucket.org. Making them public by posting them on my blog is fine, but feels much more like just pasting code into a blog post—which is what it is. Making them available to others in a convenient way is a better approach, and also forces me to get them to the point where I can call them done and feel good about it.

I selected my sorted Subversion status script for this, because it was fairly small but might conceivably be useful to others (I found the concept useful enough to write it, after all). I cleaned up it, which made it faster and better, and wrote a quick README for it, and put it on bitbucket.

I hope to do the same with other almost-finished scripts I have lying around.

Tags: coding, psychology, python

Refactoring, Geeky Enjoyment, and Readability

Tadhg — Mon, 05 Apr 2010 20:16:55 +0000

Refactoring is rewriting source code while preserving the functionality of that code. I’m currently refactoring my Python script for Subversion status, because I want to experiment with sharing code on bitbucket and it seemed like a reasonable first project to put up there.

Some people hate refactoring, but I often enjoy it. I get a kick out of figuring out how to make code “better”, although that’s often a subjective judgment. In this case, when I opened up the script to have a look at it, I immediately saw a function that I knew I wanted to refactor. I’m amused by the enjoyment I derived from making the fairly simple change.

The function gets the length of the longest filename in a list[1], returning 0 if for some reason there aren’t any filenames in that list. This was the original version:

def get_filename_width(self, files):
    width = 0
    lines = files.split("\n")
    for line in lines:
        if len(line) > width:
            width = len(line)
    return width

It expects the files variable to be a multiline string. So it assigns a value to width, then splits the string into individual lines, and goes through each line, testing its length against the value of width and making width into that value if it’s greater. After going through all the lines, it returns width.

It’s fine as a first pass, but looking at it now, it just seems clunky and too long. I rewrote it into this:

def get_filename_width(self, files):
    return max([len(line) for line in files.split(u"\n")]) if files else 0

Six lines into one. But is this too opaque?

If I’m the intended reader, clearly not, as I’m quite familiar with reading list comprehensions, and I also consider the Python 2.5+ ternary expression to be entirely legible.

If someone else is the reader? The ease with which they can break the line into chunks is the main issue, as first they have to see that the line returns the result of max() if files has some value and 0 otherwise. The next bit is seeing that max() operates on a list comprehension, and finally they have to see that the list comprehension creates a list of the lengths of the lines in files.

I can see the argument that the six-line version is clearer, but the fact that it’s six lines as opposed to one needs to be taken into account as well—more lines mean more cognitive overhead for the reader, something that is often underestimated in judging the readability of code.

I think that a reader familiar with Python would grasp the one-liner faster than the six-liner. The use of max is helpful here (whereas it might hinder a reader who doesn’t know it)—rather than having to discern what the for loop in the six-liner is doing, it’s clear that the one-liner is interested in getting the highest value. I personally think that the lack of variable assignment in the one-liner makes it easier as well, despite the familiarity that almost all readers would have with variables.

Clearly, this is a rather trivial example[2]. But the issues are relevant to writing shared code, and “shared” often applies even to personal code if you’re ever going to revisit it. While I definitely want to get something working quickly, it’s important to balance that with writing readable software. Readability requires finding the line between elegance and impenetrability, between concision and obscurity—just like writing more generally.

[1] Yet it’s called get_filename_width—because it’s concerned with the width (in columns) that the longest filename will take up in a terminal window. The purpose of the function in the larger context is related to the concept of width, while in the function itself it’s dealing with the length of individual filenames. You could argue for changing the name to get_filename_length, but I think the overall purpose of the function is more important to its name than what it’s doing internally.

[2]	It took me less than 3 minutes to write the new version of the function, whereas this post discussing it has taken rather a lot longer than that…

Tags: coding, python, writing

Clearing Growl Notifications

Tadhg — Fri, 02 Apr 2010 22:43:22 +0000

I use a notification system for OS X called Growl, which provides a single channel for various applications to use when they have something to tell me. I mainly use it for IM and email. I’m fairly happy with it, but one issue that irked me was dealing with clearing a bunch of the notifications off the screen.

You can set them to disappear on their own, but then you run the risk of missing them if you’re away from your keyboard, or if you just don’t happen to catch their content in time. So I set mine to all stay on the screen until they’re dismissed. In order to dismiss them, you either click to close each one individually, or you can restart Growl from the OS menu bar.

To do the latter, you more or less have to use the mouse, and that eventually began to annoy me. I looked around, and found that there was a commandline controller called growlctl, but that it was no longer included in Growl releases. It can be found in the Extras section of the Growl 0.7.6 release, though, which they still have available for download.

I downloaded that, installed it (which really just means moving it to somewhere on your path), and verified that it does indeed still work. I then created a rather simple script:

/usr/local/bin/growlctl restart

And put it in my homedir, calling it gr.sh. Quicksilver knows that it’s there, so now clearing the notifications requires, at most, the following keystrokes: gr. Itch scratched.

Tags: coding, OS X, personal, software, tech

2010 Goals: First Quarter Review

Tadhg — Tue, 30 Mar 2010 19:12:33 +0000

Yes, clearly my progress (or lack thereof) needs quarterly reports. And here’s the first one for this year.

I laid out the full list at the start of the year. I’ve accomplished some of them:

Celebrate my birthday. This still seems like a strange goal to have—but putting it on here helped me get it done, and I had a great time and don’t get why I didn’t do this for years. I’ll do it again next year!
Participate in the CrossFit Sectionals. Done!
Ten dead-hang pullups. Done; currently my best is 11, and I’m now aiming for 20 before going back to kipping.

I’ve made some progress on these:

Revive sfmagic.org. “Some progress” here means I’ve tried to get the code running again and have run into incompatibilities between Pylons versions. So, slightly closer, but not a lot of movement.
Run a roleplaying campaign. I’ve started it, and two sessions in it’s going well.
I’m presumably closer to my other fitness-related goals of a 300-pound deadlift, a sub-23:00 5K, and “Murph” in under 50:00.
It’s hard to judge progress on an amorphous goal like “building a sense of community”, but I think there’s been some.

I’ve made no progress on these:

Write at least one draft of the fantasy novel. I feel fine about this, and haven’t really tried to work on it. I know that at some point the desire to do so will resurface, and the key to getting it done is focusing on it when that happens.
Finish The Annotated Fantasy Bedtime Hour. Still something I just need to get down to doing…

Tags: coding, community, CrossFit, exercise, games, goals, MTG, personal, roleplaying, web-development, writing

Giles Bowkett on Finding New Programming Jobs

Tadhg — Tue, 23 Mar 2010 23:52:45 +0000

It’s too long, it’s full of self-promotion, and really it’s an ad for a video he’s selling, but it’s entertaining and contains some good advice: “Programmers: What to do if You Get Fired”.

This might be the best line:

If you’re looking for a better job, writing an amazing resume is a good place to start. I don’t mean just a better resume; I mean a resume that makes people stop asking if they should hire you and start asking if they can afford you.

—Giles Bowkett. “Programmers: What to do if You Get Fired”. Giles Bowkett’s blog, 8 Mar 2010.

Annoyingly, I find myself tempted to buy that video after reading his post…

So far, his 2008 presentation at RubyFringe seems pretty entertaining too. Maybe you should watch it after you’ve updated your résumé.

Update: at the end of that presentation, he says “build something because you believe it should exist”, which I agree with 100%.

Tags: advertising, coding, work, writing

A Little More Functional Programming

Tadhg — Fri, 26 Feb 2010 07:55:51 +0000

After reading Dhananjay Nene’s comment on my post about a functional style approach to the “find longest repeater” problem, I decided to follow the line from that comment and divide the program into functions for finding the longest contiguous block and then for comparing the blocks. Naturally, I wanted to do this without using any variables…

It wasn’t too hard to write a variable-free function to return a list of blocks of letters from a string, e.g. continuous_blocks("anna") produces ["a", "nn", "a"]:

def contiguous_blocks(string):
    def listbuilder(blocks, char, rest):
        if not rest:
            return add_to_blocks(char, blocks)
        return listbuilder(add_to_blocks(char, blocks), rest[0], rest[1:])

    def add_to_blocks(char, blocks):
        if blocks and char in blocks[-1]:
            return blocks[:-1] + ["".join([blocks[-1], char])]
        return blocks+[char]

    return listbuilder([], string[0], string[1:])

And then findlongestrepeater becomes:

def findlongestrepeater(string):
    def compare_blocks(previous, current):
        return current if len(current)>len(previous) else previous

    return reduce(compare_blocks, contiguous_blocks(string))

Note that this approach provides a clear use for reduce, which is on the outs in Python these days.

A crucial difference between Dhananjay Nene’s version and mine, however, is that mine returns a list and not a generator. So I decided to write a variable-free generator version… and just couldn’t do it. I don’t know what it was, but I kept getting hung up on trying to track the previous, current, and next characters, and to do that without variables while trying to skip at the right times in the for loop just seemed impossible. While writing this blog post, however, I figured it out. I had intended to put it up as a puzzle for other people, so I guess I’ll put my solution at the end. If you plan to try it, don’t scroll beyond the warning below.

There’s an easier way, too, which is simply to do this:

def contiguous_blocks(string):
    from itertools import groupby
    for k,g in groupby(string):
        yield u"".join(list(g))

I found that last night after being unable to bend my mind in the right way, because I was sure that this problem couldn’t be so rare that itertools wouldn’t have something to handle it.

However, I felt like that was cheating… so I kept trying to build my own variable-free generator.

Because of the fact that reduce isn’t going to be in Python 3, I decided I’d write a version of it that would work as its replacement for this problem. It didn’t seem too hard to come up with this:

def reducereplacement(func, generator):
    def runfunc(product, gen):
        try:
            return runfunc(func(product, gen.next()), gen)
        except StopIteration:
            return product

    return runfunc([], generator)

This only works with generators and not with lists, unlike the real reduce. There’s no reason to use this, either, given that reduce is in Python 2 and will be in the functools module for Python 3; I just wanted to try it as an experiment.

So, to restate the puzzle problem:

Write a version of contiguous_blocks that is functionally equivalent to the version above using itertools.groupby, without using the assignment operator anyhere, or anything from a module. This is a puzzle I found impossible last night but reasonably easy today. Like a lot of puzzles, looking at it in a slightly different way can make all the difference—advice which is often maddeningly useless while you’re trying to solve something.

The function, plus the version of findlongestrepeater I have above, should pass the following tests:

def test_contiguous_blocks():
    testgen = contiguous_blocks("googlee")
    assert testgen.next() == "g"
    assert testgen.next() == "oo"
    assert testgen.next() == "g"
    assert testgen.next() == "l"
    assert testgen.next() == "ee"
    assert [c for c in contiguous_blocks("google")] == ["g", "oo", "g", "l", "e"]
    assert [c for c in contiguous_blocks("googlee")] == [
        "g", "oo", "g", "l", "ee"]

def test_findlongestrepeater():
    assert findlongestrepeater("google") == "oo"
    assert findlongestrepeater("gooooogle") == "ooooo"
    assert findlongestrepeater("eeffi") == "ee"
    assert findlongestrepeater("abcdef") == "a"
    assert findlongestrepeater("eeffeeeffeeeee") == "eeeee"

WARNING, SOLUTION

Don’t read past here if you want to solve it yourself, obviously.

The key insight that helped me get it (which seems utterly obvious now, of course) is that you’re not limited to the current and next (and/or previous) characters. I’d been approaching it with logic like this:

Check the character to see if it’s different from the next character.
- If it is, return it.
- If it isn’t, pass.
But also (somehow) store it so that you know what the current sequence of repeated characters is, and when you next return something, return that sequence instead of just the current character.

I couldn’t figure out how to make that work, and it may be impossible to get Python to do that without using variables (although maybe I just missed the trick).

However, that wasn’t the right way to look at it. With access to the index of the current character, you can get not only the next character, but also string up to that point. That was the realization: the logic should be as follows:

Check the character to see if it’s different from the next character.
- If it isn’t, pass.
- If it is, return all the characters in the string up to that point that are the same as the character.

As I said, that seems awfully obvious now…

This is the solution I came up with:

def contiguous_blocks(string):
    def inarow(row, string):
        if not string or row[0] != string[0]:
            return row
        return inarow(u"".join([row, string[0]]), string[1:])

    for i,c in enumerate(string):
        if i >= (len(string) -1) or c != string[i+1]:
            yield inarow(c, string[:i][::-1])

So, if the character is different from the next one, or you’ve run out of string, yield the result of passing the character and the (reversed) string up to that point to inarow. Otherwise, if the character is the same as the next character—do nothing.

inarow doesn’t have to track anything but the characters until a different character shows up or you run out of string, so it returns what it has at that point if either of those happen, otherwise it calls itself with a new row of characters (the current one tacked onto the end) and the rest of the string.

The combination of find_longest_repeater and contiguous_blocks seems like a fairly elegant solution. Not only does it come in at two lines shorter than my version from Tuesday, but it also has the benefit of being more neatly divided in logical terms. (Which is something I wouldn’t have seen without the suggestion that the determination of contiguous blocks could be a discrete function.)

Tags: coding, python

Minor Foray into Functional Programming

Tadhg — Tue, 23 Feb 2010 23:56:05 +0000

Last night a friend asked me what functional programming was, and as part of my answer I decided to rewrite a trivial program in the functional style to see what it was like. I did this in Python without using the functional module.

The program finds the longest sequence of repeated characters in a string; if there’s a tie, it returns the first sequence of that length. My (imperative) version was six lines of Python:

def findlongestrepeater(string):
    longest, curr = "", ""
    for c in string:
        curr = "".join([curr, c]) if c in curr else c
        longest = curr if len(curr) > len(longest) else longest
    return longest

There are doubtless better ways to do it, but this is relatively succinct and clear. I can’t say the same about the functional version I came up with.

With my attempt at this example, “functional programming” basically meant “not using the assignment operator”.

I couldn’t see any way to do it without recursion, and it came out looking like a recursive decision tree:

def findlongestrepeater(string):
    def flr(lr, curseq, s):
        return lr if len(s) == 0 else character_match(lr, curseq, s)

    def character_match(lr, curseq, s):
        if s[0] == curseq[0]:
            return sequence_length(lr, u"".join([s[0],curseq]), s)
        else:
            return flr(lr, s[0], s[1:])

    def sequence_length(lr, newseq, s):
        if len(newseq) > len(lr):
            return flr(newseq, newseq, s[1:])
        else:
            return flr(lr, newseq, s[1:])

    return flr(string[0], string[0], string[1:])

flr takes arguments of the longest repeating sequence found so far, the current sequence, and the rest of the string. We kick it off by passing it the first character as the first two arguments, and then the rest of the string after the first character.

If the rest of string has a length of zero, we’ve reached the end of the string and flr returns the longest repeating sequence it was passed.

Otherwise, it calls character_match, which checks whether or not the current sequence matches the first character in the string. If not, it goes back to flr passing in the next character and the rest of the string; if so, it goes on to sequence_length passing in the longest repeater, the new sequence (which is the first character in the string plus the current sequence), and the string. sequence_length will then call flr with either the new sequence as the longest repeater, or the current longest repeater as the longest repeater.

It was pretty hard for me to approach the problem this way. It took me about forty minutes to get a working version, and another fifteen to clean it up (the first version was big jumble of conditionals). It’s also much harder to troubleshoot than the imperative version, at least for me. Recursion isn’t that hard to handle, but I really wanted to use variables just to make things clearer—for example, in character_match, I really want to replace u"".join([s[0],curseq]) with newseq = u"".join([s[0],curseq]) and then call sequence_length with newseq as the second argument, purely to make what’s going on more explicit.

It was definitely an interesting experiment, and in future I might try some exercises where I have to take functions and rewrite them without using assignment. I’m sure there are cases where this makes sense; I’m also sure that the above script would be a lot easier to deal with in a proper functional language (or perhaps using the functional module).

After doing that last night, this morning I came across an introduction to functional programming with Python.

Tags: coding, python

Some Vim Script Implementation, Testing, and Hackery

Tadhg — Wed, 17 Feb 2010 07:50:47 +0000

As a result of my porting over jEdit (Jython) macros to Vim, I now have a fair amount of (Python) Vim scripts, and have learned some things about how to set up those scripts. I’ll go through some of that below, and hopefully other people writing Python scripts for Vim will find it useful.

In jEdit, you invoke macros either by selecting them from a menu (clearly unacceptable) or by opening the Action Bar (Ctrl-Enter, for me) and then typing the name of the macro, where “name” means the non-extension part of the filename. I put my macro files went in a subdirectory of the macros directory, and there were namespace issues—all macro names had to be unique, and also couldn’t match any internal jEdit actions. Furthermore, you couldn’t pass arguments to them. Despite those issues, it was a powerful and quick piece of functionality, and I wanted to make Vim at least match it.

You can put Python directly into Vim script files, but I wanted to do that as minimally as possible. I added a Vim script that would (essentially) include a Python file, and then I had that Python file import from other Python files as necessary. The Python file includes a class, TBase, that contains the functions I want to run; as much code as possible is kept out of those functions and is in other files—files which ideally are individually testable.

To get to those functions from within Vim, I did this:

if filereadable($HOME."/.vim/plugin/tadhg/tadhg.py")
    pyfile $HOME/.vim/plugin/tadhg/tadhg.py
endif

" Create the commands pointing the Python in tadhg.py:
if !exists(":T")
  command! -range -nargs=+ T python tadhgbase = TBase('rs= rf='); tadhgbase('')
endif

nmap  :T

That code goes in a Vim script that goes in ~/.vim/plugins/tadhg/; it’s loaded when Vim is. This means that the code in tadhg.py is evaluated at Vim start, so the TBase class is available. Furthermore the user Ex mode command “T” creates a new instance of TBase and then calls it with whatever the user enters after “T”. Finally, Command-Enter is mapped to enter Ex mode and type “T ”, which is quite close indeed to the jEdit functionality I had, but without the disadvantages. I have much more control over the namespace, and I can pass arguments.

When the T command is invoked, a new TBase instance is created and it gets passed the range start and range end line numbers from Vim, which is pretty important for some of the functions. Then the new instance is called, with the argument of whatever was typed after “T ”.

This is the code for handling that latter aspect:

def __call__(self, argstring):
    """
    The first arg is the command; the rest are arguments to that command.
    """
    args = argstring.split(u" ")

    #Need to extract the keyword args out of args
    kwargs = dict([(arg.split("=")[0], arg.split("=")[1]) for arg in s.split(" ") if "=" in arg])
    args = [arg for arg in s.split(" ") if "=" not in arg]

    if args and args[0] in self.__class__.__dict__.keys():
        import types
        f = self.__class__.__dict__[args[0]]
        if type(f) == types.FunctionType:
            if args[1:]:
                f(self, *args[1:])
            else:
                f(self)
    else:
        print "no command by that name"

In other words, if the first (non-keyword) argument matches the name of a function in the class, run it and pass the rest of the (non-keyword) arguments to it. If I ever need keyword arguments as well, I’ll add the ability to pass them along too.

In order to actually manipulate Vim from these scripts, you have to use import vim. When I started writing this, I wasn’t sure how I’d test that, and I tried to keep the meat of the code in other files, e.g.:

def mpc(self):
    """
    Count the words, then insert them into the document's word count line.
    """
    from write_wordcount import WriteWordcount
    wwc = WriteWordcount(tadhgbase)
    wwc.vim_main(vim)

(Note that I use import sys; sys.path.append(myscriptdir) at the top of the file to make these imports work.)

One of the advantages of this construction is that WriteWordCount can theoretically be used as a standalone script, on any file, and also that its vim_main method takes vim as an argument. For testing purposes, I had to created a MockVim class, which currently looks like this:

class MockVimBuffer(object):

    def __init__(self):
        self.lines = []

    def __contains__(self, item):
        return item in self.lines

    def __iter__(self):
        return self.lines.__iter__()

    def __getitem__(self, index):
        return self.lines.__getitem__(index)

    def __setitem__(self, index, value):
        return self.lines.__setitem__(index, value)

class MockVimCurrent(object):

    def __init__(self):
        self.__dict__["buffer"] = MockVimBuffer()

    def __setattr__(self, attr, value):
        if attr == "buffer":
            self.__dict__["buffer"].lines = value
        else:
            self.__dict__[attr] = value

class MockVim(object):

    def __init__(self):
        self.current = MockVimCurrent()
        self.commands = []

    def eval(self, command):
        command_list = {
            "&ft": lambda: self.rawmodes,
            "exists('b:TotalWordCount')": lambda: False,
            "tvar": lambda: self.tvar,

        }
        return command_list.get(command, lambda: "")()

    def command(self, command):
        self.commands.append(command)

The eval method in MockVim simply returns whatever was appropriate for various tests I was running; I should change it to return self.command_list.get(command, lambda: "")() instead, and make command_list an instance variable so that the various tests can manipulate it as needed instead of having to put specific commands in the base code.

Because the methods in TBase pass vim along to the instances or methods of the modules they import, testing those modules is relatively simple: in the test, you create a new MockVim object, call it vim, and then pass it along instead.

So everything is great (and testable). Unless, that is, you (meaning me) lapse from full test-driven development discipline and bits and pieces of functionality creep into TBase. Because TBase has that import vim line at the top of it and running it when you’re not actually in Vim produces an ImportError.

The right way to deal with this is to refactor TBase to take vim as one of its initialization arguments, and then have it set that as an instance variable, and then pass that in its methods, while also altering the line in the parent Vim script to:

command! -range -nargs=+ T python import vim; tadhgbase = TBase(vim, 'rs= rf='); tadhgbase('')

That way TBase is much easier to test, since the fake vim can be passed in at test time. However, for a bunch of reasons I didn’t want to do that just yet, partly because it means some reasonably heavy refactoring of pieces of code that I use every day—without tests, because the whole issue here is that TBase isn’t testable. So I wanted a way to make it testable while altering as little of its existing code (which I know currently works, after all) as possible. This took me a while to figure out, and is rather hacky—but it works.

First, in the test file, do this:

from mockvim import MockVim
#Hack a mock Vim into the global namespace so that we can actually test:
import __builtin__
mv = MockVim()
__builtin__.mv = mv
from tadhg import TBase

Then alter the import vim line above the TBase class to instead read:

try:
    import vim
except ImportError:
    import __builtin__
    vim = __builtin__.mv

What this really does is hack a giant global variable, vim, into all of the Python that’s run after those lines in the test file. Normally doing this seems like a rather bad idea, but it’s also rather necessary to handle the less-than-ideal situation I’m dealing with. Once I have the test harness using this hackery running properly, I’ll feel a lot better about switching over to a better architecture.

Tags: coding, Jython, python, software, tech, text editing, Vim

tadhg.com » coding

Metaweb Acquired by Google

Related Posts

Unit Testing as Game

Related Posts

8-Bit Cities

Related Posts

iPad First Impressions: Consumption Machine

Related Posts

sabbatical.close()

Related Posts

Rather Better Than a Cover Letter

Related Posts

Improving a Python Word Counting Function

Related Posts

How to Solve It and Project Euler

Related Posts

Python Flatten/Concantenate Comparisons

Related Posts

Public Projects, Finishing Things, and bitbucket

Related Posts

Refactoring, Geeky Enjoyment, and Readability

Related Posts

Clearing Growl Notifications

Related Posts

2010 Goals: First Quarter Review

Related Posts

Giles Bowkett on Finding New Programming Jobs

Related Posts

A Little More Functional Programming

WARNING, SOLUTION

Related Posts

Minor Foray into Functional Programming

Related Posts

Some Vim Script Implementation, Testing, and Hackery

Related Posts