tadhg.com » coding

A Little More Functional Programming

Tadhg — Fri, 26 Feb 2010 07:55:51 +0000

After reading Dhananjay Nene’s comment on my post about a functional style approach to the “find longest repeater” problem, I decided to follow the line from that comment and divide the program into functions for finding the longest contiguous block and then for comparing the blocks. Naturally, I wanted to do this without using any variables…

It wasn’t too hard to write a variable-free function to return a list of blocks of letters from a string, e.g. continuous_blocks("anna") produces ["a", "nn", "a"]:

def contiguous_blocks(string):
    def listbuilder(blocks, char, rest):
        if not rest:
            return add_to_blocks(char, blocks)
        return listbuilder(add_to_blocks(char, blocks), rest[0], rest[1:])

    def add_to_blocks(char, blocks):
        if blocks and char in blocks[-1]:
            return blocks[:-1] + ["".join([blocks[-1], char])]
        return blocks+[char]

    return listbuilder([], string[0], string[1:])

And then findlongestrepeater becomes:

def findlongestrepeater(string):
    def compare_blocks(previous, current):
        return current if len(current)>len(previous) else previous

    return reduce(compare_blocks, contiguous_blocks(string))

Note that this approach provides a clear use for reduce, which is on the outs in Python these days.

A crucial difference between Dhananjay Nene’s version and mine, however, is that mine returns a list and not a generator. So I decided to write a variable-free generator version… and just couldn’t do it. I don’t know what it was, but I kept getting hung up on trying to track the previous, current, and next characters, and to do that without variables while trying to skip at the right times in the for loop just seemed impossible. While writing this blog post, however, I figured it out. I had intended to put it up as a puzzle for other people, so I guess I’ll put my solution at the end. If you plan to try it, don’t scroll beyond the warning below.

There’s an easier way, too, which is simply to do this:

def contiguous_blocks(string):
    from itertools import groupby
    for k,g in groupby(string):
        yield u"".join(list(g))

I found that last night after being unable to bend my mind in the right way, because I was sure that this problem couldn’t be so rare that itertools wouldn’t have something to handle it.

However, I felt like that was cheating… so I kept trying to build my own variable-free generator.

Because of the fact that reduce isn’t going to be in Python 3, I decided I’d write a version of it that would work as its replacement for this problem. It didn’t seem too hard to come up with this:

def reducereplacement(func, generator):
    def runfunc(product, gen):
        try:
            return runfunc(func(product, gen.next()), gen)
        except StopIteration:
            return product

    return runfunc([], generator)

This only works with generators and not with lists, unlike the real reduce. There’s no reason to use this, either, given that reduce is in Python 2 and will be in the functools module for Python 3; I just wanted to try it as an experiment.

So, to restate the puzzle problem:

Write a version of contiguous_blocks that is functionally equivalent to the version above using itertools.groupby, without using the assignment operator anyhere, or anything from a module. This is a puzzle I found impossible last night but reasonably easy today. Like a lot of puzzles, looking at it in a slightly different way can make all the difference—advice which is often maddeningly useless while you’re trying to solve something.

The function, plus the version of findlongestrepeater I have above, should pass the following tests:

def test_contiguous_blocks():
    testgen = contiguous_blocks("googlee")
    assert testgen.next() == "g"
    assert testgen.next() == "oo"
    assert testgen.next() == "g"
    assert testgen.next() == "l"
    assert testgen.next() == "ee"
    assert [c for c in contiguous_blocks("google")] == ["g", "oo", "g", "l", "e"]
    assert [c for c in contiguous_blocks("googlee")] == [
        "g", "oo", "g", "l", "ee"]

def test_findlongestrepeater():
    assert findlongestrepeater("google") == "oo"
    assert findlongestrepeater("gooooogle") == "ooooo"
    assert findlongestrepeater("eeffi") == "ee"
    assert findlongestrepeater("abcdef") == "a"
    assert findlongestrepeater("eeffeeeffeeeee") == "eeeee"

WARNING, SOLUTION

Don’t read past here if you want to solve it yourself, obviously.

The key insight that helped me get it (which seems utterly obvious now, of course) is that you’re not limited to the current and next (and/or previous) characters. I’d been approaching it with logic like this:

Check the character to see if it’s different from the next character.
- If it is, return it.
- If it isn’t, pass.
But also (somehow) store it so that you know what the current sequence of repeated characters is, and when you next return something, return that sequence instead of just the current character.

I couldn’t figure out how to make that work, and it may be impossible to get Python to do that without using variables (although maybe I just missed the trick).

However, that wasn’t the right way to look at it. With access to the index of the current character, you can get not only the next character, but also string up to that point. That was the realization: the logic should be as follows:

Check the character to see if it’s different from the next character.
- If it isn’t, pass.
- If it is, return all the characters in the string up to that point that are the same as the character.

As I said, that seems awfully obvious now…

This is the solution I came up with:

def contiguous_blocks(string):
    def inarow(row, string):
        if not string or row[0] != string[0]:
            return row
        return inarow(u"".join([row, string[0]]), string[1:])

    for i,c in enumerate(string):
        if i >= (len(string) -1) or c != string[i+1]:
            yield inarow(c, string[:i][::-1])

So, if the character is different from the next one, or you’ve run out of string, yield the result of passing the character and the (reversed) string up to that point to inarow. Otherwise, if the character is the same as the next character—do nothing.

inarow doesn’t have to track anything but the characters until a different character shows up or you run out of string, so it returns what it has at that point if either of those happen, otherwise it calls itself with a new row of characters (the current one tacked onto the end) and the rest of the string.

The combination of find_longest_repeater and contiguous_blocks seems like a fairly elegant solution. Not only does it come in at two lines shorter than my version from Tuesday, but it also has the benefit of being more neatly divided in logical terms. (Which is something I wouldn’t have seen without the suggestion that the determination of contiguous blocks could be a discrete function.)

Tags: coding, python

Minor Foray into Functional Programming

Tadhg — Tue, 23 Feb 2010 23:56:05 +0000

Last night a friend asked me what functional programming was, and as part of my answer I decided to rewrite a trivial program in the functional style to see what it was like. I did this in Python without using the functional module.

The program finds the longest sequence of repeated characters in a string; if there’s a tie, it returns the first sequence of that length. My (imperative) version was six lines of Python:

def findlongestrepeater(string):
    longest, curr = "", ""
    for c in string:
        curr = "".join([curr, c]) if c in curr else c
        longest = curr if len(curr) > len(longest) else longest
    return longest

There are doubtless better ways to do it, but this is relatively succinct and clear. I can’t say the same about the functional version I came up with.

With my attempt at this example, “functional programming” basically meant “not using the assignment operator”.

I couldn’t see any way to do it without recursion, and it came out looking like a recursive decision tree:

def findlongestrepeater(string):
    def flr(lr, curseq, s):
        return lr if len(s) == 0 else character_match(lr, curseq, s)

    def character_match(lr, curseq, s):
        if s[0] == curseq[0]:
            return sequence_length(lr, u"".join([s[0],curseq]), s)
        else:
            return flr(lr, s[0], s[1:])

    def sequence_length(lr, newseq, s):
        if len(newseq) > len(lr):
            return flr(newseq, newseq, s[1:])
        else:
            return flr(lr, newseq, s[1:])

    return flr(string[0], string[0], string[1:])

flr takes arguments of the longest repeating sequence found so far, the current sequence, and the rest of the string. We kick it off by passing it the first character as the first two arguments, and then the rest of the string after the first character.

If the rest of string has a length of zero, we’ve reached the end of the string and flr returns the longest repeating sequence it was passed.

Otherwise, it calls character_match, which checks whether or not the current sequence matches the first character in the string. If not, it goes back to flr passing in the next character and the rest of the string; if so, it goes on to sequence_length passing in the longest repeater, the new sequence (which is the first character in the string plus the current sequence), and the string. sequence_length will then call flr with either the new sequence as the longest repeater, or the current longest repeater as the longest repeater.

It was pretty hard for me to approach the problem this way. It took me about forty minutes to get a working version, and another fifteen to clean it up (the first version was big jumble of conditionals). It’s also much harder to troubleshoot than the imperative version, at least for me. Recursion isn’t that hard to handle, but I really wanted to use variables just to make things clearer—for example, in character_match, I really want to replace u"".join([s[0],curseq]) with newseq = u"".join([s[0],curseq]) and then call sequence_length with newseq as the second argument, purely to make what’s going on more explicit.

It was definitely an interesting experiment, and in future I might try some exercises where I have to take functions and rewrite them without using assignment. I’m sure there are cases where this makes sense; I’m also sure that the above script would be a lot easier to deal with in a proper functional language (or perhaps using the functional module).

After doing that last night, this morning I came across an introduction to functional programming with Python.

Tags: coding, python

Some Vim Script Implementation, Testing, and Hackery

Tadhg — Wed, 17 Feb 2010 07:50:47 +0000

As a result of my porting over jEdit (Jython) macros to Vim, I now have a fair amount of (Python) Vim scripts, and have learned some things about how to set up those scripts. I’ll go through some of that below, and hopefully other people writing Python scripts for Vim will find it useful.

In jEdit, you invoke macros either by selecting them from a menu (clearly unacceptable) or by opening the Action Bar (Ctrl-Enter, for me) and then typing the name of the macro, where “name” means the non-extension part of the filename. I put my macro files went in a subdirectory of the macros directory, and there were namespace issues—all macro names had to be unique, and also couldn’t match any internal jEdit actions. Furthermore, you couldn’t pass arguments to them. Despite those issues, it was a powerful and quick piece of functionality, and I wanted to make Vim at least match it.

You can put Python directly into Vim script files, but I wanted to do that as minimally as possible. I added a Vim script that would (essentially) include a Python file, and then I had that Python file import from other Python files as necessary. The Python file includes a class, TBase, that contains the functions I want to run; as much code as possible is kept out of those functions and is in other files—files which ideally are individually testable.

To get to those functions from within Vim, I did this:

if filereadable($HOME."/.vim/plugin/tadhg/tadhg.py")
    pyfile $HOME/.vim/plugin/tadhg/tadhg.py
endif

" Create the commands pointing the Python in tadhg.py:
if !exists(":T")
  command! -range -nargs=+ T python tadhgbase = TBase('rs= rf='); tadhgbase('')
endif

nmap  :T

That code goes in a Vim script that goes in ~/.vim/plugins/tadhg/; it’s loaded when Vim is. This means that the code in tadhg.py is evaluated at Vim start, so the TBase class is available. Furthermore the user Ex mode command “T” creates a new instance of TBase and then calls it with whatever the user enters after “T”. Finally, Command-Enter is mapped to enter Ex mode and type “T ”, which is quite close indeed to the jEdit functionality I had, but without the disadvantages. I have much more control over the namespace, and I can pass arguments.

When the T command is invoked, a new TBase instance is created and it gets passed the range start and range end line numbers from Vim, which is pretty important for some of the functions. Then the new instance is called, with the argument of whatever was typed after “T ”.

This is the code for handling that latter aspect:

def __call__(self, argstring):
    """
    The first arg is the command; the rest are arguments to that command.
    """
    args = argstring.split(u" ")

    #Need to extract the keyword args out of args
    kwargs = dict([(arg.split("=")[0], arg.split("=")[1]) for arg in s.split(" ") if "=" in arg])
    args = [arg for arg in s.split(" ") if "=" not in arg]

    if args and args[0] in self.__class__.__dict__.keys():
        import types
        f = self.__class__.__dict__[args[0]]
        if type(f) == types.FunctionType:
            if args[1:]:
                f(self, *args[1:])
            else:
                f(self)
    else:
        print "no command by that name"

In other words, if the first (non-keyword) argument matches the name of a function in the class, run it and pass the rest of the (non-keyword) arguments to it. If I ever need keyword arguments as well, I’ll add the ability to pass them along too.

In order to actually manipulate Vim from these scripts, you have to use import vim. When I started writing this, I wasn’t sure how I’d test that, and I tried to keep the meat of the code in other files, e.g.:

def mpc(self):
    """
    Count the words, then insert them into the document's word count line.
    """
    from write_wordcount import WriteWordcount
    wwc = WriteWordcount(tadhgbase)
    wwc.vim_main(vim)

(Note that I use import sys; sys.path.append(myscriptdir) at the top of the file to make these imports work.)

One of the advantages of this construction is that WriteWordCount can theoretically be used as a standalone script, on any file, and also that its vim_main method takes vim as an argument. For testing purposes, I had to created a MockVim class, which currently looks like this:

class MockVimBuffer(object):

    def __init__(self):
        self.lines = []

    def __contains__(self, item):
        return item in self.lines

    def __iter__(self):
        return self.lines.__iter__()

    def __getitem__(self, index):
        return self.lines.__getitem__(index)

    def __setitem__(self, index, value):
        return self.lines.__setitem__(index, value)

class MockVimCurrent(object):

    def __init__(self):
        self.__dict__["buffer"] = MockVimBuffer()

    def __setattr__(self, attr, value):
        if attr == "buffer":
            self.__dict__["buffer"].lines = value
        else:
            self.__dict__[attr] = value

class MockVim(object):

    def __init__(self):
        self.current = MockVimCurrent()
        self.commands = []

    def eval(self, command):
        command_list = {
            "&ft": lambda: self.rawmodes,
            "exists('b:TotalWordCount')": lambda: False,
            "tvar": lambda: self.tvar,

        }
        return command_list.get(command, lambda: "")()

    def command(self, command):
        self.commands.append(command)

The eval method in MockVim simply returns whatever was appropriate for various tests I was running; I should change it to return self.command_list.get(command, lambda: "")() instead, and make command_list an instance variable so that the various tests can manipulate it as needed instead of having to put specific commands in the base code.

Because the methods in TBase pass vim along to the instances or methods of the modules they import, testing those modules is relatively simple: in the test, you create a new MockVim object, call it vim, and then pass it along instead.

So everything is great (and testable). Unless, that is, you (meaning me) lapse from full test-driven development discipline and bits and pieces of functionality creep into TBase. Because TBase has that import vim line at the top of it and running it when you’re not actually in Vim produces an ImportError.

The right way to deal with this is to refactor TBase to take vim as one of its initialization arguments, and then have it set that as an instance variable, and then pass that in its methods, while also altering the line in the parent Vim script to:

command! -range -nargs=+ T python import vim; tadhgbase = TBase(vim, 'rs= rf='); tadhgbase('')

That way TBase is much easier to test, since the fake vim can be passed in at test time. However, for a bunch of reasons I didn’t want to do that just yet, partly because it means some reasonably heavy refactoring of pieces of code that I use every day—without tests, because the whole issue here is that TBase isn’t testable. So I wanted a way to make it testable while altering as little of its existing code (which I know currently works, after all) as possible. This took me a while to figure out, and is rather hacky—but it works.

First, in the test file, do this:

from mockvim import MockVim
#Hack a mock Vim into the global namespace so that we can actually test:
import __builtin__
mv = MockVim()
__builtin__.mv = mv
from tadhg import TBase

Then alter the import vim line above the TBase class to instead read:

try:
    import vim
except ImportError:
    import __builtin__
    vim = __builtin__.mv

What this really does is hack a giant global variable, vim, into all of the Python that’s run after those lines in the test file. Normally doing this seems like a rather bad idea, but it’s also rather necessary to handle the less-than-ideal situation I’m dealing with. Once I have the test harness using this hackery running properly, I’ll feel a lot better about switching over to a better architecture.

Tags: coding, Jython, python, software, tech, text editing, Vim

First Post With Vim

Tadhg — Mon, 15 Feb 2010 04:05:48 +0000

Over the last couple of weeks I’ve been hacking away on scripts to customize Vim, replicating the scripts I made for jEdit. I’m more or less done, and this blog post is being written in MacVim. This hopefully means that when I’m done with it I’ll be able to publish it from within Vim, the same as with jEdit.

Writing scripts for Vim has been rather different from writing them for jEdit. jEdit is an editor with a robust API; Vim is an editor that kind of gives the user access to the API all the time, via keyboard commands, and scripting for it often consists of simply passing keystrokes. As an example, this is the Jython code for getting the current selection in a jEdit macro:

text = init.textArea.getSelectedText()

This is the Python for getting the current visual selection in Vim:

self.vim.command('normal gv"wygv')
self.vim.command("let tvar = getreg(’t’)")
text = unicode(self.vim.eval("tvar"), "utf-8")

From a programming perspective, jEdit is clearly a lot nicer to deal with.

I have two primary reasons for making the switch:

Future proofing: I couldn’t switch to the various pre-release versions of jEdit 4.3 because I couldn’t get my Jython scripts to work with them, and that made me worry that there simply weren’t enough people scripting jEdit in Python to make it likely that the plugin allowing Jython macros would be supported. Given how useful my scripts are, that would have trapped me with jEdit 4.2, or some future version, if support for JythonInterpreter flagged. Vim, on the other hand, seems to have a lot of people scripting for it in Python, and I don’t see Python support simply disappearing anytime soon with a new release.
Power/speed: I’m someone who’s quite inclined towards keyboard shortcuts in all applications, and this means Vim should be a highly efficient environment.
Mentality: Related to the previous point, I think that being able to manipulate text more powerfully will reinforce the approach of scripting as many steps as possible, rather than doing them (no matter how efficiently) manually. Even over the last couple of weeks, I’ve noticed that I’m more inclined to write scripts than to put up with various manual steps in my daily workflow.

So far I’ve mainly worked to replicate my jEdit setup in Vim, with a few small improvements here and there; the largest one is probably the live word count, which means that I can glance at the status line to see the current word count instead of having to select the text to be counted and then invoke the word counter. I’ve also tweaked it so it only counts the relevant parts of the document—for example, in this blog post, it doesn’t count the title or the datestamp.

I don’t intend to abandon jEdit, and have a feeling that I’ll use it for various things here and there, but I’ve switched over the Vim for the majority of my writing and programming.

Tags: coding, Jython, python, software, tech, text editing, Vim, writing

The Python Challenge

Tadhg — Thu, 04 Feb 2010 23:18:35 +0000

The Python Challenge seems like a good way to have fun with Python through puzzle-solving. As with all riddles, it’s important to read the questions carefully…

Tags: coding, python

jQuery Project

Tadhg — Fri, 22 Jan 2010 07:36:54 +0000

I’ve been a big fan of jQuery more or less since it came out, and I’m happy to see the launch of The jQuery Project. I’ve used jQueryUI a couple of times and find it fairly useful; I haven’t tried Sizzle yet but it looks great for situations where you’re really concerned about keeping file sizes low but need decent CSS selector support; and I wish QUnit had been around when I was writing a lot of client-side code.

Tags: coding, JavaScript, web-development

Better Word Count in Vim

Tadhg — Mon, 18 Jan 2010 07:40:14 +0000

I’m currently trying out Vim (again), and have made more progress this time, mainly due to Seth’s help. The key things that have made it better:

:set hidden. Absolutely critical, this. Stops Vim from complaining when you try to switch buffers and your current buffer has unsaved changes.
bufexplorer. Makes switching buffers a lot easier.
A better Python syntax file. I didn’t like the defaults.
My own indentation and syntax files for reStructuredText.

Really, though, the key first one was :set hidden. Before that I felt that I had completely misunderstood Vim’s file management model.

Once I got the syntax highlighting to a reasonable state, I ported my word count macro over to Vim. This wasn’t too hard after the inevitable character encoding problems. For working within Vim scripts, I strongly suggest using the following:

ulines = [unicode(line, "utf-8") for line in vim.current.buffer]

(Assuming your Vim encoding is set to UTF-8, of course.)

The Vim version of my script isn’t as versatile yet as the jEdit version, because I don’t know how to make it work with only selected lines (something that’s easy in jEdit). Apart from that, though, it seems to work quite well; the next version of it might do “live” word count in the status bar.

I’m currently writing this in Vim, or kind of: the actual writing is in Vim, but the creation of the file and the template, the automated expansion of various reStructuredText entities, output to reStructuredText, and the publication, are in jEdit as I haven’t ported those over yet.

The script:

function! WordCount()
python << EOF
import re
import vim

class WordCounter(object):
    """
    Vim script for better word count.
    """

    LINE_SEPARATORS = (
        "\r",
        "\n"
    )

    WORD_SEPARATORS = (
        " ",        # space
        "\t",       # tab
        "/",        # slash
        "&",        # ampersand
        ’"’,        # double quotation mark, straight
        u"\u201C",  # double quotation mark, left
        u"\u201D",  # double quotation mark, right
        u"\u2018",  # single quotation mark, left
        u"\u2013",  # en dash
        u"\u2014",  # em dash
        ">",        # greater than symbol
        "<",        # less than symbol
        "+",        # plus
        "=",        # equals
    )

    REPEATER_SEPARATORS = (
    #These are only separators if they’re present consecutively, e.g. -- or ..
        "-",
        "."
    )

    IGNORE = (
    #Not separators per se, but should not be treated as word content
        "’",        # single quotation mark, straight
        u"\u2019",  # single quotation mark, right
        "(",        # left parenthesis
        ")",        # right parenthesis
        "[",        # left bracket
        "]",        # right bracket
        "{",        # left curly bracket
        "}",        # right curly bracket
        "|",        # bar
        "-",        # hyphen
        "#",        # hash mark
        ".",        # period
        "_",        # underscore
        "`",        # backtick
        "\\",        # backslash
    )

    def word_count(self):

        ulines = [unicode(line, "utf-8") for line in vim.current.buffer]
        text = u"\n".join(ulines)

        chars = len(text) #Pretty sure I want the actual char count, not the adjusted char count.

        text = self.remove_directives(text)
        text = self.adjust_for_rest(text)
        words, lines = self.count_words(text)

        print "chars: %s, words: %s, lines: %s" % (chars, words, lines)

    def remove_directives(self, text):
        textlines = text.split("\n")
        newlines = []
        comment = re.compile(r"[ ]*\.\. [a-zA-Z0-9_\|]")
        argument = re.compile(r"    :[^\:]*:")
        for line in textlines:
            if not comment.match(line) and not argument.match(line):
                newlines.append(line)
        return "\n".join(newlines)

    def adjust_for_rest(self, text):
        """
            Go through each of the special cases for reST.
        """
        text = self.rest_adjust_pipe_space(text)

        return text

    def rest_adjust_pipe_space(self, text):
        """
            Special-case "|\ " to make sure e.g. "|Hypnotic Specter|\ s"
            doesn’t get counted as three words.

            |Incinerate|\ s |Hypnotic Specter|\ —|Hypnotic Specter|\ s
            The above line should be counted as five words.
        """
        spacere = re.compile(r"\|\\ ([^ ]{1})")
        finds = spacere.findall(text)
        text = spacere.sub("|\g<1>", text)
        return text

    def count_words(self, text):
        words, lines = 0, 1
        #go through the text character by character:
        word, previous_character = 0, None
        for character in text:
            if character in (self.LINE_SEPARATORS + self.WORD_SEPARATORS) or (character in self.REPEATER_SEPARATORS and previous_character in self.REPEATER_SEPARATORS):
                #it’s a separator
                word = 0
                if character in (self.LINE_SEPARATORS):
                    lines = lines + 1
            elif character in (self.IGNORE):
                pass
            else:
                #it’s part of a word.
                if not word:
                    words = words + 1
                    word = 1
            previous_character = character

        return (words, lines)

WordCounter().word_count()
EOF
endfunction

if !exists(":WW")
  command! WW  :call WordCount()
endif

Tags: coding, Jython, python, software, tech, text editing, Vim, writing

Code Katas

Tadhg — Sat, 16 Jan 2010 01:26:06 +0000

I really like this idea from Dave Thomas: code katas, small pieces of programming practice involving some repetition. I came by it via Katacasts, a collection of screencasts of people doing the katas. I particularly recommend Gary Bernardt’s String Calculator in Python and Vim. (Which has inspired me to try once again to get past the vim file management issues I have.)

Tags: coding, python, Vim

WordPress 2.9 Upgrade

Tadhg — Sun, 10 Jan 2010 23:56:30 +0000

I upgraded this blog to WordPress 2.9 today, and it appeared to go entirely smoothly. Please let me know if you notice any breakage.

Tags: Blog, coding, PHP, tech, web-development, WordPress

Python Coding Exercise: Nested Dictionaries

Tadhg — Sat, 09 Jan 2010 07:54:56 +0000

I’ve been looking at a bunch of coding exercises recently, including the demo for Codility, and recalled an exercise that I came up with as an interview question. It’s not incredibly difficult, but strikes me as a good “real-world” exercise—it’s based on a task I had to perform while working on the discuss functionality for freebase.com.

You have a list of objects (the objects can be dictionaries). In addition to other, irrelevant, properties, they all have id and parent keys (or properties). The parent key contains the id of the object’s parent. There are no recursive parent relationships (two objects cannot be each other’s parents), and there’s only one parent per object.

The goal is to go from this flat list of objects to a list of nexted objects, where each object has a child key (or property), and that contains not a list of ids but the child objects themselves, each of which also has a child property containing its child objects, and so on. Each dict should still show up only once in the entire structure (each only shows up once in the original list).

I think it gets slightly more difficult if you start with the objects having their parent relationships laid out at the start instead of the child relationships, but it’s still a similar problem.

I might post an answer to this over the next couple of weeks, and might also dig up my old solution to the problem to compare against how I’d do it now (if there’s much difference).

Tags: coding, python

Python Optimization Tips

Tadhg — Tue, 05 Jan 2010 14:34:28 +0000

I came across these on Hacker News recently, and think they’re worth calling out: Python Speed Performance Tips.

This comment is also worth reading.

There’s plenty of food for thought in those. They both encourage the use of list comprehensions, which I happen to like quite a lot.

On that note, I was recently asked about doing some contrived simple programming problems in list comprehensions.

Find the overlapping elements in two lists:
```
[item for item in list1 if item in list2]
```
This can also be done using sets:
```
set(list1).intersection(list2)
```
Find the first repeating character in a string:
```
[char for char in s if s.count(char)>1][0]
```
Alec points out a far superior method in the comments:
```
next(c for i,c in enumerate(s) if c in islice(s, i+1)
```
Reverse a string (couldn’t do this in less than 3 lines until I remembered list slices):
```
s[::-1]
```

Tags: coding, python

2010 Goals

Tadhg — Fri, 01 Jan 2010 16:53:36 +0000

Happy New Year!

Once again, my goals for the coming year.

Some of these recur from last year, but that’s okay. They’re not really in any order.

Write at least one draft of the fantasy novel.
Revive sfmagic.org. Many of my MTG-playing friends simply don’t believe I’ll ever do this one, but I’m still determined to make it happen, and right now there are fewer obstacles to doing it than there have been in years.
Celebrate my birthday. A strange-sounding resolution, perhaps, but I haven’t actually done any significant celebration of my birthday for a couple of years, and that’s just dumb. So this year I’m going to do something. I might need help figuring that something out, we’ll see.
Participate in the CrossFit sectionals. I don’t expect to qualify, or even come particularly close to qualifying, but I want to take part anyway to see what it’s like, and to give myself a training target. Given that at the end of March last year I was in fairly poor physical condition, competing in them at all will be an achievement.
I have some miscellaneous fitness-related goals I’ll lump together: break three hundred pounds on the deadlift, get to ten dead-hang pullups, get my shoulders strong enough to safely do kipping pullups again and then get to thirty of those, break 23:00 for the 5K. Just because it’s the first day of the year and I think I should set one of these that’s a lot further away for me, I’ll add this: do “Murph” in under 50:00.
Finish The Annotated Fantasy Bedtime Hour. Just five episodes to write up, I simply need to get down to it.
Run a roleplaying campaign. I haven’t done this in fifteen years, and it’s time to start again. This already looks like it’s going to happen, but starting and finishing are separate things… this goal is to run at least one story arc, and to make it good (but not worry about it being perfect).
I don’t like having amorphous goals, but this last one is going in regardless. I need to do something about building a sense of community for myself. I’m not totally sure what this means, and it’s not that I utterly lack a sense of community now, but it’s not quite where I want it, and I need to figure out what it is I want to change and how to do it. I have a post kind-of in my head about this, which I’ll hopefully get down in the next few weeks.

That seems like a good set. I also want to read 80 books, but for some reason this year that doesn’t feel like a goal in the same way as the above.

Tags: coding, community, CrossFit, exercise, games, goals, MTG, personal, reading, roleplaying, web-development, writing

CrossFit and Coding (and Meat)

Tadhg — Thu, 31 Dec 2009 18:28:59 +0000

In 2009 I achieved some significant things that weren’t on my list of goals, although they’re not concrete achievements in the same sense.

One of them was starting CrossFit—and sticking with it. Prior to this year, I had never been particularly fit. I’d never been horribly unfit, either. I’d always wanted to be fitter, and it’s a fairly common theme in my morning pages throug their eleven years—wanting to start some exercise routine or other, wanting to figure out something that would really work. Many of the things that I tried would have worked, but I never stuck with them for too long. Even when I had physical activities I really enjoyed (e.g. tennis), I didn’t get into enough of a routine with them for that to have a really positive impact on my fitness.

It was something I was quite frustrated with, in a low-level way, for years. As with many things, the gaps between where I was and where I wanted to be, and between my intellectual motivation and my realized action, were wide and extremely frustrating to me. I was somewhat resigned to this continuing, and to perpetually going through patches of being slightly more fit but never truly getting too far.

CrossFit has changed that rather significantly. Not only am I probably the fittest I’ve been in my life, but both my determination to keep going and my confidence that I will are strong. The approach to fitness that I’ve been following for the last six months or so is one that I intend to stick with for the rest of my life, with the confidence that in terms of major features it is the right way for me. Those are all new states of mind for me, and I’m happy about them.

I spend a lot of my time using a computer. My profession, my writing, and many of my hobbies all involve computer use, and this has been true for years. I’ve always been rather interested in optimizing my software environment, and tend to quickly pick up shortcut keys and the like. At the same time, I’ve been progressing as a programmer. The last few years in particular have been challenging in a good way, and I’ve learned quite a lot. This year, I finally put these things together and wrote code to help my daily computer use in a serious way.

Python, a language that really helps me get things done rather than getting in my way, was a big part of this. The work I did this year to build a reStructuredText–WordPress blogging workflow, my RTF/Word–reStructuredText toolchain, and a better word count for jEdit, all constitute a positive shift in my attitude to programming. I’d always thought of programming as a useful tool (obviously), but this increased willingness to use it to directly alter my working environment has put me in a different place in regard to it—a place that I’m happy to be in.

Finally, meat. Not an achievement like the prior two, and whether it’s an unequivocally positive thing like them remains to be seen, but I have felt better since resuming meat-eating this year, after almost ten years of vegetarianism.

Tags: coding, CrossFit, diet, exercise, food, health, personal, python

Some Python Tips and Tricks

Tadhg — Mon, 21 Dec 2009 13:08:53 +0000

Python Tips, Tricks, and Hacks at Siafoo is an excellent overview of useful Python knowledge. I was familiar with most of it but still think it’s worth reading over. I did learn a couple of new things, too.

“Burmese python 2” by Tambako the Jaguar

I’ve found myself wanting the equivalent of list comprehensions (e.g. newlist = [do_something_to(item) for item in oldlist]) for dictionaries quite a few times, but never spent the time to figure out how to do that. The article does—you can use dict on a list of tuples that you’ve generated in a list comprehension in combination with dict.iteritems().

The first piece of new knowledge for me was that you can create new dictionaries not just with keyword arguments (d = dict(key1="value1", key2="value2")) and curly braces (d = {"key1": "value1", "key2": "value2"}) but also with a list of tuples of key/value pairs:

d = dict([("key1", "value1"), ("key2", "value2")])

That method of dictionary construction isn’t that useful… unless you’re dealing with large numbers of keys, or you’re moving between lists and dictionaries. You can go the other way, from a dictionary to a list of tuples, using dict.items() or dict.iteritems(). So here’s how to remove keys with empty values from a dictionary:

d = dict([key, value] for key, value in d.iteritems() if value)

That seems quite readable to me, and it’s definitely cleaner than the for and if statement approach that I used previously.

I also didn’t know about any and all, which can be used to evaluate whether any or all items in a list match a condition.

This next trick doesn’t come from the article, but was inspired by it: how to perform an operation on a specific set of keys in a dictionary only if those keys are in the dictionary. The obvious way:

for vehicle in ("planes", "trains"):
    if d.has_key(vehicle):
        d[vehicle] = d[vehicle].split(",")

There’s nothing wrong with that way, but I rather like being able to do this:

for vehicle in set(("planes", "trains")).intersection(d.keys()):
    d[vehicle] = d[vehicle].split(",")

I suspect that there are all kinds of operations that could be streamlined using various set functions.

(The last example could be reduced to a one-liner, but at that point we’re really losing readability:

d = dict(map(lambda (k,v): (k, d[k].split(",")) if k in ("planes", "trains") else (k, d[k]), d.iteritems()))

and it just doesn’t seem worth it.)

Tags: coding, python

GPS, URLs, Math, Python, Featuritis

Tadhg — Wed, 09 Dec 2009 07:53:21 +0000

Earlier this evening Gever suggested a service dedicated to shortening URLs that had geolocation data in them. My immediate responses were that a) this was a great idea, and b) that I wanted the shortened URLs to still be human-readable in some sense—specifically, I wanted a person to be able to look at two URLs returned by this service and have some idea of how close to each other they were.

Gever hadn’t had that feature in mind at all, and so I had added a major feature as soon as I encountered this idea for a service.

We were at PyWebSF, so there were a number of programmers present, all of whom liked the idea, and all of whom appeared to have entirely different approaches to implementing it. Quadtrees and octrees were mentioned by the more CS-minded, but my approach was more basic (no pun intended): I thought it would be possible simply to encode the numbers in some higher base system and effectively shorten them that way.

I started with base 64, and looked at Base64 and at the relatively easy Python implementation of an encoding system using it, but found that I didn’t like it, for a few reasons. The main one was that the encoding just wasn’t that friendly: incrementing by one wasn’t that trivial, for example, and to me that made comparison of numbers difficult. I decided that the Base64 approach was clearly not designed with human readability in mind, and that one that was would map to base 10 for the numbers 0–9, then go to lowercase letters a–z, with a being 10, b 11 … z 35, and then uppercase letters A-Z, with A being 36, B 37 … Z 61. Then I tacked on _ for 62 and * for 63, for no particular reason. Apart from the last two, my choices were based on what I felt most people would grasp easily: numbers were their own values, uppercase letters were more than lowercase letter, and letters progress in alphabetical order.

Of course, that meant implementing my own system for the conversion, which I did in likely inefficient manner using Python dicts and some math for converting base systems in rather hacky fashion.

Given that I wanted the coordinates to remain somewhat readable, I decided to leave the integer parts of them alone. So the coordinates 53.10680448323265, -9.580078125 would still have 53 and 9—those numbers are always small enough that there’s not much point changing them to something else. That just left me with the task of compressing the numerals after the decimal point.

I had to account for the possibility of a string of zeroes, and chose a rude hack for this: the first numeral in my compressed number would always represent how many zeroes there were after the decimal point. (I intend to only deal with about 18 digits after the point, so there’s no chance of there being more than 63 zeroes in any data I’m planning to accept.) This means an extra digit in all cases where there are no zeroes, which is unfortunate.

After the zeroes are handled, I convert the rest to a base 64 number—which is really a string composed of the characters [0-9a-zA-Z_*]—using the method of storing the modulo 64 of the number and then dividing by 64 and repeating. I had to look up this method because I’d entirely forgotten how to convert from different base systems.

I got this working, and it converts e.g.

53.10680448323265, -9.580078125

to:

53.02rqYl0X1, -9.0yAQEJ

Unfortunately, I’m not that impressed with those results. Yes, we get from 17 to 12 and 12 to 9 characters, but I’m not convinced that’s sufficient progress to make the whole thing worth it.

At this point a certain degree of madness crept into the proceedings.

Clearly, the problem wasn’t with the overall approach, but with the disgraceful paucity of symbols. 64 just wasn’t a large enough base. Using a larger base would ensure fewer characters to represent large numbers. The only problem here is that while progressing from zero to 63 with [0-9a-zA-Z_*] makes at least some logical sense to most people familiar with the English alphabet, how to go beyond that?

I started looking at larger character sets. Unicode can be supported in URLs (e.g. http://?.ws/??), so why not start using whatever Unicode symbols make sense in a higher base system?

I thought that base 128 would be ideal, but ran into immediate problems. Unfortunately, there doesn’t seem to be any single additional feature or accent that’s available for all of the letters of the English alphabet—I looked for dots, macrons, umlauts, overlines, underlines… I even looked at the possibility of upside-down letters, but not all of them are supported. If they were, I might have pursued that, but there’s an additional problem: they don’t aid experimentation.

I wanted the user of this shortening system to be able to change the characters with relative ease, and once you get into additional character sets, that’s tricky. Even if the upside-down letters had all been available, and the user realized that the progression was something like zero to nine in numerals, ten to thirty-five in lowercase letters, thirty-six to sixty-one in uppercase letters, sixty-two to eighty-seven in upside-down lowercase letters, eighty-eight to one hundred and thirteen in upside-down uppercase letters, and the remaining fourteen numbers in some pseudo-logical progression through (URL-safe) other symbols, how could they apply this knowledge to manipulate the strings? I don’t know of any OS that supports easily producing any of those upside-down letters. The same goes for any of the other alterations I considered, such as dots, umlauts, etc.

Having spent plenty of time exploring this, I eventually backed off it, and decided that my implementation took my approach about as far as it would go. We’ll see if any of the others end up with a better solution.

Incidentally, one positive aspect of my way of doing it is that unlike with traditional URL shorteners, no storage would be necessary, just a translation service, which could be trivially written in just about any language.

Tags: coding, geolocation, python, web-development

Open Source Thanks

Tadhg — Fri, 27 Nov 2009 07:54:28 +0000

It’s traditional on Thanksgiving to list things you’re thankful for. It struck me today that I should be extremely thankful for the existence of free software, and the contributions of thousands and thousands of programmers who have made their work freely available for others to use.

When I startd on a path towards software engineering, there wasn’t as much softwre libre around. The server-side operating systems and services did exist, and Apache is one of the first of those I used at any serious level. The first “consumer-oriented” piece of open source code I used regularly was, I think, MAME. I loved the concept of open source as soon as I encountered it, however, and gradually I’ve adopted more and more of it.

Apart from a few experiments, I haven’t gone fully to an open source platform, but there’s not a lot other than my OS that’s proprietary at this point. (That’s on the desktop side—on the server side everything is Free Software.) So I’m extremely thankful that as I’ve tried to adopt it fully, more and more people have contributed, so that almost everything I use regularly on the desktop side now is open source software:

Firefox for browsing.
jEdit for writing and coding.
Thunderbird for email.
VLC for video.
Songbird for music.
OpenOffice.org for when I have to interactively deal with RTF or .doc files.
Adium for IM.
Inkscape for vector graphics.
The GIMP for bitmap graphics (admittedly I still have some issues here, but don’t do enough bitmap work at this point to justify giving up on The GIMP).
Subversion for version control.

That covers almost everything. I’ll move to a Free desktop OS at some point. The only other tool I really want that I haven’t found in the open source world yet is photo management software. That’s a small gap, and for all the excellent software that I can use now that’s non-proprietary, I’m thankful.

To all you open source and free software contributors out there: thank you. You rock.

Tags: coding, personal, software

addons.mozilla.org Moving from CakePHP to Django

Tadhg — Wed, 18 Nov 2009 04:43:29 +0000

This post details why they’re making the move. I find this of interest partly because it’s a move from a very popular web language (PHP) to one that’s become vastly more popular in the last couple of years (Python), and also because Django is the one major Python framework I haven’t tried out yet. Because of my liking for Python, I have a personal bias that makes me happy to see a prominent project such as this one move to the language.

I find it odd that they’re going from Subversion to git instead of to Mercurial, but I like the fact that they’re moving their documentation to the reStructuredText-based Sphinx.

Tags: coding, python, web-development

tadhg.com » coding

A Little More Functional Programming

WARNING, SOLUTION

Related Posts

Minor Foray into Functional Programming

Related Posts

Some Vim Script Implementation, Testing, and Hackery

Related Posts

First Post With Vim

Related Posts

The Python Challenge

Related Posts

jQuery Project

Related Posts

Better Word Count in Vim

Related Posts

Code Katas

Related Posts

WordPress 2.9 Upgrade

Related Posts

Python Coding Exercise: Nested Dictionaries

Related Posts

Python Optimization Tips

Related Posts

2010 Goals

Related Posts

CrossFit and Coding (and Meat)

Related Posts

Some Python Tips and Tricks

Related Posts

GPS, URLs, Math, Python, Featuritis

Related Posts

Open Source Thanks

Related Posts

addons.mozilla.org Moving from CakePHP to Django

Related Posts