tadhg.com » tech

MapDroyd

Tadhg — Mon, 09 Aug 2010 00:10:01 +0000

I like it when I think that an app should exist and then find that it does. That’s what happened when I realized that a) I couldn’t use my phone’s GPS capabilities with its native (Google Maps) map application without a data connection and b) that it should be possible to download maps that work with GPS but no data connection.

The answer: MapDroyd. It does precisely that, letting you download vector maps to your phone and see where you are on them. I’ve used it a little bit over the last three days, and it’s been pretty good. There have been a few odd bits on the maps—a major street in central Manchester appeared to be missing, for example—but overall it’s been great, and the utility of having maps of more or less everywhere that can point to where you are on them should not be underestimated.

Tags: phone, software, tech

Travel, Ebooks, and Real Books

Tadhg — Sun, 01 Aug 2010 16:27:18 +0000

Despite my techie nature, I’ve never been enamored of books in electronic format. I love the feel of books, and while I have no trouble reading large amounts on screens of various kinds, I don’t like the idea of doing so for books.

I like reading to be a focused activity, and the fact that books are monofunction devices absolutely makes this focus easier to achieve. Simply having the option of switching activities on a computer (or whatever computer-like device I might read ebooks on) makes focusing that much harder. That’s certainly true when I watch movies on my computer. I don’t want the same thing to happen with reading. Even though it’s clearly a choice, the very presence of that choice is disruptive.

Dedicated ereading devices such as the Kindle might be an answer to this, but I haven’t come across any I find compelling, and I have no intention of supporting any that use a closed and authoritarian model. On top of that, there’s my love of physical books.

Despite all of the above reservations, I find myself considering some other approach, because physical books take up a lot of space. I’ve always been willing to deal with this in the past, but have recently found myself running closer and closer to the practical limits of what I can carry while travelling. This is despite packing reasonably well (perhaps not up to Monika standards, but not bad) and exploiting quite ruthlessly the ability of my Jacket of Holding to carry almost as much as my bag.

I generally err on the side of carrying too many books, but on my recent trip to Italy miscalculated and only brought three, two of which I read on the way out. The way back required at least two as well, so I was a book short. Physical space considerations are what caused me to bring only three books, and so I’m forced to admit that if I had some ereading device (which could be my phone, not necessarily another piece of hardware) this is simply not a problem I would run into.

So I’m going to have to experiment, start investigating getting books onto my Nexus One and my iPad, and seeing whether acceptable software can be found. This is a step I’m still feeling wary about taking, a testament to how emotionally attached I am to the physical book form.

Tags: books, personal, reading, tech, travel

8-Bit Cities

Tadhg — Tue, 13 Jul 2010 04:35:14 +0000

As a geek of my generation, I cannot help but find an eight-bit map of San Francisco (there are maps of other cities too) a wonderful thing. It became even cooler when I found out it was programmed in Python.

Tags: coding, culture, tech

Some Futurist Speculation on Screen Interfaces

Tadhg — Sat, 29 May 2010 06:52:46 +0000

The current interface upheaval is centered on touchscreens. I think this is an important step, and one which may allow for some significantly different interaction paradigms to emerge. I wonder how long touchscreens will remain dominant, however, even though the interfaces they help spawn may stick around for a long time.

Screens are becoming more and more advanced, meaning they’re getting smaller and smaller while supporting higher and higher resolutions. It’s not unrealistic to think that soon displays could be embedded in glasses, and once there, why not contact lenses? How far away are contact lenses that either have their own onboard computers or can receive a signal from a small device (like a phone)?

There are clearly obstacles to be overcome before that point, but it doesn’t seem all that far away. Another question is whether or not it will be possible for these devices to tell where you’re looking (this might be easier for a pair of glasses to do than contacts). If they can tell where you’re looking, they can overlay whatever you like onto what you’re looking at.

This isn’t a new idea, but it’s interesting to me how close it seems right now.

If we reach that point, then screens will have gone from being small in comparison to computers in the early stages, then large in comparison to computers today (that is, for desktop setups, where people generally want them to be as large as possible) back down to small again. On the other hand, the computer will also be small, so the largest part of a computer may end up being the input devices.

We might be relatively close to a setup where you sit down at your desk, and you have a keyboard (maybe I’m a dinosaur, but I don’t think keyboards are going away anytime soon; they’re too efficient—especially if you use something like Vim), a very small computer, and a motion tracker somewhat similar to the ones currently used for the Wii. All those, and your contact lenses, which might be passive receptors but might also need to emit something for the motion tracker to pick up. What that gives you, essentially, is as much screen real estate as you want, and something very close to those Hollywood-style holographic displays you can touch—with the one major difference that only you would be able to see your interface.

The only part of this that’s really missing right now is the contact lenses part. Computing power is already sufficient to calculate the three-dimensional view from a given point—maybe not well enough make you think you’re seeing something real, but certainly well enough to give you the correct angle for viewing a GUI.

After that, the next part is for it to figure out where your hands are. It might get to the point where it can let you type without a keyboard, although people might keep them for the feeback. But it should be easy enough at that stage to detect where your fingers are, thus allowing for three-dimensional “touch” interfaces.

I’m sure there are already research projects working on this kind of thing. I hate making predictions about future technologies, but I’d be surprised if this stuff weren’t available within twenty years.

What happens with interface design will be fascinating. Being able to manipulate computer artifacts with your hands, in three dimensions, will eliminated the need for a lot of abstractions, and I think that’s how much of the early work will go, just as that’s how the touch interfaces now are going. However, the increasing volume of information, personal and otherwise, will require abstractions of some kind; otherwise there’s just too much to deal with. What’s most interesting to me is not the “ease of use” side of things, but rather what will happen when you combine these (effectively) touchable holographic interfaces with a focus on user power and efficiency.

Here’s one idea: meta keys that can let you move the cursor with your eyes, and then “click” with another key (I refer to keys, but these could be foot taps, or who knows what, although here I’m aiming at something that doesn’t requires much movement of the hands, because I still think that keeping the hands in a typing position is likely to be fastest) to do what the mouse does today, but without having the mouse. That would already make editing better; I’d love to have the ability to do that in Vim right now.

Note that current touch interfaces eschew the concept of a pointer, figuring that the pointer is an unecessary throwback given the presence of touch ability. But control by visual attention could well bring it back, and similar things apply to many other interface approaches. In some ways the danger is that old lessons and methods will be forgotten in favor of alluring new playthings.

I have no qualms about suggesting that in a new order of interfaces where three-dimensional holographic touch is available, I’ll probably still want to use Vim. I suspect that Vim is the most effective interface I use every day. In fact, it’s the most powerful text manipulation tool I’ve found in years of searching (no, I haven’t used Emacs, and it may well be Vim’s equal, but most of this applies to it also) Cosidering that it doesn’t use any of the new interface technologies, and in fact obviates the need for the now-ancient mouse, that’s impressive, and important: a tremendous amount of interface power can be created not by focusing on making everything easier, but on exploring what increaing levels of abstraction and user learning can do—and, hopefully, combining that with the best of the revolutionary pieces yet to come.

Tags: future, interfaces, software, tech, text editing, Vim

1 in 6 Shocked by Cellphone Bill

Tadhg — Fri, 28 May 2010 06:51:32 +0000

Ars Technica reports on a new FCC survey [PDF] which found that about 30 million Americans have experienced unpleasant surprise at unexpected jumps in their cellphone bills. I’ll take this opportunity to complain about AT&T, who recently decided to simply tack a data plan (at $30/month) onto my bill when I switched my SIM card into my new Nexus One—and this wasn’t a request, they just did it and informed me by text message. I was able to get them to credit my account with the amount, which works since I’ve left them for T-Mobile, but that was extremely irritating.

In addition, the amount they charge for text messages is simply ludicrous, all the more so given that I had to pay not merely to send but also to receive them. Finally, I hate the fact that they concentrate on “minutes” so much while refusing to make talk minutes fungible—other services should be expressed in minute cost, e.g. one text message should deduct one minute of voice from your account. I realize they have every incentive not to do this, but it’s still extremely irritating. So far T-Mobile seem better; it looks like my T-Mobile bill will be around the same as, or maybe slightly more than, my AT&T bill, but now I have an unlimited data plan as part of that charge.

Tags: economics, personal, phone, tech

Your Online Life Might be an Open Book

Tadhg — Tue, 25 May 2010 06:09:57 +0000

My friends Will Moffat and James Home, and their friend Peter Burns, created a site to highlight just how exposed your Facebook updates are: Openbook. It’s an interface to Facebook’s public search API, and the first thing you should probably do with it is search for a phrase from one of your recent status updates. If it shows up, change your privacy settings!

The purpose of the site is to pressure Facebook into making its privacy controls far simpler, which is a noble goal. The current system is so bad it’s been called out by the New York Times, among others, and over time the set of who can see things about you by default has expanded significantly.

They’ve done a great job of exposing this, resulting in quite a lot of publicity

It’s worth it to spread the word about this—otherwise, only the internet-savvy will protect their privacy, and Facebook will find it easier to get away with the exposure of your personal data.

Tags: culture, internet, privacy, tech

iPad First Impressions: Consumption Machine

Tadhg — Mon, 24 May 2010 01:52:52 +0000

I got an iPad for work on Friday, and have been playing around with it. I would not have bought one for myself, and have grave misgivings about the device, primarily due to its highly proprietary, locked-down, walled-garden approach.

That being said, I think it’s an extremely slick, well-designed device, and may represent the first steps towards a new phase in accessing computer and/or internet artifacts.

Before I get to that, I want to note that I hated having to synch it to iTunes before I could use it. For a “next-generation” device, one that supposedly breaks with older computing paradigms, that seemed rather pathetic. Also, I can’t stand iTunes, and the iPad setup was the only time I’d started it in the years I’ve had my Macbook Pro.

Once that was out of the way, however, it was all slick and smooth thereafter. Using the device: easy. Installing applications: easy. Browsing the web: easy.

The interface is, as promised, extremely intuitive. It works as you expect it to work. It’s also just fun to use, somehow.

So what’s the problem? Well, all of the criticisms about its closed model remain true. That it’s not easy to tinker with, that applications have to be approved by Apple, that getting data on and off it has to be managed by iTunes or some other application—these are all really bad things, in my opinion. That ground has been covered before, and if you’re interested you can read Cory Doctorow’s take, which I largely agree with.

What struck me even more than those issues, however, is just how geared towards consumption the iPad is. You add functionality via one mechanism: the App Store. While you can get some things for free there, the name makes clear that it’s about buying things. You can get content through the iTunes store. Many of the other apps, like the amazon.com application (or the Kindle application), are focused on making it easy to buy things.

Beyond that, the activities the device is suited for are consumption activities. Reading, browsing, buying, and watching videos are all really easy. I haven’t tried games on it yet, but I suspect that while games on it will be fun, they won’t (yet, at least) be as deep as games on dedicated platforms or on fully general-purpose computers. It does, however, support “casual games” quite well.

Up until this point, all general-purpose computing devices have been geared towards productivity of some kind. Even if they ended up being used as gaming boxes or as entertainment and netsurfing devices, they were designed for production of some kind, usually writing, programming, or management. This is not the case with the iPad. Regardless of whether or not you could actually edit video, or create music, or write efficiently, on it, its primary purpose is consumption. As such, I think it represents a significant break in the history of computing and the internet as the first mass-market, general-purpose device that is geared towards consuming rather than both consuming and producing.

This raises a question, one that is critical to assessing the validity of many of the criticisms of it (such as Doctorow’s): have computers been combined production/consumption machines up until now merely due to technological limitations, or is that combination an essential part of what a computer is? I think that what Jobs has done, deliberately and effectively, is tried to separate those two parts and get rid of the irritating bits that support production so that consumption can be made accessible and easy to more and more people.

Furthermore, Apple recognized that what made this attempted break with computing history possible was the touch interface. Crucially, the touch interface removes a layer of abstraction. Now there’s no cursor to move around with a mouse, and no commands to invoke via unknown (or clumsily announced) keystrokes. No abstraction, and hence almost no need for training. In many ways, the iPad is a device aimed squarely at the people who can’t tell the difference between an article about Facebook’s login page and Facebook’s login page itself.

This is in many ways an inevitable step. URLs aren’t the best-designed things in the world (having the hierarchy go specific-to-general for domains and general-to-specific for resources on a domain, for example, is clearly screwy), and turning sites (accessible by URLs) into applications (accessible like other applications, by an easy launch mechanism) is probably inevitable too. But there’s clearly a loss with this, as abstraction in computing generally means power. That is, being able to manipulate resources with abstractions gives the user power, and abstracting things out before they reach the user takes power away from the user while making the smaller set of tasks they’re still able to do easier.

Many people will be in favor of this tradeoff; I’m not one of them, although I’d feel a lot better about it if it were a transient state that one could get past on a device (which I hope will be the case with the post-iPad tablets that emerge) rather than the state the device desires to keep its users in (which is the iPad’s approach).

Tags: coding, consciousness, culture, internet, tech, usability

sabbatical.close()

Tadhg — Mon, 17 May 2010 06:36:56 +0000

After a highly enjoyable, productive, and extended period, it’s time for me to return to the world of paid work.

I’m quite happy with the things I’ve done during my time off. Many of them are important only to me, but then, it’s been my time off.

Here are some of the things I’ve done since leaving Metaweb that I consider significant:

CrossFit. I’m approaching the one year mark, and it’s hard to overstate the importance of that for me. This is the first time in my life I’ve managed to keep an exercise routine of any kind going for that long. Not coincidentally, I’m also in the best shape of my life. I now feel that some kind of regular, intense, exercise has a similar place in my life to my morning pages—that is, something I I regard as mandatory and more or less permanent.
Diet. Not just switching back to eating meat, but also radically cutting down on sugar and processed foods. This also has a lot to do with my being in good shape.
Document format. Until this year, I hadn’t found a text-based document format I liked, but I have now finally settled on reStructuredText. One of the effects of this is that I shifted to doing all of my writing in a text editor rather than doing some of it in word processors. Another effect is that all of my writing is now much more amenable to programmatic manipulation.
Programming my own environment. Following from the previous item, I spent quite a bit of time working on solutions to various small niggles in my workflows, so that far less of my time is consumed by setup for writing. Some examples of this include scripts to insert the template for blog posts into my editor, scripts to publish to my blog directly from my editor, scripts to do word counting precisely as I wish it to be done, and many other small utilities that remove minor annoyances that I previously encountered almost every day. In many respects this kind of thing is precisely what programming and computers are for: to make it easier to create, and to lessen the gap between a thought and expression of that thought.
Digital organization. This includes recovering my subversion repository, restoring my blog to proper working order, and various other steps that have made my digital domain much more manageable. This is bound up with the previous item too.
Switching to Vim. This was probably a long time coming, but nevertheless required a fair amount of work, not least in porting all my scripts for jEdit over to it. I’ve been extremely happy with Vim. I think that to most people this might sound trivial; how many people would regard switching word processors as an “achievement”? But a text editor is different from a word processor, and switching to Vim isn’t like switching between non-modal text editors. This, combined with moving all my writing to text editors and thus to Vim, is a profound change for me, and one that I’m very happy I took the time to get through.
WordPress plugins. I wrote some of these to improve my blog after I got it working again.
PyWordCount. My word counting script/Vim plugin is done, but I have to finish the documentation for it and make it public. It’s significant to me because it’ll be my first self-contained public Python project, and also because it’s an outgrowth of the document format switch, the Vim switch, and the focus on programming my own environment to a much greater degree.
Q’Rith. I ran my first roleplaying sessions in 15 years, and am currently running my first roleplaying campaign in 15 years, in a world that I’ve created and am quite happy with so far.
Tennis. I started playing it again, and don’t know why I ever stopped.

Those are the main highlights. Many of them are internal, or are things that only I notice, but I think the effect they’ve had on me is actually quite major, and in positive ways.

As for what’s next, I’ll write about that tomorrow.

Tags: coding, CrossFit, diet, document formats, exercise, food, games, goals, personal, python, reStructuredText, roleplaying, software, sports, T, tech, tennis, text editing, version-control, Vim, WordPress, writing

Profiling Political Leanings by Browsing History

Tadhg — Fri, 30 Apr 2010 19:44:52 +0000

Slate has put together a tool that gives a very rough indication of a user’s political tendencies by checking which sites on a list the user has visited; each of the sites has a score based on readership by people with declared political affiliations.

It doesn’t actually read your browser history per se, instead just checking to see if you’ve visited the home pages of the sites on its list.

There are obviously many flaws with this approach, the first being the limitation to just two viewpoints, “liberal” and “conservative” (in the American senses). In addition, it can’t account for how you feel about any particular news source. And the list of sites it checks for is (almost inevitably) quite limited; a number of sites I visit regularly for news content don’t show up at all (counterpunch.com, for example). It would be far more interesting if the list of sites were larger—only 112 were used in the study that it was based on.

Tags: culture, politics, tech

Improving a Python Word Counting Function

Tadhg — Thu, 29 Apr 2010 20:33:36 +0000

This post could be summarized as “regular expressions are a lot faster than naive for loops”.

I’ve been working on improving the script I use for live wordcount in Vim, partly for performance and partly so that I can package it up as a plugin and share it with other people. Along the way I’ve improved the speed of the script rather significantly, and will go through the key part of that change here.

The core part of the script, unsurprisingly, is the count_words() function. As is common, it treats groups of letters surrounded by spaces as “words”. New lines also mark the end of a word. So do a variety of other characters, such as em dashes—in fact, one of the motivations for writing my own wordcount was because I wanted em dashes to be recognized as word separators.

In the following code, self.WORD_SEPS and self.LINE_SEPS are lists containing characters that separate words and lines. self.REPEATER_SEPS contains characters like “-”, which in my opinion does not separate words when used singly (e.g. “twenty-three” is one word) but does when used consecutively; “–” is a simple way to write an em dash when you’re restricted to ASCII. self.IGNORE is a list of characters that cannot form words on their own but aren’t necessarily separators, either.

The code is quite naive in its approach. It goes through the text letter by letter, checks the letter to see whether or not it’s in one of those lists, adjusts the count accordingly, and also stores previous_char in order to be able to handle the repeater separators.

def count_words(self, text):
    words, lines = 0, 1
    word, previous_char = 0, None
    lineword_separators = self.LINE_SEPS + self.WORD_SEPS
    repeater_test = lambda c, pc: bool(
        c in self.REPEATER_SEPS and pc in self.REPEATER_SEPS)
    separator_test = lambda c, pc: bool(
        char in lineword_separators or repeater_test(c, pc))
    for char in text:
        if separator_test(char, previous_char):
            word = 0
            if char in (self.LINE_SEPS):
                lines = lines + 1
        elif char in (self.IGNORE):
            pass
        else:
            #it’s part of a word.
            if not word:
                words = words + 1
                word = 1
        previous_char = char

    return (words, lines)

With that code as the central function, the script takes about three seconds to process a text file that’s approximately 400K/68,000 words.

The new version eliminates the need for a separate list of repeater separators, is fewer lines of code, and is faster:

def count_words(self, text):

    def ors(l): return r"|".join([re.escape(c) for c in l])
    def retext(text, chars, sub):
        return re.compile(ors(chars)).sub(sub, text)

    lines = text and len(re.compile(ors(self.LINE_SEPS)).split(text)) or 0

    text = retext(text, self.WORD_SEPS + self.LINE_SEPS, u" ")
    text = retext(text.strip(), self.IGNORE, u"")
    words = text and len(re.compile(r"[ ]+").split(text)) or 0

    return (words, lines)

The basic approach here is different. Rather than going through the text and evaluating it character by character, instead it replaces all of the characters in the separator lists with spaces, removes all the characters in the ignore list, uses split() to get a list of all the words in the text, and then counts the length of that list.

The most awkward part of it is having to construct the regular expressions out of the lists of characters, which is what the ors() function does. I could have replaced the lists with regular expressions, but I want it to be easy for users to alter the lists to suit their own preferences, and so left them as plain lists.

(Note that I’m restricted to Python 2.3 here since this needs to run in MacVim; hence the old-style and and or instead of new-style ternary expressions.)

Using that code instead of the previous version, and operating on the same 400K/68,000-word file, the script takes about 0.12 seconds—a 25-fold speed increase. Since this is a “live” wordcount, one that has to recount the entire file whenever you do something other than typing or adding a single new line, that’s an important difference, and it makes it much more usable when working with large files.

I had no illusions that my original version was anything close to optimal when I first wrote it, but I didn’t realize that the speed gains from doing it better would be so significant. Python’s regular expression handling is done by a C module, and I suspect that C module is quite highly optimized indeed. So, if speed is important, let highly-optimized C do the work, instead of doing it in your own code.

Tags: coding, python, software, tech, text editing, Vim, writing

Market Efficiency in Action

Tadhg — Wed, 21 Apr 2010 00:18:43 +0000

It should be clear that having better access to information than others will make it pretty easy to make money in market trading. And:

While markets are supposed to ensure transparency by showing orders to everyone simultaneously, a loophole in regulations allows marketplaces like Nasdaq to show traders some orders ahead of everyone else in exchange for a fee.

—Charles Duhigg. “Stock Traders Find Speed Pays, in Milliseconds”. The New York Times, 23 July 2009.

That seems like quite the loophole.

Predictably, the major players are exploiting it:

The deep-pocket bank/brokerages actually pay the NYSE and the NASDAQ to “colocate” their behemoth computers ON THE FLOOR OF THE EXCHANGES so they can shave off critical milliseconds after they’ve gotten a first-peak [sic] at incoming trades.

—Mike Whitney. “High-Frequency Trading as High-Tech Robbery”. CounterPunch, 16 April 2010.

The players on Wall Street are not going to stop doing this kind of thing anytime soon. Their entire existence is predicated on finding stuff like this and profiting from it. Call it arbitrage, cheating, deceit, exploitation, whatever, but it’s what makes them hugely rich, and if one avenue is closed off they’ll find another. Regulation would certainly help, but what’s really needed is more direct (and democratic) connections between people and their money—much of the dodgy financial history of the last thirty years is of Wall Street figuring out how to induce the shepherds of other people’s money (pension fund managers, corporate officers, public officials) to make bad bets, and then cleaning up thereafter. This is pervasive enough to have significantly warped the economy of New York’s cultural events, and it’s not getting any better.

The moral hazard in inducements to gamble with other people’s money should be clear. The technology certainly exists today to make concentrations of “public” wealth (as pension funds and, indeed, corporations ostensibly are) much more responsive to that public, but this would require wiping away a large chunk of the entrenched financial interests (and they have the money to fight that, clearly).

It almost goes without saying that cheating, particularly of the kind outlined by The New York Times and Mike Whitney above, should be forbidden both right now and in any future better market arena, but fixing things like that just isn’t enough. The whole thing needs to be changed utterly (note that it would be possible to change it utterly while retaining capitalism and democracy, incidentally—an anarchist revolution isn’t necessarily required here, although hypothetically a true anarchist revolution would solve these and a lot of other problems).

Tags: economics, politics, power, tech

Clearing Growl Notifications

Tadhg — Fri, 02 Apr 2010 22:43:22 +0000

I use a notification system for OS X called Growl, which provides a single channel for various applications to use when they have something to tell me. I mainly use it for IM and email. I’m fairly happy with it, but one issue that irked me was dealing with clearing a bunch of the notifications off the screen.

You can set them to disappear on their own, but then you run the risk of missing them if you’re away from your keyboard, or if you just don’t happen to catch their content in time. So I set mine to all stay on the screen until they’re dismissed. In order to dismiss them, you either click to close each one individually, or you can restart Growl from the OS menu bar.

To do the latter, you more or less have to use the mouse, and that eventually began to annoy me. I looked around, and found that there was a commandline controller called growlctl, but that it was no longer included in Growl releases. It can be found in the Extras section of the Growl 0.7.6 release, though, which they still have available for download.

I downloaded that, installed it (which really just means moving it to somewhere on your path), and verified that it does indeed still work. I then created a rather simple script:

/usr/local/bin/growlctl restart

And put it in my homedir, calling it gr.sh. Quicksilver knows that it’s there, so now clearing the notifications requires, at most, the following keystrokes: gr. Itch scratched.

Tags: coding, OS X, personal, software, tech

@: A Triumph of Design

Tadhg — Tue, 23 Mar 2010 04:50:26 +0000

The New York Museum of Modern Art has added the @ symbol to its architecture and design collection. Originating perhaps as a Latin abbreviation for “toward”, it showed up on one of the early Underwood typewriters (possibly the Underwood 1; it was definitely on the Underwood No. 5) and was used for “at the rate of”, which usage still survives.

But it’s really being recognized as a landmark of design because of its use as the separator between username and hostname—specifically, in email addresses, chosen by Ray Tomlinson in 1971.

It has plenty of other uses, and while we’re here I’ll note that I use it as an imformal extension to reStructuredText: I surround links that have titles with @ symbols and have a script run through the text to insert the actual reST necessary to support that (which, sadly, does require a custom directive).

Tags: design, language, reStructuredText, software, tech, writing

The Future of Tabletop Games? D&D on the Microsoft Surface

Tadhg — Sat, 27 Feb 2010 02:15:35 +0000

Microsoft Surface is an advanced touchscreen display built into a table, backed by a fairly advanced suite of software for gesture recognition. I hadn’t seen many compelling uses for this technology… until SurfaceScapes, a group at the Carnegie-Mellon Entertainment Technology Center, released demos of Surfaces customized to hangle playing miniature-based D&D on them.

I strongly recommend watching the video of the demos, and reading the commentary and an interview with the project lead:

Some of it is quite clumsy, but the potential is clearly there. And looks amazing. While this could definitely aid the playing of D&D, there are many games that could benefit from this kind of treatment. Specifically, any games that benefit both from having people physically present and that would also benefit from computation and helpful interfaces (yes, I’m looking at you, Twilight Imperium) could get a huge boost from this technology.

It’s a long way away from being available or affordable, and probably a long way from being easily modifiable, too. But we don’t seem to be that far from having commonplace enhanced tabletop games. In the meantime, for roleplaying games there’s the RPTools suite—a suite which may well eventually be a big part of an open source tabletop gaming toolset.

Tags: entertainment, games, roleplaying, software, tech

Some Vim Script Implementation, Testing, and Hackery

Tadhg — Wed, 17 Feb 2010 07:50:47 +0000

As a result of my porting over jEdit (Jython) macros to Vim, I now have a fair amount of (Python) Vim scripts, and have learned some things about how to set up those scripts. I’ll go through some of that below, and hopefully other people writing Python scripts for Vim will find it useful.

In jEdit, you invoke macros either by selecting them from a menu (clearly unacceptable) or by opening the Action Bar (Ctrl-Enter, for me) and then typing the name of the macro, where “name” means the non-extension part of the filename. I put my macro files went in a subdirectory of the macros directory, and there were namespace issues—all macro names had to be unique, and also couldn’t match any internal jEdit actions. Furthermore, you couldn’t pass arguments to them. Despite those issues, it was a powerful and quick piece of functionality, and I wanted to make Vim at least match it.

You can put Python directly into Vim script files, but I wanted to do that as minimally as possible. I added a Vim script that would (essentially) include a Python file, and then I had that Python file import from other Python files as necessary. The Python file includes a class, TBase, that contains the functions I want to run; as much code as possible is kept out of those functions and is in other files—files which ideally are individually testable.

To get to those functions from within Vim, I did this:

if filereadable($HOME."/.vim/plugin/tadhg/tadhg.py")
    pyfile $HOME/.vim/plugin/tadhg/tadhg.py
endif

" Create the commands pointing the Python in tadhg.py:
if !exists(":T")
  command! -range -nargs=+ T python tadhgbase = TBase('rs= rf='); tadhgbase('')
endif

nmap  :T

That code goes in a Vim script that goes in ~/.vim/plugins/tadhg/; it’s loaded when Vim is. This means that the code in tadhg.py is evaluated at Vim start, so the TBase class is available. Furthermore the user Ex mode command “T” creates a new instance of TBase and then calls it with whatever the user enters after “T”. Finally, Command-Enter is mapped to enter Ex mode and type “T ”, which is quite close indeed to the jEdit functionality I had, but without the disadvantages. I have much more control over the namespace, and I can pass arguments.

When the T command is invoked, a new TBase instance is created and it gets passed the range start and range end line numbers from Vim, which is pretty important for some of the functions. Then the new instance is called, with the argument of whatever was typed after “T ”.

This is the code for handling that latter aspect:

def __call__(self, argstring):
    """
    The first arg is the command; the rest are arguments to that command.
    """
    args = argstring.split(u" ")

    #Need to extract the keyword args out of args
    kwargs = dict([(arg.split("=")[0], arg.split("=")[1]) for arg in s.split(" ") if "=" in arg])
    args = [arg for arg in s.split(" ") if "=" not in arg]

    if args and args[0] in self.__class__.__dict__.keys():
        import types
        f = self.__class__.__dict__[args[0]]
        if type(f) == types.FunctionType:
            if args[1:]:
                f(self, *args[1:])
            else:
                f(self)
    else:
        print "no command by that name"

In other words, if the first (non-keyword) argument matches the name of a function in the class, run it and pass the rest of the (non-keyword) arguments to it. If I ever need keyword arguments as well, I’ll add the ability to pass them along too.

In order to actually manipulate Vim from these scripts, you have to use import vim. When I started writing this, I wasn’t sure how I’d test that, and I tried to keep the meat of the code in other files, e.g.:

def mpc(self):
    """
    Count the words, then insert them into the document's word count line.
    """
    from write_wordcount import WriteWordcount
    wwc = WriteWordcount(tadhgbase)
    wwc.vim_main(vim)

(Note that I use import sys; sys.path.append(myscriptdir) at the top of the file to make these imports work.)

One of the advantages of this construction is that WriteWordCount can theoretically be used as a standalone script, on any file, and also that its vim_main method takes vim as an argument. For testing purposes, I had to created a MockVim class, which currently looks like this:

class MockVimBuffer(object):

    def __init__(self):
        self.lines = []

    def __contains__(self, item):
        return item in self.lines

    def __iter__(self):
        return self.lines.__iter__()

    def __getitem__(self, index):
        return self.lines.__getitem__(index)

    def __setitem__(self, index, value):
        return self.lines.__setitem__(index, value)

class MockVimCurrent(object):

    def __init__(self):
        self.__dict__["buffer"] = MockVimBuffer()

    def __setattr__(self, attr, value):
        if attr == "buffer":
            self.__dict__["buffer"].lines = value
        else:
            self.__dict__[attr] = value

class MockVim(object):

    def __init__(self):
        self.current = MockVimCurrent()
        self.commands = []

    def eval(self, command):
        command_list = {
            "&ft": lambda: self.rawmodes,
            "exists('b:TotalWordCount')": lambda: False,
            "tvar": lambda: self.tvar,

        }
        return command_list.get(command, lambda: "")()

    def command(self, command):
        self.commands.append(command)

The eval method in MockVim simply returns whatever was appropriate for various tests I was running; I should change it to return self.command_list.get(command, lambda: "")() instead, and make command_list an instance variable so that the various tests can manipulate it as needed instead of having to put specific commands in the base code.

Because the methods in TBase pass vim along to the instances or methods of the modules they import, testing those modules is relatively simple: in the test, you create a new MockVim object, call it vim, and then pass it along instead.

So everything is great (and testable). Unless, that is, you (meaning me) lapse from full test-driven development discipline and bits and pieces of functionality creep into TBase. Because TBase has that import vim line at the top of it and running it when you’re not actually in Vim produces an ImportError.

The right way to deal with this is to refactor TBase to take vim as one of its initialization arguments, and then have it set that as an instance variable, and then pass that in its methods, while also altering the line in the parent Vim script to:

command! -range -nargs=+ T python import vim; tadhgbase = TBase(vim, 'rs= rf='); tadhgbase('')

That way TBase is much easier to test, since the fake vim can be passed in at test time. However, for a bunch of reasons I didn’t want to do that just yet, partly because it means some reasonably heavy refactoring of pieces of code that I use every day—without tests, because the whole issue here is that TBase isn’t testable. So I wanted a way to make it testable while altering as little of its existing code (which I know currently works, after all) as possible. This took me a while to figure out, and is rather hacky—but it works.

First, in the test file, do this:

from mockvim import MockVim
#Hack a mock Vim into the global namespace so that we can actually test:
import __builtin__
mv = MockVim()
__builtin__.mv = mv
from tadhg import TBase

Then alter the import vim line above the TBase class to instead read:

try:
    import vim
except ImportError:
    import __builtin__
    vim = __builtin__.mv

What this really does is hack a giant global variable, vim, into all of the Python that’s run after those lines in the test file. Normally doing this seems like a rather bad idea, but it’s also rather necessary to handle the less-than-ideal situation I’m dealing with. Once I have the test harness using this hackery running properly, I’ll feel a lot better about switching over to a better architecture.

Tags: coding, Jython, python, software, tech, text editing, Vim

First Post With Vim

Tadhg — Mon, 15 Feb 2010 04:05:48 +0000

Over the last couple of weeks I’ve been hacking away on scripts to customize Vim, replicating the scripts I made for jEdit. I’m more or less done, and this blog post is being written in MacVim. This hopefully means that when I’m done with it I’ll be able to publish it from within Vim, the same as with jEdit.

Writing scripts for Vim has been rather different from writing them for jEdit. jEdit is an editor with a robust API; Vim is an editor that kind of gives the user access to the API all the time, via keyboard commands, and scripting for it often consists of simply passing keystrokes. As an example, this is the Jython code for getting the current selection in a jEdit macro:

text = init.textArea.getSelectedText()

This is the Python for getting the current visual selection in Vim:

self.vim.command('normal gv"wygv')
self.vim.command("let tvar = getreg(’t’)")
text = unicode(self.vim.eval("tvar"), "utf-8")

From a programming perspective, jEdit is clearly a lot nicer to deal with.

I have two primary reasons for making the switch:

Future proofing: I couldn’t switch to the various pre-release versions of jEdit 4.3 because I couldn’t get my Jython scripts to work with them, and that made me worry that there simply weren’t enough people scripting jEdit in Python to make it likely that the plugin allowing Jython macros would be supported. Given how useful my scripts are, that would have trapped me with jEdit 4.2, or some future version, if support for JythonInterpreter flagged. Vim, on the other hand, seems to have a lot of people scripting for it in Python, and I don’t see Python support simply disappearing anytime soon with a new release.
Power/speed: I’m someone who’s quite inclined towards keyboard shortcuts in all applications, and this means Vim should be a highly efficient environment.
Mentality: Related to the previous point, I think that being able to manipulate text more powerfully will reinforce the approach of scripting as many steps as possible, rather than doing them (no matter how efficiently) manually. Even over the last couple of weeks, I’ve noticed that I’m more inclined to write scripts than to put up with various manual steps in my daily workflow.

So far I’ve mainly worked to replicate my jEdit setup in Vim, with a few small improvements here and there; the largest one is probably the live word count, which means that I can glance at the status line to see the current word count instead of having to select the text to be counted and then invoke the word counter. I’ve also tweaked it so it only counts the relevant parts of the document—for example, in this blog post, it doesn’t count the title or the datestamp.

I don’t intend to abandon jEdit, and have a feeling that I’ll use it for various things here and there, but I’ve switched over the Vim for the majority of my writing and programming.

Tags: coding, Jython, python, software, tech, text editing, Vim, writing

Better Word Count in Vim

Tadhg — Mon, 18 Jan 2010 07:40:14 +0000

I’m currently trying out Vim (again), and have made more progress this time, mainly due to Seth’s help. The key things that have made it better:

:set hidden. Absolutely critical, this. Stops Vim from complaining when you try to switch buffers and your current buffer has unsaved changes.
bufexplorer. Makes switching buffers a lot easier.
A better Python syntax file. I didn’t like the defaults.
My own indentation and syntax files for reStructuredText.

Really, though, the key first one was :set hidden. Before that I felt that I had completely misunderstood Vim’s file management model.

Once I got the syntax highlighting to a reasonable state, I ported my word count macro over to Vim. This wasn’t too hard after the inevitable character encoding problems. For working within Vim scripts, I strongly suggest using the following:

ulines = [unicode(line, "utf-8") for line in vim.current.buffer]

(Assuming your Vim encoding is set to UTF-8, of course.)

The Vim version of my script isn’t as versatile yet as the jEdit version, because I don’t know how to make it work with only selected lines (something that’s easy in jEdit). Apart from that, though, it seems to work quite well; the next version of it might do “live” word count in the status bar.

I’m currently writing this in Vim, or kind of: the actual writing is in Vim, but the creation of the file and the template, the automated expansion of various reStructuredText entities, output to reStructuredText, and the publication, are in jEdit as I haven’t ported those over yet.

The script:

function! WordCount()
python << EOF
import re
import vim

class WordCounter(object):
    """
    Vim script for better word count.
    """

    LINE_SEPARATORS = (
        "\r",
        "\n"
    )

    WORD_SEPARATORS = (
        " ",        # space
        "\t",       # tab
        "/",        # slash
        "&",        # ampersand
        ’"’,        # double quotation mark, straight
        u"\u201C",  # double quotation mark, left
        u"\u201D",  # double quotation mark, right
        u"\u2018",  # single quotation mark, left
        u"\u2013",  # en dash
        u"\u2014",  # em dash
        ">",        # greater than symbol
        "<",        # less than symbol
        "+",        # plus
        "=",        # equals
    )

    REPEATER_SEPARATORS = (
    #These are only separators if they’re present consecutively, e.g. -- or ..
        "-",
        "."
    )

    IGNORE = (
    #Not separators per se, but should not be treated as word content
        "’",        # single quotation mark, straight
        u"\u2019",  # single quotation mark, right
        "(",        # left parenthesis
        ")",        # right parenthesis
        "[",        # left bracket
        "]",        # right bracket
        "{",        # left curly bracket
        "}",        # right curly bracket
        "|",        # bar
        "-",        # hyphen
        "#",        # hash mark
        ".",        # period
        "_",        # underscore
        "`",        # backtick
        "\\",        # backslash
    )

    def word_count(self):

        ulines = [unicode(line, "utf-8") for line in vim.current.buffer]
        text = u"\n".join(ulines)

        chars = len(text) #Pretty sure I want the actual char count, not the adjusted char count.

        text = self.remove_directives(text)
        text = self.adjust_for_rest(text)
        words, lines = self.count_words(text)

        print "chars: %s, words: %s, lines: %s" % (chars, words, lines)

    def remove_directives(self, text):
        textlines = text.split("\n")
        newlines = []
        comment = re.compile(r"[ ]*\.\. [a-zA-Z0-9_\|]")
        argument = re.compile(r"    :[^\:]*:")
        for line in textlines:
            if not comment.match(line) and not argument.match(line):
                newlines.append(line)
        return "\n".join(newlines)

    def adjust_for_rest(self, text):
        """
            Go through each of the special cases for reST.
        """
        text = self.rest_adjust_pipe_space(text)

        return text

    def rest_adjust_pipe_space(self, text):
        """
            Special-case "|\ " to make sure e.g. "|Hypnotic Specter|\ s"
            doesn’t get counted as three words.

            |Incinerate|\ s |Hypnotic Specter|\ —|Hypnotic Specter|\ s
            The above line should be counted as five words.
        """
        spacere = re.compile(r"\|\\ ([^ ]{1})")
        finds = spacere.findall(text)
        text = spacere.sub("|\g<1>", text)
        return text

    def count_words(self, text):
        words, lines = 0, 1
        #go through the text character by character:
        word, previous_character = 0, None
        for character in text:
            if character in (self.LINE_SEPARATORS + self.WORD_SEPARATORS) or (character in self.REPEATER_SEPARATORS and previous_character in self.REPEATER_SEPARATORS):
                #it’s a separator
                word = 0
                if character in (self.LINE_SEPARATORS):
                    lines = lines + 1
            elif character in (self.IGNORE):
                pass
            else:
                #it’s part of a word.
                if not word:
                    words = words + 1
                    word = 1
            previous_character = character

        return (words, lines)

WordCounter().word_count()
EOF
endfunction

if !exists(":WW")
  command! WW  :call WordCount()
endif

Tags: coding, Jython, python, software, tech, text editing, Vim, writing

tadhg.com » tech

MapDroyd

Related Posts

Travel, Ebooks, and Real Books

Related Posts

8-Bit Cities

Related Posts

Some Futurist Speculation on Screen Interfaces

Related Posts

1 in 6 Shocked by Cellphone Bill

Related Posts

Your Online Life Might be an Open Book

Related Posts

iPad First Impressions: Consumption Machine

Related Posts

sabbatical.close()

Related Posts

Profiling Political Leanings by Browsing History

Related Posts

Improving a Python Word Counting Function

Related Posts

Market Efficiency in Action

Related Posts

Clearing Growl Notifications

Related Posts

@: A Triumph of Design

Related Posts

The Future of Tabletop Games? D&D on the Microsoft Surface

Related Posts

Some Vim Script Implementation, Testing, and Hackery

Related Posts

First Post With Vim

Related Posts

Better Word Count in Vim

Related Posts