I can plot that data in two keystrokes

(plus a carriage return)

One of the ideas I most gleefully stole from YASQL for sqlpython is special terminators, sequences like \g and \c that replace a SELECT statement's ending semicolon. When a query ends with a special terminator, the output is specially formatted: \c gives CSV, \h gives HTML, \t gives transposed (columns as rows / rows as columns), etc. Type help terminators for details.

sqlpython 1.5.0 is out today, with the most demented special output format yet: CHARTS! Instant ad-hoc grapical goodness direct from your query, no tedious mucking around in spreadsheets or exporting to another program. Just terminate your query with \l (line graph), \L (scatter graph - no lines), \p (pie chart), or \b (bar graph).





Also, as of 1.5.0, it's pretty easy to define your own special terminators and formats. Just install sqlpython in uncompressed form (easy_install -UZ will do that), open up output_templates.py, and follow the pattern.

ponyshow: showing off in Python

PyOhio is getting a table at Ohio LinuxFest to advertise PyOhio and Python in general. We're going to set up a computer running demonstrations of eye-catching Python tricks - stuff passing geeks can look at and think, "Hunh! That's pretty cool! I'm going to try this Python thing."

To run the demo, I've written a little script called ponyshow. You can use it yourself (on *nix) - install Mercurial, then
hg clone http://hg.assembla.com/ponyshow ponyshow

I need suggestions for what to put in the show! If you had a few lines of code to show why you love Python, what would they be? Importing modules is fine - I'm certainly going to show off vPython and pyglet, for example. What would you show?

more area events

Ohio LinuxFest isn't the only major geeky event coming up in our area. Also check out:

• the Miami Valley Joint Societies meeting: more geeky groups than you can shake a USB drive at, all coming together for their annual dinner meeting. Sep. 25 in Dayton.
• .NET University: a free 101-level education in .NET. Nov. 8 in Columbus.
• 50 Years of Computing at UC: the big picture. Nov. 10-11 in Cincinnati.

Geek Event Finder: now working

It's working! The Geek Event Finder on Google App Engine! Go play!

In some ways, the Google App Engine is a dream. Not thinking about the app server is wonderful. Deploying couldn't be easier.

I hate the GAE datastore with the passion of a thousand blazing suns, however. 90% of the work of this project has been trying to figure out workarounds and kludges for its bizzare limitations, like

• There is no mass delete. None. No truncate. No way to get rid of a large number of records at once.
• Bulk upload exists, but it always appends to the datastore - never replaces - which brings you right back to the "no mass delete" problem.
• No long-running operations - anything that would take more than a couple seconds dies - so you can't loop over all your records to do something (like delete them).
• Countless unexpected restrictions on queries. Can't filter on one property while sorting on another. Can't do inequality filters on more than one property. Can't filter on a string property if it is multiline (has \n's) or is longer than 500 characters (type Text). Can't use any function calls or arithmetic within a WHERE clause. Queries that fetch a large number of records die instead of completing (so I fetch in LIMIT 20 batches and assemble the results on the app side... crazy).

So if you look at the code and see some incredibly stupid stuff going on with data access - trust me, I tried fifteen different sane ways first. I am so not buying the buzz about this being "the future of databases". Fighting for hours to try to kludge your way to your data... that's the Bad Old Days, not the future.

My workaround for mass deletion was to write pages that would delete one record, then invoke it in a loop from my client computer. That has to run all night to clean out the datastore when new data is uploaded.

But anyway. I'm still very happy. It works!

Ohio LinuxFest

Ohio LinuxFest is Oct. 11 - one month from today!

It's free, it's fabulous. Missing it would be like missing your own birthday. Go get registered!


The PyOhio gang is going cook up some good stuff to do - a table in the midway, a Python workshop in the Open Spaces, etc. Let me know if you have ideas and/or if you'd like to help staff the table.

BigTable blues

This was supposed to be the blog entry where I would announce the Geek Event Aggregator's successful port to Google App Engine.

(sigh)

I've read an awful lot of buzz about how GAE's BigTable is the Next Big Thing in data, makes RDBMS obsolete, etc. Maybe I'm just doing it wrong, but right now I am utterly unimpressed.

The Geek Event Aggregator needs to search its database of 5000 or so events for events whose longitude and latitude are close enough to the user to be of interest. Does that sound so impossible?

I couldn't do it in GAE. First, "Inequality Filters Are Allowed On One Property Only" - so I can filter for longitude or latitude, but not both. I had to filter only for longitude, pull all resulting records into the application, and finish boiling the ocean in my app. It was slow, in the local application environment, but I hoped it would run faster once uploaded to the actual GAE production servers.

In production, though, it doesn't run at all - "Timeout: datastore timeout: operation took too long.". Querying from 5000 records - too much for the mighty BigTable, apparently. Dropping the filters on longitude (to do all the filtering in the app, in case inequality filtering is just so poisonous) didn't help, either.

Oh well. I still enjoyed working with GAE at first, and maybe I'll use it again for something with very light data demands. For the Geek Event Aggregator, I do have a server available where I can host in TurboGears - it'll just take a bit of rewriting. Later this week, hopefully.

recursive dowload with wget

I often need to download documentation for offline use. Online docs typically come as a finely branched tree of separate .html files; sometimes a one-page or PDF version is also available, but those aren't as convenient to use.

I'd gotten sick of using "File/Save As..." from my browser again and again, but Firefox shortcuts like DownThemAll flattened the directory structure, broke the internal links, and didn't crawl them to get sub-pages.

The answer, of course, was in my /usr/bin all along: wget.

catherine@Elli:~/docs$ wget --recursive --convert-links \
> --page-requisites --no-parent \
> http://www.postgresql.org/docs/8.3/interactive/index.html

or, shortcut form,

catherine@Elli:~/docs$ wget -rkp -np \
> http://www.postgresql.org/docs/8.3/interactive/index.html

pulls down the everything beneath the index page, and it turns out perfectly, with a single command!

--convert-links adjusts all the internal hyperlinks so they'll work perfectly in your downloaded version.

Without --no-parent, wget would find the link to "Home" and download everything under that.

--page-requisites makes sure you get the images, stylesheets, etc. for your pages to work properly.

Now bookmark the downloaded top page in firefox, give the bookmark a keyword, and voila! Now just typing your keyword into the Firefox URL bar gets you to your perfect offline mirror.

Regex skirt review

I got my xkcd regex skirt as a birthday present (funny what subtle hints like "this is exactly what I want for my birthday please please please" will do), so I thought I'd give a review. An hReview-microformatted review, of course.

gloriously geeky, if flawed

Aug 22, 2008 by
Catherine Devlin
(sam18xgy4fz@liquidid.net)product

XKCD Regex Cheat Skirt

Rating: 4 of 5

We've all got a drawer full of geeky T-shirts. But what about a geeky skirt? Thanks to XKCD, it finally exists!

The text is regular expressions syntax reminders, oriented upside down so you can read it while you sit. Simple stuff, like:

Must escape: \|()[{^$*+? 
^ start of string 
$ end of string 
[^abc] ^ means not 
\n Newline 
{N,M} N to M

The font is large, comfortable to read while sitting. I wish the lines weren't double-spaced - I think 1.5 spacing would make the text look more like a trendy decorative hem accent.

The sizes run big. I wear 10's or 12's, ordered the medium, and it is more than big enough, even without the stretchiness. It's a soft, stretchy fabric, stretchy enough to do Rockettes high-kicks if that's what you do (but programming pays better).

The hem falls a bit above the knee, but not at all "you are not leaving the house like that young lady" high.

I'm a little disappointed in the craftsmanship, to tell the truth, for the $35. It kind of looks like a skirt made by T-shirt makers - it's two simple trapezoidal panels sewn together. The text leaves a fairly broad gap at each edge of the panels, making it really obvious where the side seams are. (I guess that could be considered a fashion choice, but I prefer hiding the seams.) The print is that slightly raised plasticky print, and I have zero confidence in its longevity. I will obviously not only have to launder it gently as advised, but also refrain from wearing it every single day. Save it for geeky occasions where it will be properly appreciated.

Nonetheless, it is far and away my favorite skirt, because it is far and away the winner of the "geekiest skirt ever" contest, and that trumps everything in my book.

0.3

This hReview brought to you by the hReview Creator.

taking FOSS to the people

It's all very well to invite newbies to come to your Linux group, but you're not going to get really new people that way. Coming to a meeting or an Installfest is a commitment that demands strong, specific curiosity at the very least. Chances are, if someone knows enough about Linux to know that they want to know Linux, they're going to learn it, helpful usergroup or not. That's especially true now that the entry barrier is so low with super-easy distributions like Ubuntu. But what about those who don't know what they're missing?

That's why some of us from Dayton Linux User Group are considering renting a booth at the Montgomery County Fair. We can set up several computers and let people touch, try, and see just how easy it is - people who had never even heard of Linux or open-source, people who had no idea there was a choice, who thought that Microsoft was just how computers run. Step right up, folks. Surf a webpage, write a document, edit a spreadsheet. It's real, and it doesn't bite.

"Do you know that Tux loves you, and has a wonderful plan for your computer?"

(WOW, if only I had cartooning talent, I would make THE BEST mockup of a lowbrow evangelical tract there has ever been.)

(Wait a minute. Since when did that require cartooning talent?)

(Incidentally, I really am a hardcore Christian. Seriously. Maybe that's why this sort of thing appeals to me so much. "Freely you have received, freely give"; Jesus wrote the GPL!)

The only thing that may stop us is the booth fee: $270. It's reasonable, all things considered, but it's still a sizable hat to pass around for a small group.

Anyway, whether we get this together in time for the fair or not, I'd love to get other ideas for where to take a show like this. The fair is perfect because it has a whole lot of people milling around looking for something interesting. Where else?

Good Kids and Open Spaces

Not everything at PyOhio was perfect, of course. In particular, we were frustrated that we didn't get more participation in the Open Spaces. Some of the attendees pretty much took a spot in the auditorium, watched the scheduled talks, and went home. I think they were satisfied, but those attendees missed out on a lot.

I wasn't too surprised, since even PyCon wrestles with how to encourage Open Space participation. Many PyOhio attendees were inexperienced conference-goers with absolutely no experience with Open Spaces. But I don't think unfamiliarity was the only barrier.

Back in high school, I was one of the Good Kids. I was Born to be Mild. I went to class early, sat down, readied my notebooks, and waited for the teacher to start. I did not hang around in the hallway gossiping. I certainly did not cut class!

I think most people who become programmers started out as Good Kids, and that's part of why this is hard. We can put a preachy message in the program guide singing the virtues of Open Spaces and the hallway track, of informal, collaborative learning - but they were Good Kids, and they're not buying it. When we say, "get into unplanned discussions, skip some scheduled talks in favor of Open Spaces", it sounds to them like "cut class and hang out." No wonder it's a tough sell! Closing their ears to that is exactly how they became educated enough to be at a programming conference in the first place. In high school, the things your peers could teach you would probably get you arrested, hospitalized, and/or pregnant.

So we've got some ideas to get people over this psychological hump for PyOhio 2009. We're going to use panel discussions to ease people from listening into talking, try a schedule that forces people to get up and move around and bump into each other, and leave time slots where there are no scheduled talks.

Honest, everybody, it's OK. One Good Kid to another.

[EDIT: Let me make clear that this wasn't the only thing keeping people in their seats. We had a jam-packed schedule - gave in to the "fit everything in" temptation, I'm afraid; the Open Space rooms were distant from the main lecture room; and there was no natural transition to a hanging-out space when the library closed at 6. We'll certainly be looking hard for ways to improve those for PyOhio 2009.]

PyOhio: wow

Wow. PyOhio went great. I have so much to say, I hardly know where to start. Expect multiple posts.

The hope was to duplicate some of the learning and atmosphere of PyCon on the small scale. I really think we achieved it. The talks were excellent, attendee enthusiasm was high, and plenty of people stepped up to the tasks that needed doing.

THANK YOU to everybody who was involved!

Not everything was perfect, of course. I'll devote a full post to glitches and mistakes. But in general, I'm extremely happy. We could have sat around wondering whether this area could really support a good regional conference, but instead, we just went ahead and did the experiment, and the answer is clear: yes, it can and it did!

Check out some of the neat swag donation we got from WearPython and Apress!

PyOhio: one more week

PyOhio is Saturday, July 26!

Please register right away - it helps us schedule talks properly and guarantees you entry, food, and swag.

What sort of Open Spaces do you want to take part in? Do you want to do a poster? How about a Lightning Talk?

See you there!

serendipity

I just met with Amanda Laucher at the Blue Moose Cafe in Morgantown, WV.

I don't live here; she doesn't live here; we weren't at a conference; we didn't expect to meet. We were both in town on family visits, and both seeking out an internet fix. (I'm visiting my father-in-law at Ruby Hospital - but writing Python there, hah.)

She's a serious geek-community addict like me - she's organizing the DevLink Bus. She gave me great advice for PyOhio and tipped me off to other upcoming events, and I gave her some ideas from PyCon. It was great!

That's just really cool. Someday, when wireless access is ubiquitous, geekish nomads will no longer concentrate themselves in a handful of WiFi hotspots, and unforeseeable meetings like these will become more rare. That will be the downside of the future.

cx_Oracle and Oracle XE on Ubuntu

Simplifying options have appeared since the last time I posted about installing cx_Oracle against Oracle XE on Ubuntu, so I thought I'd produce an updated, simplified summary.

1. Install Oracle XE from Oracle's own repository of .deb files
2. export ORACLE_HOME=/usr/lib/oracle/xe/app/oracle/product/10.2.0/server/
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH
3. sudo apt-get install python-dev python-setuptools
sudo -E easy_install cx_Oracle
   The -E flag carries your environment variables into your sudo session; if you omit it, you may get a cannot locate an Oracle software installation error.
4. Test:
$ python
>>> import cx_Oracle
>>> cnxn = cx_Oracle.connect('system/mysystempassword@xe')
>>> crsr = cnxn.cursor()
>>> crsr.execute('SELECT * FROM dual')

cmd2 0.3.4

I just keep having fun with cmd2 (and with sqlpython, which is built on cmd2 and inherits its goodies). The latest:

• If you use > to redirect output and omit a destination filename, your output will go into the paste buffer. Then you can Ctrl-V and paste it into any window.
• You can use the pipe symbol (|) just like in Unix, to redirect output into any shell command.

geographical anonymity

On the Internet, nobody knows you live in Dogtown.

I've volunteered to do the publicity for PyOhio. I guess it's going really well. Two months out, we've already got 80 people signed up for the PyOhio announcements mailing list. If we assume that most of them will come to PyOhio, plus some others, and that the numbers will keep growing... maybe I'd better relax before we max out the library's capacity.

Still, bootstrapping a regional community is an interesting challenge. It's easy to go out on the internet and find a bazillion people who use Python - but how many of them are in or near Ohio? The wonderfully free-floating nature of the Internet and the open-source community, where geography is irrelevant, can become frustrating when geography actually is relevant.

Finding local companies is even harder. Most companies' websites try to give the impression that they have no geographical location whatsoever - wouldn't want to put off a potential client just because they're on another continent, after all. They also try to avoid mentioning the languages they use, again trying not to appear limited. (In a sense, that's a very good thing, because end users should shop for brains, not tools - it's just difficult for this application.) Finally, with a technology like Python, many of us users sneak it into our jobs without our higher-ups even knowing it, much less promoting it as a strength of their companies.

Any ideas? Googling for "Python Ohio" gets... well, me. Looking for "Python" plus the domain names of Ohio universities is helping a bit. (Universities, unlike businesses, do publicize their physical locations and the technologies they use.)

Anyway, if you've been thinking of submitting an abstract for PyOhio, quit thinking and write; our deadline is June 1 (this Sunday). And don't forget to blog...

That number again:

PyOhio

Regional Python programming miniconference

July 26th, 2008

Columbus Metropolitan Library

96 S. Grant Ave.

Columbus, OH 43215
USA

Tags: python open-source programming conference ohio

(Side note: Blogger's captchas are very near the threshhold of legibility, and getting worse. If I quit posting, I guess you'll know why.)

slides from SQL*Plus Alternatives

I've posted my slides from today's GCOUG presentation here.

Thanks for being a great audience! Feel free to contact me to ask questions about SQLPython or request new features.

cmd2 0.3.0

I'm speaking on SQL*Plus alternatives again on Wednesday at GCOUG, and trying to polish up sqlpython a bit in preparation for that. That required polishing cmd2; I've been increasingly embarrassed that cmd2 used homemade code (flagReader) to interpret --options --like --these, instead of using a perfectly good module from the standard library (optparse).

Optparse is not designed for iterative cmd-type apps where you remain at an application prompt after issuing each command - it keeps issuing sys.exit()s to kick you back to the shell. That was my excuse for not using it earlier, but once I decided to fix it, a little subclassing took care of it.

Next I got a little deeper into decorators than I'd ever been, to provide a concise way to specify those options:

@options([make_option('-k', '--ketchup', action="store_true", help="Ketchup on that?"),
         [make_option('-r', '--relish', help="relish (sweet or dill)")])
def do_hotdog(self, arg, opts):
    if opts.relish:
        self.stdout.write('One dog with %s relish\n' % opts.relish)
    if opts.ketchup:
         self.stdout.write('Hope you're not wearing white!\n')

That was pretty fun. Took some experimentation and this article, though, to figure out how to write a decorator that would accept an argument.

I didn't throw away the homemade ("flagReader") code, for backward compatibility; I issue a DeprecationWarning instead. I did throw away the guts and reimplement it as a wrapper around optparse. I managed to do that part on a single 20-minute bus ride, thanks to good doctests on flagReader. I feel so grown-up!

Finally, inspired by Brandon Rhodes' article in the March 2008 Python magazine, I tried supporting buildout deployments this time. I'm not sure I'm getting it quite right yet; expect it to actually work in cmd2 0.3.1.

PyOhio Call for Proposals

PyOhio: Call for Presentations

PyOhio, the first annual Python programming mini-conference for Ohio and surrounding areas will take place Saturday, July 26, in Columbus, Ohio. The conference is free of change and will include scheduled presentations, Lighting Talks and unconference-style Open Spaces.

You can read more about the conference at http://pyohio.org

PyOhio invites all interested people to present scheduled talks. All presentations are expected to last 40 minutes with a 10 minute question-and-answer period. PyOhio will accept abstracts covering any area of Python programming. A classroom area with computers will also be available for possible hands-on sessions.

All presentations proposals should submit abstracts no longer than 500 words in length. Abstracts must include the title, summary of the presentation, the expertise level targeted, and a brief description of the area of Python programming it relates to.

All proposals should be emailed to for review.

The submission deadline will be June 1, 2008. Accepted proposals will be notified by July 1.

choose the tool last

Yesterday, someone asked me if I could design "an MS Access database" for him.

Growl.

As I reflected on it, I realized that the problem is not really the particular technology specified - the problem is that the technology was specified at all before the need was described.

If I offered to make you anything you wanted for dinner, would you ask for "something made with a Teflon spatula?" If you did, I'd tell you, "No no no no no. Describe what you want to eat, and let me figure out what I'm going to use to cook it."

My potential client didn't literally mean that he wanted MS Access to be used, he meant that he wants a database with a GUI interface. He thinks of "Access database" as a way to describe that, because it's the only thing he's ever seen used to build products vaguely resembling the one he's imagining. It's nothing more than a limitation of vocabulary and experience.

The problem is that, so often, that kind of pidgin is taken literally. Tools are pre-selected far too early - while the business users are imagining their needs, before the geeks themselves are even involved,

As a result, there are a lot of geeks out there trying to beat eggs with Teflon spatulas.

My story ends happily, however. My potential client got it when I explained this, and we're going to meet so I can find out what he wants. Then - and only then - will we start discussing which tools to use.