I can plot that data in two keystrokes

(plus a carriage return)

One of the ideas I most gleefully stole from YASQL for sqlpython is special terminators, sequences like \g and \c that replace a SELECT statement's ending semicolon. When a query ends with a special terminator, the output is specially formatted: \c gives CSV, \h gives HTML, \t gives transposed (columns as rows / rows as columns), etc. Type help terminators for details.

sqlpython 1.5.0 is out today, with the most demented special output format yet: CHARTS! Instant ad-hoc grapical goodness direct from your query, no tedious mucking around in spreadsheets or exporting to another program. Just terminate your query with \l (line graph), \L (scatter graph - no lines), \p (pie chart), or \b (bar graph).





Also, as of 1.5.0, it's pretty easy to define your own special terminators and formats. Just install sqlpython in uncompressed form (easy_install -UZ will do that), open up output_templates.py, and follow the pattern.

ponyshow: showing off in Python

PyOhio is getting a table at Ohio LinuxFest to advertise PyOhio and Python in general. We're going to set up a computer running demonstrations of eye-catching Python tricks - stuff passing geeks can look at and think, "Hunh! That's pretty cool! I'm going to try this Python thing."

To run the demo, I've written a little script called ponyshow. You can use it yourself (on *nix) - install Mercurial, then
hg clone http://hg.assembla.com/ponyshow ponyshow

I need suggestions for what to put in the show! If you had a few lines of code to show why you love Python, what would they be? Importing modules is fine - I'm certainly going to show off vPython and pyglet, for example. What would you show?

more area events

Ohio LinuxFest isn't the only major geeky event coming up in our area. Also check out:

• the Miami Valley Joint Societies meeting: more geeky groups than you can shake a USB drive at, all coming together for their annual dinner meeting. Sep. 25 in Dayton.
• .NET University: a free 101-level education in .NET. Nov. 8 in Columbus.
• 50 Years of Computing at UC: the big picture. Nov. 10-11 in Cincinnati.

Geek Event Finder: now working

It's working! The Geek Event Finder on Google App Engine! Go play!

In some ways, the Google App Engine is a dream. Not thinking about the app server is wonderful. Deploying couldn't be easier.

I hate the GAE datastore with the passion of a thousand blazing suns, however. 90% of the work of this project has been trying to figure out workarounds and kludges for its bizzare limitations, like

• There is no mass delete. None. No truncate. No way to get rid of a large number of records at once.
• Bulk upload exists, but it always appends to the datastore - never replaces - which brings you right back to the "no mass delete" problem.
• No long-running operations - anything that would take more than a couple seconds dies - so you can't loop over all your records to do something (like delete them).
• Countless unexpected restrictions on queries. Can't filter on one property while sorting on another. Can't do inequality filters on more than one property. Can't filter on a string property if it is multiline (has \n's) or is longer than 500 characters (type Text). Can't use any function calls or arithmetic within a WHERE clause. Queries that fetch a large number of records die instead of completing (so I fetch in LIMIT 20 batches and assemble the results on the app side... crazy).

So if you look at the code and see some incredibly stupid stuff going on with data access - trust me, I tried fifteen different sane ways first. I am so not buying the buzz about this being "the future of databases". Fighting for hours to try to kludge your way to your data... that's the Bad Old Days, not the future.

My workaround for mass deletion was to write pages that would delete one record, then invoke it in a loop from my client computer. That has to run all night to clean out the datastore when new data is uploaded.

But anyway. I'm still very happy. It works!

Ohio LinuxFest

Ohio LinuxFest is Oct. 11 - one month from today!

It's free, it's fabulous. Missing it would be like missing your own birthday. Go get registered!


The PyOhio gang is going cook up some good stuff to do - a table in the midway, a Python workshop in the Open Spaces, etc. Let me know if you have ideas and/or if you'd like to help staff the table.

BigTable blues

This was supposed to be the blog entry where I would announce the Geek Event Aggregator's successful port to Google App Engine.

(sigh)

I've read an awful lot of buzz about how GAE's BigTable is the Next Big Thing in data, makes RDBMS obsolete, etc. Maybe I'm just doing it wrong, but right now I am utterly unimpressed.

The Geek Event Aggregator needs to search its database of 5000 or so events for events whose longitude and latitude are close enough to the user to be of interest. Does that sound so impossible?

I couldn't do it in GAE. First, "Inequality Filters Are Allowed On One Property Only" - so I can filter for longitude or latitude, but not both. I had to filter only for longitude, pull all resulting records into the application, and finish boiling the ocean in my app. It was slow, in the local application environment, but I hoped it would run faster once uploaded to the actual GAE production servers.

In production, though, it doesn't run at all - "Timeout: datastore timeout: operation took too long.". Querying from 5000 records - too much for the mighty BigTable, apparently. Dropping the filters on longitude (to do all the filtering in the app, in case inequality filtering is just so poisonous) didn't help, either.

Oh well. I still enjoyed working with GAE at first, and maybe I'll use it again for something with very light data demands. For the Geek Event Aggregator, I do have a server available where I can host in TurboGears - it'll just take a bit of rewriting. Later this week, hopefully.