templating engine cookoff updated

Ian wished for loop syntax examples in the Templating Engine Cookoff, so I just added them.

And none of you Django programmers have given me code to include Django templates in the cookoff. You can't throw an exception in this town without hitting a Django programmer, so what's up? C'mon, it's easy - I just need something analogous to this:

The answer to the Ultimate
Question is ${everything['universe'].answer}.

Among our weapons are: 
  % for weapon in weapons:
  ${weapon}              
  % endfor

import mako
import mako.template
class MakoDemo(Demo):
    name = 'mako'
    version = mako.__version__
    def demo(self, vars):
        self.template_name = 'mako.txt'
        template = mako.template.Template(filename=self.template_name)
        return template.render(**vars)

[EDIT: Well done, Django folks! Django is in the cookoff now. Except... hmm... it reacts to missing variables by leaving blanks instead of throwing an exception... so I need a new trick if I want to actually see the Django error stack.]

dbapiext.py

Richard Jones alerted me to Martin Blais' dbapiext.py, and I like it very much.

Richard's post focuses on the improvements to bind variable handling. I love dbapiext.py for another reason: it provides query results as namedtuples, a class so perfect for database resultsets that they make me sob with relief.

>>> import cx_Oracle, dbapiext
>>> connection = cx_Oracle.connect('username/password@SID')
>>> results = dbapiext.execute_obj(connection, 'SELECT * FROM crew')
>>> row = results.next()
>>> row
Row(ID=1, NAME='Kaylee Frye', ROLE='Mechanic')
>>> row[1]
'Kaylee Frye'
>>> row.ROLE
'Mechanic'

When memory isn't an issue, I like to transform the result into a list and tack on a simple equality filter method for convenience.

class Filterable(list):
    def filter(self, **kw):
        for itm in self:
            for k in kw:
                if not (getattr(itm, k) == kw[k]):
                    break
            else:
                yield itm    
        yield StopIteration     

>>> result = Filterable(dbapiext.execute_obj(connection, 'SELECT * FROM crew')) 
>>> result.filter(NAME='Jayne Cobb').next()
Row(ID=5, NAME='Jayne Cobb', ROLE='Public Relations')

It is, of course, no SQLAlchemy, and isn't meant to be. It's meant to be a roadmap toward a less awful DB-API for Python. (I still wake up in the middle of the night, gasping, "Five incompatible bind variable formats! It's heresy!") In the meantime, dbapiext.py is a nice lightweight layer for times when SQLAlchemy is too much. For instance, I write lots of database administration tools that poke around in lots of different views of Oracle's data dictionary, and setting up SQLAlchemy mappings for each of them is a pain. dbapiext.py is perfect for that. Thanks, Martin!

Penguicon wrap-up

Thanks to the wonderful audience at my Ren'Py talk. Your patience and your questions got us through the painful time of waiting for the projector to work... bless you all! I hope you have fun with Ren'Py. I've posted my slides and sample game in the Ren'Py forum * here *.

Oh, and I saw a flashbulb during the talk... if you're reading this, could you send me a copy of the photo? I don't have a good recent photo of myself presenting. Unless it sucks, that is. :)

Some of Penguicon's other highlights, from my POV:

Adrian Crenshaw's talk on Mutillidae was delightful. I am so tempted to quit my job and become a security researcher; they have all the fun. He tipped us off to some Firefox extensions for conveniently testing websites for some common vulnerabilities, too.

Matt Arnold's talk on Inform7, which showed me wonderful tools the Inform IDE comes with. It's amazing to see how much work has gone into a language that has such a specialty purpose. I hope I'll find time to put some more work into Inform7 games. I loved this exchange:

Attendee: Is there a limit to how many [options] there can be?
Matt: No! The limit is your sanity!

The dance. I love dancing with nerds. It's so much more fun when the cool kids aren't there. I miss the good old days, though, when we danced to songs, instead of to interminable dance beats. Ah well.

During Mark Ramm's talk about implementing Sourceforge's new TurboGears/MongoDB infrastructure, a questioner invited him to complain about strict schedules for completing a job. Instead, he said that he likes demanding, inflexible deadlines; having flexibility on a project's scope is how you get things done. I thought that was a wondefully wise point of view. A flexible timetable can be an excuse for not triaging a project's scope; a firm deadline helps you make those necessary decisions and actually get something done.

Dayton Oracle meeting

DAY-O, the Dayton Oracle user group, is meeting May 4 - a week from this Tuesday. We've got a full-day meeting planned with lots of great content... see daytonoracle.org for full details.

09:00 AM - 09:30 AM Meeting Registration / Door Prize Raffle Registration
09:30 AM - 10:00 AM Sun / Oracle Merger Update - Gary Hall (Sun / Oracle Corp.)
10:00 AM - 10:30 AM Leading Spaces and Their Diagnosis with Toad for Oracle, A Toad® for Oracle Case Study - Michael Allen (SmartDBA.com)
10:30 AM - 10:45 AM Break
10:45 AM - 11:45 AM Oracle Fusion Middleware 11g Update / Roadmap - Eric Rudie (Oracle Corp.)
11:45 AM - 12:40 PM Lunch - Pizza
12:40 PM - 01:25 PM Exadata 2 Database Machine - Gary Hall (Sun / Oracle Corp.)
01:25 PM - 01:30 PM Break
01:30 PM - 02:30 PM Security and Compliance with Oracle Database 11g Release 2 - Eric Rudie (Oracle Corp.)
02:30 PM - 02:45 PM Announcements / Raffle

Thanks to Matt Morrisey and Vicki Blommel for their organizational work, and to Oracle for being our meeting sponsor!

Bugfixing at PyOhio

Jesse Noller wants to know what's holding you back from working on Python.

He's looking to make the process of contributing to Python more smooth and self-explanatory, less intimidating. That's a good discussion, and I'm not sure what to tell him right now, but it does bring to mind something I really want to see.

Let's have a Python Bug Fixing Tutorial at PyOhio!

Walk everybody nice and slooooowly through the process of browsing through the Python bug tracker (for bugs in Python itself, in its standard library, etc.), checking out code, fixing and testing, making a diff file, submitting the fix. Don't assume that people are familiar with VCS, diff files, the test suite, etc. - all those tools and jargon that separate veterans from intimidated users. Let's break down that separation.

Our Call for Proposals is open right now (through May 10). What a coincidence! I'm not actually on the program committee, so I can't promise that such a proposal would be accepted, but let's just say that the committee will give it serious consideration if they know what's good for them. :)

I'm asking you to propose this, not me, because I am not crazy enough to think that I can lead such a tutorial while chairing the conference; especially since I've never actually done it before myself. Actually, maybe there's one way I could do it: taking the role of the leader/learner in a "Teach Me Twisted"-style you-teach-me session. In that case, I'd want commitments from... hmm... let me say 3+ experienced bugfixers to talk me through it in front of a live audience.

One way or the other, let's see to it that PyOhio transforms some Python users into contributors!

Pythonic PL/SQL

Pythonic PL/SQL exists!

https://code.launchpad.net/~catherine-devlin/pythonic-plsql/trunk

I imagine it will grow in fits and starts as my job needs dictate, but it shouldn't ever be abandoned, since I'm incorporating it gleefully into my own production code.

So far, format (Python 3-style string formatting) and join are available. I'm not using object-oriented PL/SQL at all (I find it very unsatisfactory), so instead of
','.join(mylist), you use pythonic.join(',', mylist).

I'm very open to contributions from others! There's plenty of room to participate, since implementation has barely begun.

There are Quest Code Tester tests for it, too, but unfortunately that's a closed-source project and I have to check on what the rules are for releasing tests produced with it.

Of course, most hard-core PL/SQL developers will already have written or found code to handle basic, common operations like this. But, by sticking close to names and APIs we're familiar with from another increasingly well-known language, I hope to reduce the amount of brainspace you need to remember how to do it all.

Usage samples::

   
    DECLARE
       crewlist pythonic.string_list
             := pythonic.string_list ('Wash', 'Kaylee', 'Mal');
       crewdictionary   pythonic.string_dictionary;
       curs             SYS_REFCURSOR;
       template         VARCHAR2 (100);
    BEGIN
       DBMS_OUTPUT.PUT_LINE (pythonic.join ('; ', crewlist));
       -- output: "Wash; Kaylee; Mal"
       OPEN curs FOR
            SELECT name FROM crew ORDER BY name;
       DBMS_OUTPUT.PUT_LINE (pythonic.join (' *** ', curs));
       -- output: "Kaylee *** Mal *** Wash"
       template := '{2}!  {1} says we''d better replace that catalyzer!';
       DBMS_OUTPUT.PUT_LINE (pythonic.format (template, NULL, 'Kaylee', 'Captain'));
       -- output: "Captain!  Kaylee says we'd better replace that catalyzer!"
       DBMS_OUTPUT.PUT_LINE (pythonic.format (template, crewlist));
       -- output: "Mal!  Kaylee says we'd better replace that catalyzer!"
       crewdictionary ('First Officer') := 'Zoe';
       crewdictionary ('Public Relations') := 'Jayne';
       template := '{First Officer}: (to {Public Relations})  "I can hurt you."';
       DBMS_OUTPUT.PUT_LINE (pythonic.format (template, crewdictionary));
    -- output: Zoe: (to Jayne)  "I can hurt you."
    END;

P(ythonic)L/SQL?

I'm itching to begin creating a bunch of PL/SQL packages that will emulate certain handy Python features. Before I sink too much time into it, does the LazyWeb have anything to recommend? Or, perhaps, volunteers to help make the dream come true?

I want to start with a package for Pythonic string manipulation:

'The {0} brown fox jumped over the {1} dog'.format('quick','lazy')
.join()
.split()
.splitlines()
... and so forth

I think this package would not just implement the ones that don't exist in SQL or PL/SQL, it will provide Pythonically familiar names and interfaces to the ones that do exist.

Granted, there's only so Pythonic you can get in PL/SQL. A perfect set of packages would be a heroic task. Still, it's easy to imagine helping PL/SQL to suck less...

templating engine cookoff

There are more Python string templating engines in the world than homemade chili recipes. Which is tastiest? Let's have a cookoff!

We all have different criteria, of course. The only organized comparisons I've seen between them are based mostly on measuring speed of rendering. Huh? I have never said to myself, "Wow, I'm wasting so much time because these templating engines aren't rendering fast enough. If only they would finish in 0.4 seconds each instead of 0.6!"

I have said to myself, "Wow, I'm wasting so much time because there are errors in my templates, and my templating engine isn't helping me find them." Writing templates is tricky. Errors are common. So, for me, clear error messages are the key virtue. My cookoff focuses on recognizing the engines that report the filename and the offending string when a rendering failure occurs. It also displays a typical simple syntax for writing and using each type of template (so it can serve as a cheatsheet as well).

My results? So far, only jinja2 and genshi report both filename and the error-causing string. The jinja2 error stack is more concise overall, so I award it the prize by a nose.

I'm really happy with the framework I wrote for doing the cookoff, too, so I posted it on Launchpad. It is really, really easy to add new engines for comparison - feel free to suggest or submit them! In fact, I hope that others can adapt my cookoff framework to other situations that call for side-by-side package comparisons.

Incidentally, why are comparisons like this usually styled "smackdowns" or "shootouts"? What's up with that? We're not trying to put anybody in the hospital! We're just trying to select our favorite from a banquet of delicious offerings. Eat up!

I'm speaking at Penguicon

Ren'Py: A Visual Novel Engine

Computer gaming, anime/manga, Python programming, fiction writing: Ren'Py is everything you love about Penguicon rolled up into one. It's an open-source visual novel engine that lets you program graphical games and stories with the greatest of ease. Creating basic linear or branched stories is almost as easy as writing them on paper - yet Ren'Py is also a full-fledged Python environment, so there's no limit to what you can program as you learn. This intro will show you everything you need to start creating your own masterpiece. Ikimashou, otaku-san!

May 1, 2010, 12 noon - 1 PM

Detroit Marriott Troy

200 W. Big Beaver Rd.

Detroit MI 48084

YOU: Okay... but will you give the presentation wearing a homemade Elizabethan-era dress?
ME: Yes!
YOU: Cool! Um... why?
ME: Because this is Penguicon!

What's Penguicon? It's an open-source software conference... it's a science fiction fandom conference... it's an awesome hybrid of the two! Super-sharp technical people come for some awesome knowledge sharing, and the SF fans help make the atmosphere even more relaxed and imaginative than at a normal FOSS con. I love both sides of the con and especially the ways they mix together, which is why I wanted to give a talk this time that would really encompass it all. (Costumes are totally optional, by the way. But they sure are fun!)

(Psst - if you install Operator, you'll be able to automatically process the microformatted data in this post and add it to your calendar. Cool, huh?)

See you in Detroit!

PyOhio Call for Proposals

PyOhio's http://www.pyohio.org/CallForProposals is out, due May 10. Let your imagination run wild; PyOhio is what we make it, no more and no less. Help us make this year's PyOhio the best yet!

daytontechevents

daytontechevents.com

A free service aggregating information about tech events in Dayton

There's too much geeky awesomeness going on in town to track with your brain alone. daytontechevents fixes that. If you're in or near Dayton, keep an eye on it so that you don't miss something fantastic! If you know of a group or event around here that isn't yet on daytontechevents, add it!

Thanks to Sarah Dutkiewicz, who got the idea and put the infrastructure in place not just for Dayton, but also for Columbus and her hometown Cleveland. Ohio thanks you, Sarah!

Pooled nonprofit IT

File this one under "Daydreaming".

There are lots of small, local nonprofit organizations that really can't afford to do their IT well. They can't afford a full-time professional, so they make do with unskilled labor, plus a smattering of donated services and what short-term contractual help they can afford. Each time, the solutions implemented depend on the skills and tastes of the implementor, and rarely build usefully on what has been implemented before.

Imagine the efficiencies if many organizations could share a small, full-time IT staff dedicated to their needs. Most organizations share many of the same needs, and could share the same set of solutions. A full-time pro would become familiar with those needs and could efficiently apply a well-known solution to each group. (S)he would provide continuity, well-informed decisionmaking, and perfectly specialized skills.

Providing this would seem like a natural role for, say, an organization like the United Way; or perhaps for a small meta-charity created for the purpose. I'm sure private firms have attempted to set themselves up in this role, but anything that demands a steady funding outflow from the benefiting agencies is a tough sell, no matter how beneficial it would be in the long run.

I'm just daydreaming, but I'm interested in people's comments. Has anybody heard of something like this being done?

Community Service Award

Facundo Batista and I received the 4th-quarter Python Software Foundation Community Service Awards!

I actually heard about it several days ago, and have been tongue-tied (keyboard-tied?) since then. I'm mostly seen in and around Ohio these days, but I grew up thoroughly Minnesotan. If you pay Minnesotans a really extravagant compliment, you'll send them into a kind of blushing panic. All of our ancestors who enjoyed the limelight were found and eaten by bears, you see, so we evolved a genetic terror of having attention called to us.

Anyway, this is one of the most flattering honors I've ever received. THANK you, PSF! I hope I'll always be one of the Python world's many valuable contributors.

cmd2 0.6.1

Many thanks to the audience at my PyCon cmd/cmd2 talk for your interest and enthusiasm! It was my first full-scale PyCon presentation and I absolutely loved it.

I need to follow up on three things I claimed in my talk:

1. "My presentation is already online at http://pypi.python.org/pypi/cmd2". FALSE (at the time I said it, and for several days afterward). I actually had posted the docs and edited the PyPI page to point to them, but forgot to update url in setup.py, so it overwrote the link when I registered cmd2's 0.6.0 release.

It's fixed now, though. The cmd2 PyPI page has a link to the cmd2 documentation, which in turn links to my talk slides. You can also watch the talk thanks to the fantastic PyCon video crew!

2. "A more stable version will be out within a couple weeks of PyCon." TRUE. 0.6.1 is not exactly stable stable, but I think I've smoothed out bugs that snuck in while I was pushing to release 0.6.0 for PyCon.

3. "sqlpython will be more presentable in a couple weeks, too." TRUE. The new sqlpython 1.7.1 brings the postgreSQL functionality (thanks Andy!) to pretty near 100% (except for the can't-see-other's-schemas problem, which should be fixed for 1.7.2.) I believe that MySQL should be fully functional, too, but that's very lightly tested because I barely use MySQL and don't know much about how to test it.

Computer Engineer Barbie

I'm happy about Computer Engineer Barbie. No doll will save the world, but every little bit helps.

It's annoying, though, that some people only see one more opportunity to trot out this classic:

def belittle(geek):
    prejudice = "Well, then, she's not much of a {0}, is she?"
    if geek.is_femme:
        excuse = 'geek'
    else:
        excuse = 'woman'
    return prejudice.format(excuse)

sqlpython 1.7 pre-release

First off: PyCon online registration ends TODAY. Go do it NOW.

I intend to show off sqlpython at PyCon. Version 1.7 works with postgreSQL and MySQL as well as Oracle. (In fact, a single sqlpython session can keep connections to all three types of databases open at once!)

1.7 isn't quite ready for PyPI yet; I've just posted an appeal for sqlpython testing, with instructions for getting the development version, to the sqlpython google group. If you'd like to take the new-and-improved sqlpython for a spin around your databases, I would hugely appreciate it. Anticipate a PyPI release just before PyCon, once the ugliest of the bugs are worked out.

See you in Atlanta!

Hey, Oracle!

You're scaring me!

In the last couple months, we've learned that the new Sun will be going forward without Frank Wierzbicki, the Jython project lead, and now without Ted Leung.

The Oracle Technology Network is working hard to foster dynamic language use with Oracle. It's got publications, PyCon sponsorship, resources, and so forth. OTN delights me.

I'm afraid, though, that the larger Oracle corporation doesn't share OTN's interest. Oracle's absorption of Sun is proceeding without any apparent interest in dynamic languages. Oracle is discarding some of the finest talent it could possibly acquire, people who could have helped bring on a real flowering of dynamic language use in Oracle environments.

I, for one, would have loved to see Jython harnessed on Oracle's many Java-based tools and even the JVM in the Oracle database - imagine the wonders Frank and Ted could have worked with that, had Oracle assigned them to! Instead... there will be nothing Oracle-related from them. Meanwhile, Microsoft funds development of several dynamic languages, including IronPython, which integrate to SQL Server through .NET.

I don't know. I just dread the thought that, five years from now, SQL Server will be the proprietary database of choice for any environment where dynamic languages are used... which will soon be most environments. Oracle, you really want to give this market away?

How to write a bad abstract

Ironically, the only talk that disappointed me at all at CodeMash was... mine. I thought my delivery was a little rough - pretty much because it's a really bad time of year for me to practice a talk.

I had a (great, attentive, but) smallish audience, too. One of the reasons was simple scheduling - there were some amazing talks opposite me. I was tempted to slip out and watch one myself! The biggest problem, though, was me. I wrote a bad title and abstract for the program guide.

reStructuredText: Plain Text Gets Superpowers
Technology/Platform: Python
Difficulty Level: Beginner

Abstract: Write a document, just once, in plain text. Enjoy all plain text's benefits - speed, simplicity, scriptability, version control. Now, from this single plain text source, automatically generate beautifully-formatted webpages, presentations, .PDFs, auto-indexed documentation trees, and more. It's time to quit slacking on documenting your software. reStructuredText will make you actually enjoy writing docs!

What's wrong with this? It's a nice narrative that fits together and builds to a nice little conclusion... but it's bad for a program guide.

I assumed attendees would read the whole abstract carefully. Then, when I got to CodeMash, what did I do to select sessions I would attend? I quickly scanned over the list of titles, and based on that, hastily read some of the abstracts. I forgot that geeks are pressed for time, especially at conferences!

There were three important messages for attendees to know to decide whether to come:

1. This is a talk about writing documentation
2. It is independent of what programming language you use
3. This talk provides a helpful tool (i.e., it's not just a preachy sermon)

You have to read all the way to the end of my abstract to get messages #1 and #3, and #2 is never communicated clearly at all. So, what would a good title and abstract have looked like?

Your Docs Won't Suck Anymore
Technology/Platform: Other *
Difficulty Level: Beginner

Abstract: No matter what programming language you write in, it's the English language that's killing you - your lack of good documentation is driving potential users away. reStructuredText is a technology and a family of tools that will make writing documentation easy, powerful, and satisfying. We'll introduce reStructuredText and get you started on writing beautiful documentation for any program or language.

* - "platform-independent" would have been the best choice for "Technology", but it wasn't in the dropdown. I worried that choosing "Other" would imply reST was implemented in Scala or something. I should have done it anyway, though, because most CodeMash attendees don't know Python and probably read "Python" as "not for me".

The take-home lesson here: when writing titles and abstracts, be very mindful of the divided and rushed attention of the typical geek. Don't be coy and save the good stuff for the fine print. Get your message across up front.

Fear is not a virtue

I know there have been a million news items like this, but this is the one that puts me over the edge.

SAN DIEGO — Students were evacuated from Millennial Tech Magnet Middle School in the Chollas View neighborhood Friday afternoon after an 11-year-old student brought a personal science project that he had been making at home to school, authorities said.

Maurice Luque, spokesman for the San Diego Fire-Rescue Department, said the student had been making the device in his home garage. A vice principal saw the student showing it to other students at school about 11:40 a.m. Friday and was concerned that it might be harmful, and San Diego police were notified.

The school, which has about 440 students in grades 6 to 8 and emphasizes technology skills, was initially put on lockdown while authorities responded.

Luque said the project was made of an empty half-liter Gatorade bottle with some wires and other electrical components attached. There was no substance inside.
...
Luque said the project was intended to be a type of motion-detector device.
...
Both the student and his parents were "very cooperative" with authorities, Luque said. He said fire officials also went to the student's home and checked the garage to make sure items there were neither harmful nor explosive.

"There was nothing hazardous at the house," Luque said.

The student will not be prosecuted, but authorities were recommending that he and his parents get counseling, the spokesman said. The student violated school policies, but there was no criminal intent, Luque said.

I finally understand Al-Qaeda's master plan, and it's freaking brilliant. Resenting American technological dominance, they have found a way to end it, convincing us to semi-criminalize technical curiosity and thus lobotomize our culture. I'm just surprised that we're choosing to participate in the plan. I thought we were on opposite sides?...

Is there a political movement or group I can join to fight this? Common-sense people grousing about how stupid each incident is is failing to hold back the tide. Something vaguely like the EFF but with a broader scope - because all science and technology is under attack now.

Fear is not patriotic. Fear is not a public service. Fear is not a virtue.

MongoDB

Gloria Jacobs told me about MongoDB at PyOhio, but I was too busy conference-chairing to see her talk, and time has flown by. Her enthusiasm prompted me to see Mike Dirolf's MongoDB talk at CodeMash, though, and wow. Thanks, Gloria and Mike! I like it, I really like it!

My frustrating experience with BigTable had given me a "Bah, humbug!" attitude toward the NoSQL fad, but it really looks like MongoDB is the cure for that. It surrenders much less query capability than the other NoSQL contenders do. The simplest of those are useful only if you already have the key for your desired record in hand, and BigTable's limitations make it feel only moderately better to me. But MongoDB's query capabilities are really rich, good enough for many (though of course not all) real query needs.

Now, don't get me wrong; there are a *whole lot* of tasks for which an RDMBS is still very much the answer. When you need transactions, or child items that aren't tucked neatly under single parents, or complex queries - and how often do you really *know* that you'll never need complex queries? - it's safer to use an RDMBS.

I think that, when the database is an enduring construct, important in itself - when multiple applications may be written against it, and new applications yet unforeseen may appear in the future - then a good RDBMS is the only way to go. In such cases, it's just impossible to safely predict what you'll need to do with the data one day, so you need database software that can do virtually anything.

But when the database will play a supporting role to a single, well-defined application, and will not outlive the application. then a non-relational database could be very convenient, and MongoDB looks to me like a fantastic choice.

Let's call this Devlin's Doggy Directive of Databases:

If the application is the dog, and the database is the tail, consider a non-relational database.
If the database is the dog, and the application is the tail, stick with a relational database.

If you doubt that I'm qualified to go naming rules of thumb after myself, let me remind you that have ten years of relational database experience, a sparse smattering of non-relational experience along the way, and that my parents owned a boarding kennel when I was young.