microfinance gift

Cash is almost always the most practical gift you can get somebody. I have trouble not feeling that it's too crass, though. Here's one possible solution.

Microplace, my favorite microcredit site, recently introduced a Gift feature where you can give an investment to somebody else. You pay the money, it goes to the developing world as a loan, and the interest and eventual repayment go to your gift recipient. Everybody wins!

This makes the most sense when you know somebody will need the money more later than they do now; it's a nice way to put the money "in transit" to them - and it does some good on the way. For instance, my sister is facing a gap in her income for maternity leave this spring; I chose a gift that would mature then. It was a perfect solution! Future students are other obvious recipients. Seasonally employed people... people planning an upcoming major trip... etc.

(Incidentally, another great microfinance site is Kiva, and there's actually a Pythonistas' lending team.)

Windows vs. Linux for Oracle

I found myself fielding a "Windows vs. Linux for Oracle" question on StackOverflow, and realized that my answer was detailed enough to deserve a blog entry... my humble contribution to the great war. (I know most experienced Oracle DBAs are chuckling, because "Which OS is best?" is more likely to make them think "Linux or commercial UNIX?")

Note that I'm speaking here of my preference strictly for Oracle-hosting purposes, and that everything I say about "Linux" really applies to all POSIX-based systems (like Solaris, which I started on as a DBA).

I've used Oracle on Windows and Linux for many years, and on Solaris many years ago. I prefer Linux because:

1. Oracle releases patches, new versions, and sometimes security updates for Linux significantly before they are available for Windows - there's usually about a two month lag for Windows (one month for Critical Patch Updates).
2. Our Windows servers have crashed or locked up occasionally, and very frequently require reboots for patch installation. Oracle itself stays up very nicely, but Oracle can't keep running on a machine that is down. This hasn't been a problem for me on Linux.
3. Oracle's interaction with Vista's User Access Control is a nightmare. I'm constantly finding that the dedicated Oracle user account, which was used to install Oracle, nonetheless lacks permission to edit or even see Oracle-generated files - like newly generated logfiles. It could be that I'm making some mistake, but permissions shouldn't be confusing; and on Linux, they aren't. (Most servers don't run Vista, but I'm afraid of what this forebodes for future versions of Windows Server.)
4. Thanks to the Windows Registry, cleanly removing an installation of Oracle from Windows is tricky and tedious. The Oracle Installer has gotten better at this since version 10g, though.
5. Better tools. Linux find is infinitely better than any native Windows search tool. It can take you minutes on Windows to track down what Linux which tells you in an instant. Also, Oracle uses and generates plenty of plain-text files, and Linux comes with better tools for handling text files - good text editors (unlike Notepad), shell commands like head, tail, grep, etc. You can try to catch Windows up by installing Geanie, Cygwin, Google Desktop, etc. on a Windows machine, but it's better not to have to (especially since Cygwin installation is not completely newbie-friendly).

I can only think of two Windows advantages over Linux:

1. In Oracle's command-line tools like sqlplus, rman, etc., you can scroll through and re-run past commands using the up- and down- arrow keys - but only on Windows. You can fix this on Linux by installing rlwrap and always invoking the Oracle tools under rlwrap: rlwrap sqlplus me@myinstance
2. Quest's TOAD is only available for Windows, and it is an infamously useful tool. It doesn't need to be on the database server, though, just on a machine that can connect to the database. Also, Oracle's free sqldeveloper is one among several viable alternatives; it will probably never catch up to TOAD completely, but it's good for the bulk of what most people use TOAD for.

In general, however, I should emphasis that Oracle's ancient boast of running on everything really is true. As a DBA, I prefer Linux for the reasons listed above, but to my database users, it's absolutely irrelevant (downtime aside). I could move my production databases to a different OS overnight, and tomorrow, my users would have no idea that there had been a change.

Viva Tortuga

Joseph Lisee, author of the upcoming Python submarine robot PyCon talk, left a comment on my last post. I think he was a little shy about me highlighting him.

I'm sorry, Joseph. You really left me no choice.

Completely Unfounded Rumors About "An Underwater Python: Tortuga the Python Powered Robot"

Roll 1d6 for each hour spent in the Atlanta Hyatt bar.

1. 1. Joseph will announce the release of asimov.py, a pure-Python implementation of the Three Laws of Robotics.
2. 2. Bring a swimsuit and snorkel. One lucky audience member will be picked to join Tortuga in the hotel pool, where Tortuga will take a fish from their hand.*
3. 3. Jozeph 'az been practeeseeng 'eez reedeeculous Jacques Cousteau accent for months and weel uze eet to deeleever zee eentire talk.
4. 4. Several minutes into the presentation, Tortuga will overpower Joseph, throw him from the stage, and announce that humankind is obsolete and has been deprecated.
5. 5. Attendees will be asked to pour out a libation to Poseidon. Any caffeinated beverage may be used.
6. 6. There will be a sprint to construct a tall, dapper companion to Tortuga for communication and protocol purposes.

* - No, Tortuga won't be physically present at PyCon. It's not that portable. Believe me, the program committee asked!

P. S. Blogger, don't you know what an <ol> is? You know, like an <ul> with numbers.

PyCon pre-favorites

When I look over the PyCon 2010 talk list, I'd like to be at about half of them (a physical impossibility, until I master self-multiplexing). Still, these are the ones that I'll move heaven and earth to be at. What about you - what are your favorites?

Extending Java Applications with Jython

I'm hopeful that this can really move Jython from my "stuff I think is cool" box to my "stuff I use every day" box.

IronPython Tooling

This is going to cover development environments and tools for debugging and profiling... pretty much a necessity in the .NET world. I also hope to use the video of this talk in the future in talking to the hordes of programmers around here who live and breathe Visual Studio.

Python in the Browser

Silverlight is way too cool to leave to the C# kids.

Think Globally, Hack Locally - Teaching Python in Your Community

As a local group-leader type geek, I'd love to start some of these Hack Nights.

Dude, Where's My Database?

There were so many proposals for descriptions of non-relational databases - but this one really stands out because it looks at the huge picture, classifying databases by their broad category and highlighting what makes each category beneficial for particular purposes.

Sprox: data driven web development

I confess - I've fallen behind the TurboGears world lately. Nobody's demanded a dynamic web app of me for a while, and TG has moved too fast for me to keep track of it. When last I was involved, Sprox was just emerging. I hope this talk will help me catch up.

Revisioned Databases for MultiUser Editing

Revisioned databases are an interesting concept, and seeing how one was actually developed should warm my datageek heart.

Easy command-line applications with cmd and cmd2

Interactive command-line interfaces were good enough for ZORK, and they're good enough for you! cmd and cmd2 make them crazy-easy. (I'll get in trouble if I don't go to this one, since I'm the speaker.)

Dealing with unsightly data in the real world

Gathering data from disparate, chaotic sources is a big part of pretty much everybody's life. I'm eager for any new insights.

An Underwater Python: Tortuga the Python Powered Robot

because, deep down inside, people everywhere are the same; we all want to be loved, and Python-powered robot submarines.

Configuring Oracle Data Guard

The official Oracle Data Guard docs are, of course, the most complete and accurate source of information about setting up Data Guard.

They're not very easy to use, though. They don't provide a walk-through of the entire process, for example, instead branching the discussion at every possible decision point.

I just fought my way through the process, with help from Chris Ruel of Perpetual Technologies, and thought I'd record my steps for the benefit of humankind. OK, that's not true - it's actually because the Internet is the only place I can leave myself notes and be certain to find them again later.

It's too bulky for a blog post, so here: Oracle Data Guard Configuration Walk-Through

pernicious python.org proxy problem

For the past few weeks, I haven't been able to access python.org or any of its pages through a proxy server. My workplace has one standard proxy server, and I also use a personal machine as a SOCKS proxy for an SSH tunnel - and both of them have been getting name resolution errors for all python.org sites. I haven't seen it for any other sites, or when using no proxy.

Does anybody know what's going on? Is there something about python.org that would make name resolution work differently for it?

[EDIT: Friends from the Dayton Dynamic Languages SIG figured this one out. My primary workplace proxy server is blocking DNS lookup on the python.org domain. Trying to use a different proxy through a SOCKS/SSH tunnel produced the same DNS failure, because - to my surprise - by default, Firefox does not make its DNS requests through the SSH tunnel even when all other traffic is tunneled. The network.proxy.socks_remote_dns preference must be set to change this. See "Proxy Firefox through a SSH tunnel".]

A PyCon program committee volunteer reflects

The PyCon program committee has finished its work. All submissions have been reviewed, debated, argued on, and voted - often through several cycles. Emails accepting and declining talks have gone out.

I wanted to blog about my impressions as a program committee volunteer. Note that this is totally unofficial, and I'm not speaking on behalf of the committee, PyCon, etc.

It was wonderful

Just reviewing the talks was a great experience. Some of the talks were fun just to visualize; watching them will be even better. I learned lots about what is going on in the Python community. The program committee is a smart and fun crowd to work with, too.

It was horrible

The problem with a programming language that can do pretty much anything is that three days of scheduled talks are nowhere near enough to see everything that's going on. I wish we could have five days of talks, but there are too many people who wouldn't have the time or money for such a conference.

With room to accept fewer than half of our submissions, we had to turn away talks that would have been great. For instance, the one talk I most wanted to see - the proposal I would have walked barefoot to Atlanta for - got declined. What can you do? Often "pretty much everybody was pretty excited about this" talks had to be sacrificed for the sake of "virtually everybody was dying to see this" talks.

But wait, there's more

Fortunately, scheduled talks are only the tip of the PyCon iceberg. We have Lightning Talks, Open Spaces, and (new this year) poster board sessions! I hope all declined speakers will consider taking their material to one or more of those formats!

PyCon is going to be wonderful

I think we have the best crop of presentations we've ever had. If you can look through the list of accepted talks and not start making Atlanta travel plans, then you are already dead.

PyCon is going to be horrible

... because, with five simultaneous tracks packed with the very best of material, I promise you will face multiple can't-miss talks going on at the same time, all day, every day. The painful decisions of the program committee are really only a preview of the difficult decisions every attendee will have to make at the conference itself.

For 2011: increasing your chances

If you want to make your future PyCon proposal more appealing to the committee (making our decisions even harder - thanks a lot), here are some of the things I saw that helped talks make the cut.

• The basics: a clear talk description, orderly-looking outline, plausible-looking timings. If reviewers ask questions, answer them. Give every impression that you're prepared to put serious effort into your talk.


• Broad appeal. It's OK to present on specialty topics, of course, but if you can point out ways that even people outside the specialty will also want to see it, it will help.


• Unusual topic. Every year, there are some hot topics that everybody in the community seems to be talking about... and submitting talks on. Since we're not going to accept a dozen talks on any topic, no matter how hot, these talks need to prevail over a lot of competitors. On the other hand, if you've got a topic that makes the committee say, "HUH? Wow, I'd never heard of anything like that!", it really helps you stand out.


• We always get more intermediate-level submissions than for beginner or advanced, so the competition was fiercest there.


• What will attendees get from your talk that they couldn't get simply from reading the docs? Make sure we can tell.


• Evidence of preparation and skill. Some speakers had established reputations as skillful, engaging presenters; some provided links to their slide decks from earlier versions of their talks given at local groups or regional conferences; a few linked to actual recordings of earlier versions of their talks. Give your talk at a nearby usergroup, then convince one of your group members to volunteer for the program committee. :)


• Scratch the itch. When committee members say, "Ah, yes, I've been puzzled by that and dying for a proper explanation!" - or, "I personally understand it, but I see misunderstanding of it throughout the community and wish somebody would help clear it up", that is a big plus.


• Keep the Py in PyCon. If the topic is one of general IT interest - database technology or rich web client programming, for instance - then make sure to emphasize the Python angle of your talk. How do you work the problem from Python specifically? What do Python users need to know about the problem that they won't learn from materials aimed at the IT community overall?

Anyway, my personal thanks to everyone who submitted, and I really hope to see you all at PyCon!

Stop! Drop that field!

For PyOhio registration, we used a nice service called eventbrite. It worked great, but I have one big problem with it: it collected way too much data from registrants. It got us the data we needed, but it also asked for home addresses, gender, job title, company... all data we had no legitimate need for or plans to use, probably just because the fields are in the eventbrite form template. Entering it was pointless nuisance for our attendees, and maybe some were actually put off by the length or intrusiveness of the registration form. (Dave Stanek, if you're reading this, let's see if we can change that for next year.)

We are so not the only offenders in this department. It's everywhere, it's endemic. At website after website, we're asked to provide information of no apparent relevance to the sites' purposes. It's so easy to throw field after field into a data collection form; templates are provided with every conceivable field already in place; and - well, why not? Isn't more data better?

No. No, it's not. Excess data takes time, clutters databases, obscures important data, increases risks of data leakage. In interpersonal interactions, we always have the option of asking "Why do you need to know that?", or just giving people that funny look that tells them they're going out of bounds. On paper forms, we can leave fields blank. Automated forms with field validation cut those safeguards off and open the door to compulsive collection syndrome. The one defense people do have against intrusive electronic forms - lying - ruins data quality, and false data is much worse than no data at all.

We need a ethos of restraint in data collection, of always asking, "Why am I collecting this field?" Data collection needs to be seen as something that is not pure good, but something that has a cost to weigh against the benefit. Not collecting data is often the responsible choice, and we need to teach each other that.

PyCon talk review

We've got a record number of volunteers working on PyCon's Program Committee - the group that reviews talk proposals and decides which ones go on the schedule. And it's a good thing, because we've also got a record number of proposals - 179! (For comparison, PyCon 2008 got 118.)

Right now, we're in the fun part - going through the proposed talks and yelling, "Oooh! Ooooh! I want that one!" Just looking through the proposed talks is a great Python education all by itself - you find out about useful packages and techniques you'd never known were out there.

The tough part comes later - when we have to winnow the list down. Without exception, there are talks I want to see that won't make the cut. Accepting all the good talks would be great, but we'd need a week of PyCon, and three days for the core conference are all we figure most attendees can spare. (My proposal for a round-the-clock talk schedule met only chuckles. Then again, with late-night Open Spaces, we already come dangerously close to a round-the-clock schedule...)

Ohio LinuxFest

Wooo, Ohio LinuxFest!

My reStructuredText slides are at catherinedevlin.pythoneers.com, down at the bottom of the page. Thanks to everybody who attended and gave great feedback!

I arrived Friday morning this time and spent a good chunk of the day at the Hackathon, working with Mark Borgerding on his idea for a new educational game for Childsplay. We made some good progress, especially since we were both 100% newbies to pygame! I enjoyed myself and learned a thing or two. Hopefully we'll be able to finish the game up remotely over the next several weeks.

Next came an impromptu lightning talk session (I looooove lightning talks) where I gave an extremely badly-organized (but well-received) glimpse at sqlpython.

I helped Todd Trichler from Oracle Technology Network with his demonstration of Oracle's free offerings. We had a good group, and the next morning Todd gave away his ENORMOUS box of Oracle software in about an hour.

The hallway track was, as usual, excellent. William McVey and Eric Floehr used the PyOhio table to stir up interest in CincyPy and Central Ohio Pythonistas, and Monday's inaugural COPy meeting had 27 attendees! Score! I enjoyed talking with people so much that I found myself on the verge of going hoarse just 90 minutes before my talk. Eek!

I had a great time. Congratulations and thank you to the OLF organizers and sponsors. Once a year, you make Ohio feel like anything but a technology backwater!

Sunday was the Diversity in Open Source workshop, which deserves its own post. To be continued...

Python at Ohio LinuxFest

The Ohio LinuxFest fun starts tomorrow! Here are the Python-related activities there that I know of.

Of the Friday hackathon projects, I believe that schoolsplay and sendoff are Python-based.
http://www.ohiolinux.org/hackathon.html

Zenoss Community day on Friday (Zenoss is a Python product):
http://www.ohiolinux.org/zenoss.html

Python for Linux System Administration - Vern Ceder
10 AM Saturday
http://www.ohiolinux.org/talks.html#PYTHON

reStructuredText - Plain Text gets Superpowers - me
5 PM Saturday
http://www.ohiolinux.org/talks.html#TEXT

PyOhio booth - all day Saturday (though not always staffed). All Python groups should take advantage of it shamelessly - bring your literature! - and anybody who wants to hang around there and have Python-related conversations with people, that's fantastic.

And, of course, don't forget Oracle's event at OLF.

See you there!

easy_install no longer easy on Vista

Since my Windows machine was upgraded from XP to Vista, managing Python packages has become absolutely horrible. Here's what I've puzzled out so far, with much wailing and gnashing of teeth:

1. No matter what rights your primary account has, you need to run easy_install from a Run as Administrator window - otherwise, easy_install runs in a separate window which pops up, flashes some feedback at you for a microsecond or so, then disappears, leaving you with absolutely no record of whether the install works and why. There doesn't seem to be any way to log the results to a file.

2. After installing any module that is deployed as an .egg into site-packages, you need to go and edit its permissions manually to give your account read privileges on the egg. (Giving your account privileges on the whole site-packages directory does not help.) Until you do, import newmodule will fail with ImportError: no module named newmodule on your account - but will succeed when run from a Run as Administrator window.

This is bad news. I fought my way through because I'm a dedicated Pythonista; how many Vista-using Py-curious are going to give up on Python because module installation now requires such hacks?

(Don't forget - Ohio LinuxFest registration ends tomorrow at noon! Move, move, move! You'll hurt my feelings if you don't go.)

Enthought's reStructuredText editor

Enthought has produced a wonderful tool for getting into reStructuredText: a side-by-side WYSIWYG rST editor.


Getting it installed, however, just about killed me. Here are the steps I finally puzzled out for Ubuntu 9.04. Miss any steps - or even change the order - and you'll get error messages that don't help even slightly.

sudo apt-get update
sudo apt-get install python-setuptools python-vtk
sudo easy_install -U numpy
sudo easy_install -U docutils sphinx TraitsBackendQt[nonets] AppTools[nonets]

[EDIT] If you're not on 9.04, or you just want to be on the safe side, it doesn't hurt to sudo apt-get install python-dev python-qt4 at the beginning of the whole process.

The other odd thing is that this lovely editor apparently has no name, and certainly no handy start script or presence in any menu. Mine got installed at /usr/local/lib/python2.6/dist-packages/AppTools-3.3.0-py2.6.egg/enthought/rst/app.py; your best bet to find yours is probably sudo updatedb; locate -r rst/app.py$

Then, set up a bash script to make it usable. I'm calling it "rsted". sudo nano /usr/local/bin/rsted and fill it with:


#!/bin/bash
python /usr/local/lib/python2.6/dist-packages/AppTools-3.3.0-py2.6.egg/enthought/rst/app.py $*


... sudo chmod +x /usr/local/bin/rsted, and live happily ever after.

(The $* in /usr/local/bin/rsted doesn't actually do anything - the editor doesn't seem to accept arguments like a filename - but I'm being hopeful for the future.)

PyCon 2010: Call for Proposals

In the opinion of most attendees I talked to, PyCon 2009 was the best one yet. If you need an excuse to come to PyCon 2010... well, what better excuse could there be than, "I'm speaking"?

---

Call for proposals — PyCon 2010 — http://us.pycon.org/2010/

Due date: October 1st, 2009

Want to showcase your skills as a Python Hacker? Want to have hundreds of people see your talk on the subject of your choice? Have some hot button issue you think the community needs to address, or have some package, code or project you simply love talking about? Want to launch your master plan to take over the world with python?

PyCon is your platform for getting the word out and teaching something new to hundreds of people, face to face.

Previous PyCon conferences have had a broad range of presentations, from reports on academic and commercial projects, tutorials on a broad range of subjects and case studies. All conference speakers are volunteers and come from a myriad of backgrounds. Some are new speakers, some are old speakers. Everyone is welcome so bring your passion and your code! We’re looking to you to help us top the previous years of success PyCon has had.

PyCon 2010 is looking for proposals to fill the formal presentation tracks. The PyCon conference days will be February 19-22, 2010 in Atlanta, Georgia, preceded by the tutorial days (February 17-18), and followed by four days of development sprints (February 22-25).

Online proposal submission is open now! Proposals will be accepted through October 1st, with acceptance notifications coming out on November 15th. For the detailed call for proposals, please see:

http://us.pycon.org/2010/conference/proposals/

For videos of talks from previous years – check out:

http://pycon.blip.tv

We look forward to seeing you in Atlanta!

BLOBs in sqlpython

Obviously, you can't query BLOBs in a command-line SQL tool.

Unless, of course, that tool is sqlpython. Bwa ha ha ha.

reStructuredText talk at OLF

It's official - I'm on the schedule!

reStructuredText: Plain Text Gets Superpowers
September 26, 2009, 5 - 6pm
at Ohio LinuxFest

Greater Columbus Convention Center

400 North High Street

Columbus, OH 43215 USA

Introduction to reStructuredText, a simple single-source format that can generate documents in HTML, PDF, .odt, and many other formats.

I also see tasty-looking talks on "Python for Linux System Administration", a "Sysadmins' Rosetta Stone" talk that should help me port my Ubuntu skills to Red Hat, and gobs more - plus the Diversity in Open Source workshop. This will be a great year!

Spend Like a Pirate Day

Why do we Americans continue to carry around drab $1 bills, struggling to cram mushy, wrinkled paper into vending machine readers, when we could carry gleaming, clinking, golden doubloons?

Admire the gleam and the weight. This is the proper sensory experience for money!

I just found out you can buy boxes of 250 coins directly from the mint. Granted, there's $5 shipping, so you're paying $255 to get $250, but your credit card kickback should cover that. Then you can eschew that lame ATM for months! Let's face it, you only use cash for little purchases these days anyway. Make every cash transaction enjoyable!

When you receive your coins, feel free to run your hands through them several times, purring, "Arrrrrr! Thar's treasure for ye, me mateys!" Do NOT, however, bury them in a sturdy wooden chest and draw a map with a dotted line and an X. I know it's tempting! I want to do it, too! But the point is to get more of these beauties into circulation.

Arrrrrr!

workaround: easy_install windows vanish

My employer-mandated Vista machine has gone almost unused for months because of an infuriating quirk in Vista's command-prompt operation. I've come as close to completely forgetting how to use Windows as I've ever been. Giles Thomas (of Resolver Systems) saved me.

The problem: Vista runs programs like easy_install in a new cmd window, separate from the one they were invoked in. The instant the program terminates, the new window is closed, and any messages it returned - like error messages - are lost. No, redirecting the messages with > and 2> does not work. "Why would you want to see error messages, anyway? They're so geeky and depressing!" *gum snap*

Thus, I was unable to install cx_Oracle, and had to turn to my trusty Ubuntu machine for absolutely every pyOraGeekish task.

Giles blogged a lifesaver workaround. If you can run the cmd window as Administrator in the first place, you are entrusted with the awesome power and responsibility of being allowed to view your own error messages.

So, now I can blow the dust off my Vista machine. Wow, controlling the font size of a cmd window is absolutely as primitive as it was in Windows 3.1. Giles, got a workaround for this, too?

Oracle at Ohio LinuxFest

It's not official yet - so you can't find it at the Ohio LinuxFest website - but it looks like Oracle will be a sponsor and exhibitor this year. They're planning to do an Oracle-on-Linux installfest. If you'd like to get your first taste of Oracle on Linux, sign up for LinuxFest (it's free) and prepare to have a blast.

If you're already pretty good at Oracle-on-Linux and would like to help others get started, send me email! I hope to gather a small group of volunteers to help out at the installfest.

bug reports from the public

From "Dr. Kronkheit and His Only Living Patient":

SMITH: Doctor, it hurts when I do this.
DALE: Don't do that.

Dear Enterprise Rent-A-Car,

That was called a "bug report". It was not actually a request for a condescending message about how I can still rent a car by navigating the website in a different way. I knew that.

I took the time to write up detailed instructions for reproducing the bug as a professional courtesy to your developers. Unfortunately, they will never see my message, since your customer service is managed strictly from a "deal with nuisance customers" point of view. Looks like I wasted a couple seconds of your time as well as several minutes of my own.

RESOLVED: If I ever work on a project large enough to have a customer service department separate from development, I will insist upon bridging this gap. I will make sure customer service has the access, knowledge, and encouragement to communicate constantly with me. I will regard customer service as a valuable conduit for end-user feedback rather than a distant, uninteresting group of non-colleagues.

logging is ugly

Instrumenting your code - whether with a PL/SQL package like Quest Error Manager as Steven Feuerstein suggests, Python's logging module, or whatever - is an important part of writing good code.

I don't. Not very often, anyway. When I do, I often delete the logging calls as soon as the code is more or less working. The biggest reason is the ugliness.

def days_ago (ndays):
    logging.info('days_ago called with arg %s' % str(ndays))
    logging.info('arg type %s' % type(ndays))
    try:
        ndays = int(ndays)
        logging.info('argument converted to integer %d' % ndays)
        current_date = datetime.datetime.now()
        logging.info('current date is %s' % str(current_date))
        result = current_date - datetime.timedelta(ndays)
    except ValueError, e:
        logging.error('Error converting ndays to integer:')
        logging.error(str(e))
        result = None
    logging.info('returning from days_ago: %s' % str(result))
    return result

EWWWWWW! That is ugly! It completely disrupts the comfortable reading of the code. It buries the actual purpose and actions of the function under a steaming heap of chatter. It offends everything that I value in beauty and readability. What to do?

One solution might be a code editor that would toggle the visibility of all logging calls in a program. You could leave them invisible most of the time, and only look at the logging statements when you have a specific reason to. I can see two problems with that, though.

• The logging calls themselves could get out of synch with the functioning code. This could be partially addressed by having logging calls become visible automatically whenever code adjacent to them is changed.
• This would create code which is readable and beautiful in my editor, but ugly when somebody else tries to read it. Perhaps if we cooked up a convention whereby a header comment could define the suggested hiding of logging calls for each program, and most editors could be trained to recognize and respect these suggestions?

I don't have the answer for this. I'd love to hear ideas.

EDIT: Gary Bernhardt tweets, "Anything that you want your editor to to be able to hide (comments, setters/getters, logging) shouldn't exist".

An interesting idea... how to eliminate logging? A good logging decorator could log arguments, return values, and error messages to any decorated function - and that level of information could suffice, if the functions are very fine-grained. Having to write fine-grained functions is the sort of constraint that might improve programming style, too, much the way unit testing demands well-defined functions. I think I'll try this philosophy and see if I can make it work.

It's not much help for the PL/SQL side of things, though. I'm trying to imagine if there's some way to make an analog to Python's decorators in PL/SQL.

Database comics

Clearly, it's somebody's responsibility to create a Hall of Fame for database-related comic strips. I may as well start gethering them here.





What else is out there?

now that's agile

My favorite anecdote from PyOhio 2009:

William McVey prepared slides for his Sunday PyOhio presentation using reStructuredText and rst2s5, but he wasn't satisfied with S5's presentation quality. He tried rst2odp to generate an OpenOffice Impress document instead, but it failed him.

So he convened a Saturday night sprint on rst2odp at PyOhio. Working past midnight, a small team fixed the rst2odp flaws. William regenerated his slides and presented successfully on Sunday.

Finally, in a mighty feat of recursion, he described the feat in a lightning talk Sunday evening, using slides generated by rst2odp, including a slide that contained the source code of the lightning talk he was giving, including the slide with the source code...

Oracle - Linux - Python tutorial slides (PyOhio)

Todd Trichler from OTN is beginning his half of our presentation while I post... my half's materials are here.

doubleplus PyOhio


>>> 2009 > (2 * 2008)
True


PyOhio 2008 had 93 registrants. PyOhio 2009 has more than doubled that number already. Hooray!

PyOhio registration, schedule, sponsors...

Things are really coming together!

• PyOhio registration is open. 38 people have signed up in just over a day!
• The talk schedule is up. 25 talks, my goodness.
• Oracle Technology Network and Intellovations have stepped forward as sponsors.
• Oracle is also sending Todd Trichler to cooperate with me on a two-hour tutorial on Oracle/Python/Linux. I'm enjoying the preparations so far - I think you'll like it if you have any interest in databases.

don't need no stinking rules engine

There's a whole class of programs called "rules engines". The idea is to remove the details of a process from the hard-code of the program, store them externally, and view/modify them easily. The engine then converts the rules, stored in some sort of custom format, back into an executable form at runtime.

In my experience, Python is an effective rule engine. Thanks to Python's readability, you can store business rules as snippets of Python code - in textfiles, a database table, or wherever you prefer - and business users should be able to read them comfortably. After that, a very lightweight Python program can load the rules and the relevant data and use exec() or eval() to apply the rules to it.

One of my main projects is an example of this. It's program that synchronizes data between two Oracle databases. That sounds easy, but business details complicate it enormously:

• Table and column naming, structure, and normalization differ
• Only some rows are transferred, according to a complex set of business rules
• Only some columns are transferred. Column values are combined, split, truncated, have functions applied, etc. Again, governed by a jungle of business rules
• The business rules change continuallyRules must be documented. Letting documentation get out of synch with implemented rules is very bad.
• Users may demand explanations for each decision made by the program, down to the row and column level

My first take on the problem was a large hard-coded PL/SQL procedure. What a nightmare!

Later, I rewrote the rules as snippets of Python. Each rule is stored a database table along with the dates it takes effect and expires, the person authorizing the rule, and a justification. This readable, self-documenting set of rules can also answer questions like, "Why did things change since last month?".

Unfortunately, I didn't know much about object-relational mappers when I wrote it, so the program has clunky data-fetching code. I'm currently working on a third version of the program that uses SQLAlchemy; the resulting program is very short. Broadly, here's what it does:

- Queries the row-level and column-level rules from their respective tables

- Fetches a row from the local database (ours) and the corresponding row from the remote database (theirs)

- The heart of the engine:

data = {'ours': ours, 'theirs': theirs}

for row_rule in row_rules:

    if not eval(row_rule, data):

        return

for column_rule in column_rules:

    exec(column_rule, data)

Actually, the data dict also includes definitions of a few functions that some of the rules invoke. The function names are chosen to be self-explanatory to business users. For example:

def fiscalYear(inDate):

    if inDate.month > 9:

        result = inDate.year + 1

    else:

        result = inDate.year

    return result



data = {'ours': ours, 'theirs': theirs, 'fiscalYear': fiscalYear}

I suppose it wouldn't be too hard to put the function definitions themselves in among the rules, then include locals() in with data, as long as execution order is controlled (easily done by putting execution_order columns in the rules tables). It hasn't been necessary for my project.

- Now row_rules has eval()-able entries like::

  id  start   end     code                      authorized   reason

   1  1/1/06  7/1/09  ours.funded == "Y"        Bob          I said so

and column_rules has exec()-able entries like::

  id  start   end     code                           authorized   reason

   1  2/2/08          if (ours.value > 1000):        Steve        to annoy Bob

                          theirs.value = ours.value

- Add some logging and a "test-run" capacity which reports on the changes without actually performing them. (It uses sqlalchemy.orm.attributes.get_history() for this; make sure to set autoflush=False if you use this, or intermediate flushes might clear the history.)

I suppose I could try writing a sample rules-engine implementation in simple, general terms, that people could crib from for their own "rules engine" applications. I wonder if that would be helpful, or if just the general idea is enough guidance.

Python for Secretaries

As if I need a new ambition, I've got an itch to create and teach a course called "Computer Programming for Secretaries". Since I got into IT via the secretarial pool, I think I'm the perfect one to do it.

To outsiders, programming has this horribly intimidating aura. You've got enterprisey Software Architects trying to sound professional, academic Computer Scientists telling you that you're oversimplifying the problem, fearsome Hacker Gods strutting their skillz. Lots of people want to make sure you know how smart they are, and that nothing could happen without their planet-sized brains.

but programming is not rocket science

If you want to launch satellites into space, you need to invest your life in the field and be part of a large, well-funded institution. Yes, you can have lots of fun with model rocketry, but you're just playing. You're not actually going to get anything into orbit.

programming is more like cooking

My friend James is a professional chef. Everything he makes involves a bunch of French words, ingredients I've never heard of, and turns out eyes-roll-back delicious. I don't have the ambition to invest the time and effort to cook that well... but I can still roast a turkey. That's what programming is like, especially dynamic language programming. With a lot of skill, you can work miracles - but with a little skill, you can work little miracles. You don't need to go in for the whole hog.

That's what I'd like a class to address. There are programming books aimed at kids, but none that I know of aimed at adult business users. There are people who could write themselves small, useful programs, but who will flee in well-justified terror if you start talking about overriding import hooks. There are people spending hours cutting-and-pasting from one file to another because they don't know how to write a six-line script. There are people could replace some of their daily tedium with just a little dose of Python. There is Resolver One, which is a fantastic way to integrate tiny dashes of Python with everyday spreadsheet work, but it's being used by thousands instead of by millions.

So... yeah. What should such a class include? More importantly, once I'm ready to teach such a class, where do I teach it?

how to tell a geek

Given a choice between spending an hour doing a task manually, or spending three hours writing a program to do it automatically... a geek will write the program, every single time. And, if not given the choice, if explicitly ordered to do the job manually, we'll disobey and write the program anyway. I've heard it said that a good geek is lazy, but I think it's more precise to say that a geek dreads boredom above all else. We'll move mountains to accomplish a task, as long as it's interesting.

This is not nearly as crazy as it sounds, because after we've "finished" a task, without fail, the requestor will return and say, "I know I said that would just be a one-time change, but...", or, "Actually, it turns out we don't need A B C D, we need A B Q D C", or whatever. You will reuse that program, no matter what they say; never throw it away!

NCR

Pardon, oh Citizens of the World who read this, while I go regional for a moment and speak as a Dayton-area resident on news almost certainly irrelevant to you.

The big news here last week was NCR's decision to leave Dayton. Basically, three reasons have been given:

• "the high availability of a skilled work force"
• More direct international flights from Atlanta than from Dayton
• $60 to $80 million in incentives from Georgia

The funny thing about the first reason is... NCR doesn't hire people. (Their manufacturing plants may hire, but I'm speaking of their HQ here in Dayton). Since I came to the Dayton area, I've had IT friends in NCR and have tried to keep up with them. The news from them has always been the same: "We just went through another round of downsizing. We keep wondering when our turn will come." For a company in continual contraction, the benefit of a larger pool of people to not hire seems... um, not clear.

That leaves shorter plane flights for those who fly to Europe - seems a strange reason to move 1,300 people - and a large amount of cash. Many people think Ohio should have tried to outbid Georgia, but that would have to come at the expense of companies that don't threaten relocation - and it begins to blur the line between "private company" and "state-funded entity", anyway.

Moving itself, of course, gets rid of those employees who choose not to relocate. I predict that most Ohioans who choose not to move with NCR will not be replaced; the company will use the natural contraction in place of one of its periodic downsizings.

Anyway, it's sad for Dayton, since the company had such a history here, but that's pretty much what NCR has been about here for years - history. For decades now, growth for Dayton - as for most cities - hasn't come from big, stable, traditional companies but from small companies, appearing and disappearing quickly as new opportunities appear and change and dry up. It's a less predictable business world, but that's the century we're in. No amount of sighing will prolong the 20th century

Python Magazine article

I've got an article in this month's Python Magazine: PyOhio: Planning and Running a Regional Python Miniconference. I try to cover some of the stuff we learned in the course of doing the first PyOhio, for the benefit of people considering staging similar conferences of their own. I feel a little silly impersonating an expert on the topic, since I'm near the beginning of a learning process that never ends - but in the open-source world, it's not being the ultimate guru that's important, it's taking the time to share whatever you can.

Python Magazine is a great publication, by the way - with all the good stuff about Python on the net, you might wonder what's the point of buying a magazine, but their articles are very well-chosen and there's a real advantage to being able to read it away from the computer.

Where did you hear about... ?

I've served as PyCon's volunteer publicity chair the past two years. This year, at my request, the attendee survey had the question, "How did you hear about PyCon?"

Thanks to everybody who took the survey and answered that annoying question. Of course, most answers were along the lines of, "Duh, I've always known about PyCon!"

Basically, the answers helped to confirm that community buzz is what brings most people. (My favorite answer: "Birds.") Problem: not everybody is plugged into the buzz. I know lots of programmers who never read blogs or attend groups, and there are lots more that I don't know because they don't do blogs or groups, or otherwise plug themselves into the community.

What I most need is for everybody who didn't hear about PyCon to answer this question: "Why didn't you hear about PyCon? Where would you have seen a PyCon announcement?" The logistics of doing that survey are tricky, however.

How do you think word spreads among geeks, in this day and age?

Wanted: pictoral Field Guide to Nerds

My grandfather was an amazing man. He seemed to know every living soul in Duluth, Minnesota. He never forgot a face.

I didn't inherit that gene. Remembering faces and names is a huge challenge for me. "Very nice to meet you, Mrs. - wait, have we met before? Oh, Mom! I'm sorry." I can spend fifteen minutes in conversation with someone, and five minutes later be unable to bring their face into my mind. It's frustrating and humiliating. I need technological help.

I'd love a website full of labelled and indexed photos of the inhabitants of geekland, something I could brush up on before a conference, or study afterward to cement the new acquaintances into my memory.

Does something like this exist? Failing that, does anybody have some good ideas for how it could be made? At present, my best idea is for some sort of Flickr mashup.

documentation rant

Everybody knows that open-source software documentation sucks.

It doesn't, however, suck nearly as much as big-company proprietary software documentation, which Steve Holden characterizes as

Threep Nardling
To nardle threeps, select the Threep tab and check the Nardling checkbox.

... and so forth... a beautifully-typeset waste of electrons.

The questions we go to docs for are: Can I nardle a threep via SSH? I tried it, but my threep still isn't nardled. I got an "ERROR: Threep nardling failure" message. What now? If those questions aren't addressed, there's really no point.

These days, I do sometimes see open-source docs that address these questions. Most often, of course, you find answers to these questions in the user community.

Anyway, I suggest that, when users run into trouble, this is the preferable order of responses:

1. Change the program so that it acts as the users expected in the first place.
2. Use program interaction and informative error messages to guide users through problems without having to look outside the program.
3. Put the answer in the documentation - and make it easy to find or it doesn't count! Expecting users to comb painstakingly through 400 pages is not realistic.
4. Respond to individual questions via some sort of support process.

Small-to-medium FOSS projects are the best at responding in this order, probably because the same people - or at least people who know each other - are responsible for writing the program, documenting it, and answering user questions. At Big Software Corp., on the other hand, Support, Documentation, and Development are likely to be completely separate groups. The professionalization of documentation is deadly, because you get reams of nearly identical manuals that are pleasantly laid-out and nicely proofread, but completely unaware of realistic user problems. Support is painfully aware of user problems, but they have no process to tell Documentation and Development, "Hey, users keep getting confused here. Can we change the program and the docs to help them? Please? Because we'd really like to quit getting these questions?" (Perhaps some companies do have processes; I can only speculate, but I do know that I don't see evidence of it.)

Oracle, incidentally, only about halfway sucks in these matters, which is pretty good for a company so large. Their docs generally have some good, non-obvious substance, and sometimes - but, alas, only sometimes - troubleshooting information.

In short, I think you need users' pain to efficiently and accurately become developers' and documenters' pain, so that the causes will get fixed. Small FOSS projects have this kind of pain transfer built-in (when the authors publish their email addresses and invite questions). Everybody else should think about how to make it happen for their product.

managing installation requirements by Python version

I always feel guilty using this blog as my LazyWeb, but it works really well, so here we go again...

I'm trying out Oracle Enterprise Linux 5 - basically a clone of RHEL - and trying to get sqlpython installed on it. OEL5 comes with Python 2.4; sqlpython 1.6.5.1 needs Python 2.5. I could take out the 2.5 dependency in sqlpython, but it uses pyparsing. Pyparsing dropped 2.4 compatibility as of 1.5.2.

[EDIT: It turns out that pyparsing 1.5.2 still works on Python 2.4. It emits a scary error message, during easy_install under Python 2.4 -

except ParseException as err:
^
SyntaxError: invalid syntax

but it installs successfully nonetheless. Thanks to Paul McGuire to pointing that out. So my job in this case is simply to release a 2.4-compatible sqlpython 1.6.5.2.]

I could change install_requires=['pyparsing>=1.5.1'] to install_requires=['pyparsing==1.5.1'] in setup.py, but that locks everybody into an obsolete pyparsing whether they need it or not.

I'd like to install_requires=(['pyparsing>=1.5.1'] if python.version >= '2.5' else ['pyparsing==1.5.1']), but that's absolutely imaginary syntax.

Obviously, I could just download a newer Python and install it on my machine, but I'm trying to make sqlpython usable for DBAs who don't have that kind of authority, or who fear to muck around with their environment that way.

I wonder what the smart people do in situations like this?

SpeakerRate

I've created a page at speakerrate.com for myself:

SpeakerRate profile

... and populated it with my two upcoming talks.

If you want to use SpeakerRate to give feedback to a speaker, you don't actually have to wait for them to create their own profile - you can enter it yourself.

I don't know if SpeakerRate will grow into the ultimate connecting-to-speakers webtool, but it might. I do know that they support microformats, which is a big plus in my book.

Hope to see some of you at Penguicon (this weekend) or IOUG Collaborate (next week)!

right to complaint

(prompted by discussion of porn use at GoGaRuCo - see here, here)

Quick thought: It's not that the community needs to ensure offensive content never happens, or that the community needs to find a single standard of what is appropriate.

The key is the right to complain safely. When complaints are predictably met with accusations of "overreacting", "political correctness", and "intolerance", the resulting message is: Be like us, be silent, or leave.

If you reject the criticism, then try something like, "I think you're wrong, but I accept your right to complain." Complaint is feedback, it's a legitimate part of a community's communication.

(Let me clarify that I've had mostly really good experiences in the software communities I participate in!)

the expanding reStructuredTextiverse

It seems I'm always coming across new uses for reStructuredText, the plaintext format that goes everywhere. (Really, more of a "set of plaintext conventions" than a format as such.) I'm beginning to imagine a talk reviewing them all for next Fall's Ohio LinuxFest, or maybe a magazine article.

The places you can go with reStructuredText - am I missing any? (I haven't checked these all for viability)

• HTML, XML, pseudo-XML, LaTeX, and S5 (PowerPoint-like presentations), with the conversion utilities in docutils
• rst2a - reStructuredText 2 Anything - docutils conversion utilities, online
• rst2pdf
• rest2web - webpages with templates, navigation structure, etc.
• rst2man - Unix man pages
• Sphinx - easy, multi-page sites for docs or other static content
• epydoc - Automated API documentation
• nabu - ReStructured Text to RDBMS
• rst2odt - Produce files for OpenOffice Writer.
• wiki pages
• Blog clients
• django-rstify - ReStructuredText in Django templates
• Editor support

Not everything has been done yet, however. Here are a couple projects yet undone - so far as I know. Comment with your own ideas... or take this as a challenge and implement something!

• rst2word - this would really be the holy grail, for communicating to the unwashed masses. (We in the know can use rst2odt and convert within OpenOffice, but rst2word would get my boss on board.)
• Fuse more templating engines to rst (perhaps not a good idea, violate the readability principle?)
• ReST lexer for Scintilla - this would allow ReST support in WingIDE, too.

[EDIT: Thanks Michael Foord for info about rest2web, rst2pdf.]

chocolate

We interrupt this blog with a special message from our sponsor.

My friend James has a new business, Two Bears Chocolates, handmaking organic chocolates. They are crazy-good... I've never been a food snob, but James's chocolates have spoiled me to the point where even "gourmet" mass-produced chocolates seem waxy and bland by comparison. They're a diet aid! I have to eat these chocolates so that I won't be tempted by lesser chocolates!

Anyway, they're expensive, but worth it, and a good gift (see: Mother's Day)... or a good way to maximize your yummy-per-calorie ratio.

He's particularly gung-ho to do custom orders. Go ahead, ask him to make a gooseberry-chocolate truffle in the shape of the state of Michigan. He'll love you for the challenge.

So, as James says, "If you love someone, send Two Bears chocolates. If not, bummer."

calm down

Yes, yes. Oracle is buying Sun, which owns MySQL and Java. No, this is not the end of MySQL. You're being silly.

Oracle is about as open-source friendly as a huge proprietary software company can be, and has been since before it was cool. Oracle adores Linux, and started pushing it vigorously since about, hm, 2002? Oracle has been Java-crazy since that time, too. Oracle's marketing strategy has long been against lock-in - it wants to plug easily into a thriving open-standards economy, not to enclose and lock a walled garden. It's also been very easygoing about licensing, eager to see casual, non-paying users gaining familiarity with its products, knowing that those are the seeds that later big-money sales will come from. It doesn't try to catch and squeeze little fish, it feeds them fish food and waits for them to grow into whales. In short, you over there, slapping MySQL on your Linux box for your brother's home business? Oracle doesn't want to shut you down. Oracle loves you, has always loved you, and wants your love and trust for when you get big.

In fact, if anything, I'm a little disappointed that Oracle's (superb) marketing power, name recognition, and corporate respect will all benefit MySQL and Java... which is all fine and good, except that I'd rather see that gust of wind behind PostgreSQL and Python. (OTN's PyCon sponsorship warmed my heart, to be sure, but I wish there was a way to make ORACLE + PYTHON stop-the-presses news all around techland.)

#amazonfail

If you haven't heard about #amazonfail, this article will catch you up quickly.

1. It's a good reminder that giving market dominance to one company in a crucial role in steering our culture is probably unwise. Let's not have a market where a single company can relegate books to obscurity, intentionally or not. Patronize a variety of booksellers.

2. A proper apology would look something like, "Mr. Doofus Middle Manager didn't think through what he was doing, and company management failed to supervise it properly. We feel humiliated and commit to new efforts to keep book culture diverse and uncensored." Amazon's lame, mealy-mouthed "glitch" non-apology suggests that it has fallen prey to Big Corporatosis. Unless, of course, Amazon really has discovered a homophobic computer glitch, in which case this is huge news for artificial intelligence researchers.

3. This is not the only case where the label "adult content" has had a strangling effect. Actions taken under the justification of "adult content" should never be blandly accepted, but should be carefully examined for accuracy, necessity, and bias.

[ EDIT: A final complaint: Labelling a bad policy, sloppily implemented and poorly supervised, as a "glitch", is blaming the company's technologists for a mistake of its management. It says that Amazon's management does not trust, understand, or respect its technologists. In a technology company, this is a strong signal of decline.)

Penguicon 7.0

If you're anywhere near Michigan, you need to consider Penguicon. It's an open-source software conference! It's a science-fiction con! It's two great tastes that really do taste great together. There's always a great deal of excellent technical content, and the SF people lend a really healthy sense of relaxation and creativity to the whole thing. Where else can you learn CSS and belly dancing in one weekend?

I'm giving a talk at Penguicon this year: "sqlpython: SQL is fun again". It's sort of a preview of my upcoming SQL*Plus Alternatives talk at IOUG Collaborate... but without the stuff about Oracle-only tools, and with more focus on sqlpython's rapidly developing cross-RDBMS powers, and a healthy plug to pull more people into the project.

In fact, I'm leaving directly from Penguicon to Collaborate. Yikes! It'll be a fun week.

PyOhio: Call for Proposals

The PyOhio Call for Proposals has been issued!

PyOhio 2009 takes place July 25-26, 2009 at the Ohio State University in Columbus, Ohio. Much like a mini-PyCon, it includes scheduled talks, tutorials, Lightning Talks, Open Spaces, and room for your own unique ideas. If you can make it to Ohio this summer, please consider participating.



PyOhio 2009, the second annual Python programming mini-conference for Ohio and surrounding areas, will take place Saturday-Sunday, July 25-26, 2009 at the Ohio State University in Columbus, Ohio. A variety of activities are planned, including tutorials, scheduled talks, Lightning Talks, and Open Spaces.

PyOhio invites all interested people to submit proposals for scheduled talks and tutorials. PyOhio will accept abstracts on any topics of interest to Python programmers.

Standard presentations are expected to last 40 minutes with a 10 minute question-and-answer period. Other talk formats will also be considered, however; please indicate your preferred format in your proposal. Hands-on tutorial sessions are also welcomed. Tutorial instructors should indicate the expected length

PyOhio is especially interested in hosting a Beginners' Track for those new to Python or new to programming in general. If your proposal would be suitable for inclusion in the Beginners' Track, please indicate so. Organizers will work with speakers and instructors in the Beginners' Track to help them coordinate their talks/tutorials into a smooth, coherent learning curve for new Python users.

All proposals should include abstracts no longer than 500 words in length. Abstracts must include the title, summary of the presentation, the expertise level targeted, and a brief description of the area of Python programming it relates to.

All proposals should be emailed to cfp@pyohio.org for review. Please submit proposals by May 15, 2009. Accepted speakers will be notified by June 1.

You can read more about the conference at http://pyohio.org

If you have questions about proposals, please email cfp@pyohio.org. You can also contact the PyOhio organizers at pyohio-organizers@python.org.

geekspeakr.com

geekspeakr.com: connecting tech women speakers with event organizers

Many organisers of technical conferences, meetups, and dinners want to have more gender-balance in their lineups, but they don't know where to find technical women speakers.

Enter geekspeakr.com, a simple directory and connections system to help technical women speakers and event organisers to find each other.

I'm really glad to see this. I'm even more glad to see, browsing through speakers, that they really are technical women. I've... um, there's no good way to say this... seen other "tech women" groups that quickly became dominated by women networking for their multilevel marketing careers. It's pretty understandable, since they have a much more obvious need to network than us geeks - but, you know, it's really not the purpose.

Anyway. geekspeakr's are the real deal. w00t! Need more Python and Oracle speakers there, though.

Note to self: just as soon as the PyOhio CFP is out (very soon), do not neglect to spam Pythonistas on geekspeakr! At least, the ones vaguely near Ohio.

Five minutes at PyCon change everything

I gave a five-minute Lightning Talk on sqlpython on Saturday. I hoped it would pique the interest of some people who sometimes use Oracle, and give them a neat example of yet another cool thing being done with Python. It certainly did that, and I got lots of gratifying feedback.

I knew people would ask when it would be available for non-Oracle databases, so I said, tongue-in-cheek, that this was my distant-future ambition for "sqlpython 3000". What I didn't expect was that several of the people buttonholing me over the next two days would ask to collaborate to get multi-RDBMS support in place. Help? Uh, yeah... I guess help would help... I honestly hadn't even been thinking about that...

Brian Dorsey in particular wanted to see the code face-to-face with me, so I put a card on the Open Space board, just in case anybody else wanted to show up, and twittered about it one bare hour in advance.

Nine people came, all of them eager to get going on writing code, bringing great ideas to get started. All this for a project that was basically personal 36 hours before.

Noooo, now other people are going to be exposed to my squiggly code! Now I know what embarrassment-driven development really is.

If I'd had $1 million of startup funding to hire a staff to work on sqlpython, I couldn't have gotten a team that large or that talented. I figure that gives me better than a 1000-to-1 return on my PyCon investment. :)

So anyway, I'm setting up a mailing list for cooperation on sqlpython, and it looks like the far-future dream of multi-RDBMS sqlpython has suddenly become imminent. Stay tuned!

sqlpython lightning talk follow-up

This morning's lightning talk on sqlpython was an example of what's so great about PyCon - it instantly really good suggestions about how to go onward with sqlpython development, and offers of collaboration. Awesome!

Except that I forgot to show the instant graphs using \b / \l terminators. Rats! That's the most eye-catching part!

I wasn't prepared for the common question, though: "Where's the repository?" Well, it's here:
https://www.assembla.com/wiki/show/sqlpython It is crazy-unstable, and if you're actually trying to use it, use the PyPI version instead.

I mean to get a link to that into the docs as soon as possible, but I don't have Sphinx configured right on this machine, so that may have to wait.

Quick review of the lightning talk:

* Unix-like powers: cat, ls, grep, >, |
* Python interactive session; access to resultsets (`r`) and bind variables (`binds`)
* Special output formats with alternate terminators (see `help terminators`)
* The magic that makes sqlpython work:
- cmd
- cmd2
- pyparsing
- code (for the embedded Python interpreter)
- cx_Oracle

For further reference, see the sqlpython docs, particularly the comparative review of sqlpython vs. SQL*Plus vs. gqlplus vs. Senora vs. YASQL.

Ada Lovelace Day: Tech Women I Admire

I will publish a blog post on Tuesday 24th March about a woman in technology whom I admire but only if 1,000 other people will do the same.

I can't stop at one; sorry if that breaks the rules.

• As a frequent speaker at conventions and as co-author of the Python Cookbook, Anna Ravenscroft has a great talent for helping people understand problems, even the brain-bending ones.
• Dianne Marsh helps run a company that vigorously fosters good, innovative programming, both among their own developers and throughout the entire region. Her energy and dedication have been crucial to CodeMash, one of the most innovative things to happen in our area, and to several user groups in Michigan.
• Sarah Dutkiewicz, aka the Coding Geekette, is a programmer from northeast Ohio who's done great things in pulling together the geek community and teaching them new technologies (like IronPython on Mono) - often the sort of stuff that requires venturing into dragon-infested realms where the documentation is scattered or nonexistent.
• In her long-time leadership of the OOUG, Coreen Walker
has helped make it one of the best Oracle groups around.

PyCon begins...

... well, the tutorials and summits begin tomorrow. But some of the organizers are already in place, getting things set up. I wish I were there! I won't arrive until Thursday afternoon - I had to sacrifice all but the core conference days so that I could make the separate trip to IOUG Collaborate. For me, it feels like arriving at a family Christmas halfway through the gift unwrapping.

Have you seen the list of PyCon sponsors this year? It's amazing! It's practically a Who's Who... if your company isn't sponsoring PyCon, your corporate headquarters is probably roofed with thatch. :) It's a tribute to Van Lindberg, PyCon's sponsorship coordinator, but even more to the growing corporate recognition that Python has become a pillar of the IT world.

Where to host docs?

I'm working on some preliminary sqlpython docs using Sphinx, which is really really nice. No guarantees that they will remain there, though - they need a home without a raw IP address for a URL!

I'm trying to figure out a permanent home for the docs. Here are the possibilities I know of.

• Host the files on my own server. Buy a domain name (but not like last time). This is obviously the best option... unless I ever decide to quit paying for the domain, or the server, or forget to, or get hit by a bus. So really, I'd prefer something that wasn't so dependent on, well, me.
• Sourceforge, Assembla, Google Code, and probably everybody else who hosts projects also let you create and host wiki-style documentation. As far as I know, though, there's no way to upload Sphinx documentation into any of them; the wiki form and the Sphinx form are not compatible.
• Google Pages gives you free webpages with a reasonable URL - but no folders. Sphinx depends on a folder structure - it creates folders "html", "doctrees", "_sources", "_static", and maybe more once I get serious.
• Creating a Google App Engine to host the docs - now this is an intriguing possibility. I'm going to check it out. Still, it would be nice if posting the docs for a technical project were not, itself, a technical project.

Any possibilities I'm missing? A convenient way to host Sphinx docs, freely available, would be a really nice service to FOSS developers. I wonder if the Google App Engine approach could be extended to provide that... hmm hmm hmm... just what I needed, one more project...

sqlpython 1.6.1 puts the "python" in "sqlpython"

I've been asked several times, "Why is it called sqlpython?"

My answer used to be, "Ask Luca [Canali]; he wrote it."

Not anymore. Now, witness the power of this fully armed and operational hybrid SQL/Python working environment.

[EDIT: As of sqlpython 1.6.2, you use quit(), exit(), or Ctrl-D/Ctrl-Z to return from
interactive Python mode. See docs.]

0:testschema@eqtest> select title, author from play;

TITLE           AUTHOR
--------------- -----------
Timon of Athens Shakespeare
Twelfth Night   Shakespeare
The Tempest     Shakespeare
Agamemnon       Aeschylus

4 rows selected.

0:testschema@eqtest> py import urllib
0:testschema@eqtest> py current_season = urllib.urlopen('http://cincyshakes.com/').read()
0:testschema@eqtest> py
Now accepting python commands; end with `end py`
>>> r[-1]
[('Timon of Athens', 'Shakespeare'), ('Twelfth Night', 'Shakespeare'), ('The Tempest', 'Shakespeare'), ('Agamemnon', 'Aeschylus')]
>>> for row in r[-1]:
...     print '%s by %s' % (row.title, row.author)
Timon of Athens by Shakespeare
Twelfth Night by Shakespeare
The Tempest by Shakespeare
Agamemnon by Aeschylus
>>> [row.title for row in r[-1] if row.title in current_season]
['Timon of Athens', 'Twelfth Night']
>>> binds['nowplaying'] = [row.title for row in r[-1] if row.title in current_season][0]
>>> end py
0:testschema@eqtest> print
:nowplaying = Timon of Athens
0:testschema@eqtest> select title, author from play where title = :nowplaying;

TITLE           AUTHOR
--------------- -----------
Timon of Athens Shakespeare

1 row selected.

A history of result sets from each query is exposed to the python session as the list `r`; the most recent result set is `r[-1]`. Bind variables are exposed as the dictionary `binds`. All variables are retained each time the python environment is entered (whether interactively, or with one-line `py` statements).

Resultsets in `r` are read-only, but `binds` can be written as well as read, and will be working bind variables in the SQL environment.

Oh, the possibilities...

Senora 1.0.1, and mutual FOSS goading

Martin Drautzburg just published version 1.0.1 of Senora, a command-line client for Oracle that makes a very attractive alternative to Oracle's SQL*Plus.

The timing - seven weeks before I present a review of Senora, sqlpython, and others at IOUG Collaborate - is not a coincidence. I asked Martin to review my draft paper submission for the conference. He graciously did, and supplied several crucial corrections and additions regarding Senora.

He also mentioned that he'd been continuing Senora development for in-house use, but that it had been years since he'd released the updates. My upcoming review spurred him to do the release.

And what a release it is - it's full of awesome! Mighty new flag options! Multiple sessions! Automatic generation of Senora commands from SQL scripts!

It sent me to scheming about how I can copy this fresh goodness to sqlpython.

Plus, of course, the paper reminded me painfully of some of sqlpython's shortcomings. I'm starting to use the sqlpython trac seriously now, as a token to myself of my goodwill. This will probably lead to a spurt of Embarrassment-Driven Development before the conference.

FOSS people know what the ancient Romans knew: money is a feeble motivator compared to glory. (Half the time, people only want money so they can buy things that will make them feel glorious...)

using TurboGears 2 model outside TurboGears

Until/unless this ticket gets incorporated into the TurboGears 2 docs, I (for one) need a reminder about how I can make use of a sqlalchemy model, set up from a TurboGears 2 instance, for projects that don't actually start or use TurboGears. (I want a single, canonical sqlalchemy model for my database, for its web-based and non-web-based applications alike.)

Including this function in my model/__init__.py does the trick.

import paste.deploy, os.path
def externally_usable_session(configfile = 'development.ini'):
    tg_home_directory = '/path/to/tg2instancehomedirectory'
    conf_dict = paste.deploy.appconfig('config:%s' % os.path.join(tg_home_directory, configfile))
    engine = create_engine(conf_dict['sqlalchemy.url'])
    init_model(engine)
    return DBSession()

sqlpython 1.6.0 with Wild SQL

I just released sqlpython 1.6.0.

SELECTing a limited, but large, set of columns from a table is a real pain. What if you could use wildcards in the column list of the SELECT statement itself? Wouldn't that be wild?

OK, then, let's SET WILD ON.

jrrt@orcl> cat party

NAME    STR INT WIS DEX CON CHA
------- --- --- --- --- --- ---
Frodo     8  14  16  15  14  16
Gimli    17  12  10  11  17  11
Legolas  13  15  14  18  15  17
Sam      11   9  14  11  16  13

4 rows selected.

jrrt@orcl> set wild on
wildsql - was: False
now: True
jrrt@orcl> select *i* from party;

INT WIS
--- ---
 14  16
 12  10
 15  14
  9  14

4 rows selected.

You can also call columns out by number...

jrrt@orcl> select #1, #5 from party;

NAME    DEX
------- ---
Frodo    15
Gimli    11
Legolas  18
Sam      11

4 rows selected.

... or use ! as NOT.

jrrt@orcl> select !str from party;

NAME    INT WIS DEX CON CHA
------- --- --- --- --- ---
Frodo    14  16  15  14  16
Gimli    12  10  11  17  11
Legolas  15  14  18  15  17
Sam       9  14  11  16  13

4 rows selected.

... and you can mix it all together.

jrrt@orcl> select n*, !#3, !c* from party;

NAME    STR WIS DEX
------- --- --- ---
Frodo     8  16  15
Gimli    17  10  11
Legolas  13  14  18
Sam      11  14  11

4 rows selected.

A bunch of limitations:

• Wild SQL is not yet a widely-accepted industry standard. Actually, I just made it up. If ANSI hears about it, they will hunt me down with dogs. That's why you need to SET WILDSQL ON to turn it on.
• Wild SQL only works on the column list - the part between the SELECT and the FROM. It doesn't work in the WHERE clause, or in subqueries.
• Wild SQL only works in SELECT statements. What, you were thinking about using it in DML? Are you crazy?
• Do I really have to say that it's very alpha? Well, it is. Expect a trickle of bugfixes over the next few months.

sqlpython 1.5.3, with version control

There are several barriers to use of version control tools for Oracle DDL (Data Definition Language: table structures, etc.)

• Tools from Oracle, Quest, etc. may have VC capabilities, but you never know whether you'll have the tool available in any particular situation, and you never know when the workings of the tools will change beyond recognition. (I'm looking at you, Oracle.)
• The tools are separate from the mainstream of version control in software development - you're learning quirky specialty tools instead of widely-known industry standards.
• They lack some of the modern capabilities of distributed version control.

We're better off using standard software development VC.

But, of course, those tools are meant for text files, so there needs to be a handy way to get this stuff from DDL inside the database to version-controlled text files.

It's easy with sqlpython 1.5.3

The new sqlpython commands svn, bzr, and hg all do the following:

1. Create or update a directory tree, beginning at your current working directory, containing text files with the DDL for all the objects in your schema
2. Put these files under version control and commit

If you don't want all the DDL, you can limit the dump using the same arguments the ls command takes.

For example,

testschema@orcl> !pwd
/home/catherine/oracle_vc
testschema@orcl> ls

NAME
--------------
INDEX/XPK_PLAY
TABLE/PLAY

2 rows selected.

testschema@orcl> bzr
added testschema
added testschema/index
added testschema/index/xpk_play.sql
added testschema/table
added testschema/table/play.sql
Committing to: /home/catherine/oracle_vc/
added testschema
added testschema/index
added testschema/table
added testschema/index/xpk_play.sql
added testschema/table/play.sql
Committed revision 1.
testschema@orcl> alter table play add (opening_night DATE);

Executed

testschema@orcl> bzr
bzr: ERROR: Already a branch: ".".
Committing to: /home/catherine/oracle_vc/
modified testschema/table/play.sql
Committed revision 2.

testschema@orcl> alter table play add (performances NUMBER(5,0));

Executed

testschema@orcl> create index xif1_play on play (opening_night);

Executed

testschema@orcl> bzr index/
bzr: ERROR: Already a branch: ".".
added testschema/index/xif1_play.sql
Committing to: /home/catherine/oracle_vc/
added testschema/index/xif1_play.sql
Committed revision 3.


"Writing about Python" at PyCon

PyCon early-bird registration deadline is TOMORROW (Saturday)! No time to lose - go register!

Most Open Spaces are not scheduled until the very day they are held, and that's good. Some are done with some advance planning, though, and that's good too. Doug Hellman is already preparing a "Writing about Python" open-space session at PyCon; I'm eager to take part.

If you're not familiar with conferences like PyCon, you may not realize that the formal schedule, goodie-packed though it is, is not the whole story by a long shot. People use the Open Spaces for a huge variety of things; last year, for instance, I got a lot out of a group organizers' freeform open space discussion, and even more out of the now-famous "Teach Me Twisted" session. If you think the published schedule leaves you in fits of indecision, wait until you see the Open Space board. There's a sort of joyful despair in seeing that you would need three months of PyCon to take part in all the PyCon you want.

Open Source wrecked the economy

Well, not really, but it did get your attention.

This is a really interesting New York Times article about VaR ("Value at Risk"), a mathematical tool. Over-reliance on this tool was arguably the reason the world's financial experts and geniuses spent the last several years acting like morons. This paragraph jumped out at me.

What caused VaR to catapult above the risk systems being developed by JPMorgan competitors was what the firm did next: it gave VaR away. In 1993, Guldimann made risk the theme of the firm’s annual client conference. Many of the clients were so impressed with the JPMorgan approach that they asked if they could purchase the underlying system. JPMorgan decided it didn’t want to get into that business, but proceeded instead to form a small group, RiskMetrics, that would teach the concept to anyone who wanted to learn it, while also posting it on the Internet so that other risk experts could make suggestions to improve it. As Guldimann wrote years later, “Many wondered what the bank was trying to accomplish by giving away ‘proprietary’ methodologies and lots of data, but not selling any products or services.” He continued, “It popularized a methodology and made it a market standard, and it enhanced the image of JPMorgan.”

I thought this was a fascinating summary of open-source advantages: prestige, benefiting from community-contributed enhancements, creating a standard, all without the effort and expense of attempting to market a proprietary technology.

OK, so it's not really a feather in Open Source's cap, seeing as the economy did end up wrecking over it 'n all, but it does demonstrate the power of openness to popularize a tool. Open-sourcing a tool can make it very popular, but it's up to us not to make a tool into a god.

(Of course, when it's applied outside the software world, we really ought to point out that the principle never came from software at all. It's simply openness, the same ancient innovation that happened in the move from alchemists guarding their secrets to scientists publishing their work.)

Recent developments in the doom of humankind

xkcd foresees it

skynet 0.1

Multi-ton, clawed semi-autonomous Python-powered robots

PyCon talks

You want an awesome badge like this for your blog, right? You can choose from a variety of badges at the publicizing PyCon site. PyCon depends on the user community (that's you) to spread the word. Bringing in a bigger and broader community is how PyCon keeps getting more exciting.

Anyway, even if you are certain you can't attend PyCon, it's worth it to browse the list of accepted PyCon talks - it's a great window into some of the exciting things going on in Python-land. It's already helped me learn about several useful packages and ideas.

Hope to see you in Chicago!

blogging about business, for once

Today my employer benefited through me, but almost despite me.

We're facing an exhausting slog through yet another incarnation of the Air Force's process for getting permission to continue to operate an IT system. My (Air Force) boss had heard somehow that Mark, one of my fellow employees at Intellitech, was shepherding some projects through this process, and asked if he'd be willing to give some advice. Mark came and spent a couple hours giving some desperately needed information, despite being warned that my boss has no prospects of funding to take on additional contracting help.

Everyone there was enormously grateful, because living, breathing survivors of the process are almost unheard of, and the available training material is of very little use. What's more, my boss had made a last-minute impromptu invitation to a friend whose project is also facing the process; she was just as happy to attend, and just might have the funding for some help. And everybody there is going to spread the word that they now know a source of much-needed information on this process.

The problem? I was passive. I hadn't thought to suggest to my boss to tap Mark's experience; good thing he knew about it and thought to ask. I hadn't thought to suggest inviting others to the meeting, either; again, credit to my boss. I really need to be more alert to this sort of thing.

The other problem? If Intellitech does end up with more work thanks to today, it's the kind of hellishly bureaucratic work that makes you want to chew your leg off to escape. Great for the company bottom line, but a blasphemous waste of a living human soul.