Wednesday, September 28, 2005

introducing sqlWrap.py

A Python script, for your use and suggestions.
Sorry, I know this is awfully long for a blog post - I have to figure out a better place to put it. I wonder if it would be appropriate for the Cheese Shop?

"""
sqlWrap
Adds convenience methods to DB-API2 connection objects.

Sep. 22, 2005 by Catherine Devlin (catherine.devlin@gmail.com, http://catherinedevlin.blogspot.com/)

This script is NOT intended as an object/relational mapper.
Rather, it helps experienced SQL users form their SQL more quickly.

Methods: insert, update, delete, select, genericSelect
These methods accept arguments for the WHERE, SET, etc. clauses
that should generally be provided as dictionaries; i.e.
whereClause = {'col1':'val1','col2':'val2%'} implies
"WHERE col1 = 'val1' AND col2 LIKE 'val2%'",
setClause = {'col1':'val1'} implies
"SET col1 = 'val1'"
You may also pass object instances for whereClause and setClause,
with instance attributes corresponding to column names... but this
has barely been tested!
They also automatically make use of bind variables, which have
performance and security benefits over hard-coding values in SQL.
The different ways of handling bind variables in various DB-API
adapters are masked from the user.

Currently supports: Oracle (cx_Oracle), sqlite (pysqlite)

Sample usage:
# setup - unchanged from cx_Oracle
conn = OraConnection('scott/tiger@orcl')
conn.cursor().execute('CREATE TABLE myTable (column1 varchar2(10), column2 varchar2(10), column3 varchar2(10))')
# now try out the sqlWrap convenience methods
conn.insert('myTable', setClause={'column1':'value1','column2':'value2'})
conn.insert('myTable', setClause={'column1':'value1a','column2':'value2a','column3':'value3a'})
for row in conn.select(source='myTable'):
print row
for row in conn.select(source='myTable', whereClause={'column1':'value1'}, resultProcessor=conn.dictionaryize):
print row
conn.update('myTable', setClause={'column1':'value1','column2':'value2'}, whereClause={'column3':'value3'})
# as always, must explicitly commit
conn.commit()
"""

Get sqlWrap.py here

Tuesday, September 27, 2005

Monday, September 26, 2005

notes to self

Do not smash your head against a problem for consecutive hours. Instead:
  • Ask for help. It doesn't matter whether you ever send the request; in the process of putting the problem into language, you're more than likely to solve it. (Yes, I know: "Pair programming!", you say. But I'm all alone here.)
  • Stand up and walk away. Do not touch anything resembling a computer for at least ten minutes.
Will someone please remind me of these next time I get stuck?

Thursday, September 22, 2005

Python for Oracle Geeks: unplugged!

I've been invited to present at the Oct. 27 Ohio Oracle User Group meeting in Columbus. Hooray! After I've practiced at a local group, I'll feel ready to start trying to get to wider areas, maybe even the national IOUG conference. I will bring tanker-trucks full of Python Kool-Aid!

We have a lot of catching up to do - Oracle's OTN has been publishing a LOT on PHP, and Oracle and Zend are providing a handy-looking integrated package. I salute the PHP folks, but I don't want that to be the only dynamic language active in Oracle-land.

sqlite: a flyswatter to kill flies

I'm now using sqlite to support an Oracle production database. I love it!

When I first went to download sqlite, I went away frustrated. I found an executable described as "A command-line program for accessing and modifying SQLite databases", and thought, "OK, so that's my SQL*Plus equivalent, but where's the actual server? The part that keeps the database process running?"

Because I have been an Oracle-only person so long, I didn't understand that sqlite doesn't need anything like that. It's simply this:
  1. a single executable program that creates and modifies a database file
  2. A database file
And that's it. Really! You can't imagine how dizzying and refreshing that is to someone who's spent hours before a certification exam memorizing Oracle architecture diagrams.

Anyway, there are other simple database engines, of course, but sqlite has gotten a lot of attention recently (like an Open Source Award) for its efficiency and its support across many languages. There's an excellent and honest rundown of its powers and limitations here.

So, anyway - why sqlite to support Oracle? Well, one of my Oracle instances has a bunch of logic applied to it by a nightly batch job. For every data record, a series of decisions are made, and our users ask questions like, "For record #12945, why did it decide X instead of Y last Tuesday?" And I have to be able to answer, "Well, the seventh of the nine tests conducted on that record determined that, since column 'product' was 'lutefisk' and 'quantity_kg' was 22, blah blah blah..." So all those decisions need to be logged every night.

That generates a quantity of data far outweighing the application data itself. It can be discarded after a week or two, but all that inserting and deleting was causing out-of-control generation of archive log files and making a mess of my disk space allocation. By moving that data out into a sqlite database, it becomes a single simple file that can be moved or deleted as easily as any other file.

So, thanks to sqlite, I'm living happily ever after. Hooray!

Tuesday, September 13, 2005

more on XML

The problem with my XML-generating views is that they break down for rows whose XML is longer than 4000 characters.

Meanwhile, the troubles I cited with XMLELEMENT really are limited to SQL*Plus. I was deterred at first, because I want to experiment with unfamiliar features through SQL*Plus first, but when I swallowed that reluctance and remembered that there are other ways to experiment with ad-hoc SQL - TOAD's SQL window, for instance - I was OK.

But then, I wanted to create views that would store the particular combinations of XMLELEMENTs I wanted for various circumstances. Unfortunately, I found that a SELECT query that runs fine can't be used to generate a working view when the result is longer than - you guessed it - 4000 characters.

I'm having a little trouble figuring out where exactly the 4000 character problem kicks in - at first I thought CLOBS were simply converted back to VARCHAR2's whenever I used the concatenation operator |, but that doesn't explain why views fail when the equivalent bare SELECT statements don't.

Anyway, the end of my adventure was that I simply wrote Python scripts to get what I want. I'm beginning to get some nice conveniences built into my personal wrapper for cx_Oracle, and maybe I'll eventually float it around for others' use. There are plenty of tools like SQLObject out there, of course, but as far as I know, they're all focused on OO programmers who hate to handle SQL directly or to think of their data as "rows in a table" rather than "instances of an object". I, on the other hand, think very naturally in SQL statements and relational concepts, and just use some functions to streamline my use of SQL.

Finally, there's one cute little possibility for producing XML that I haven't explored. SQL*Plus can generate an HTML table as output really easily. That could be coupled with a XSLT to produce XML.