Wednesday, September 28, 2005

introducing sqlWrap.py

A Python script, for your use and suggestions.
Sorry, I know this is awfully long for a blog post - I have to figure out a better place to put it. I wonder if it would be appropriate for the Cheese Shop?

"""
sqlWrap
Adds convenience methods to DB-API2 connection objects.

Sep. 22, 2005 by Catherine Devlin (catherine.devlin@gmail.com, http://catherinedevlin.blogspot.com/)

This script is NOT intended as an object/relational mapper.
Rather, it helps experienced SQL users form their SQL more quickly.

Methods: insert, update, delete, select, genericSelect
These methods accept arguments for the WHERE, SET, etc. clauses
that should generally be provided as dictionaries; i.e.
whereClause = {'col1':'val1','col2':'val2%'} implies
"WHERE col1 = 'val1' AND col2 LIKE 'val2%'",
setClause = {'col1':'val1'} implies
"SET col1 = 'val1'"
You may also pass object instances for whereClause and setClause,
with instance attributes corresponding to column names... but this
has barely been tested!
They also automatically make use of bind variables, which have
performance and security benefits over hard-coding values in SQL.
The different ways of handling bind variables in various DB-API
adapters are masked from the user.

Currently supports: Oracle (cx_Oracle), sqlite (pysqlite)

Sample usage:
# setup - unchanged from cx_Oracle
conn = OraConnection('scott/tiger@orcl')
conn.cursor().execute('CREATE TABLE myTable (column1 varchar2(10), column2 varchar2(10), column3 varchar2(10))')
# now try out the sqlWrap convenience methods
conn.insert('myTable', setClause={'column1':'value1','column2':'value2'})
conn.insert('myTable', setClause={'column1':'value1a','column2':'value2a','column3':'value3a'})
for row in conn.select(source='myTable'):
print row
for row in conn.select(source='myTable', whereClause={'column1':'value1'}, resultProcessor=conn.dictionaryize):
print row
conn.update('myTable', setClause={'column1':'value1','column2':'value2'}, whereClause={'column3':'value3'})
# as always, must explicitly commit
conn.commit()
"""

Get sqlWrap.py here

Tuesday, September 27, 2005

Monday, September 26, 2005

notes to self

Do not smash your head against a problem for consecutive hours. Instead:
  • Ask for help. It doesn't matter whether you ever send the request; in the process of putting the problem into language, you're more than likely to solve it. (Yes, I know: "Pair programming!", you say. But I'm all alone here.)
  • Stand up and walk away. Do not touch anything resembling a computer for at least ten minutes.
Will someone please remind me of these next time I get stuck?

Thursday, September 22, 2005

Python for Oracle Geeks: unplugged!

I've been invited to present at the Oct. 27 Ohio Oracle User Group meeting in Columbus. Hooray! After I've practiced at a local group, I'll feel ready to start trying to get to wider areas, maybe even the national IOUG conference. I will bring tanker-trucks full of Python Kool-Aid!

We have a lot of catching up to do - Oracle's OTN has been publishing a LOT on PHP, and Oracle and Zend are providing a handy-looking integrated package. I salute the PHP folks, but I don't want that to be the only dynamic language active in Oracle-land.

sqlite: a flyswatter to kill flies

I'm now using sqlite to support an Oracle production database. I love it!

When I first went to download sqlite, I went away frustrated. I found an executable described as "A command-line program for accessing and modifying SQLite databases", and thought, "OK, so that's my SQL*Plus equivalent, but where's the actual server? The part that keeps the database process running?"

Because I have been an Oracle-only person so long, I didn't understand that sqlite doesn't need anything like that. It's simply this:
  1. a single executable program that creates and modifies a database file
  2. A database file
And that's it. Really! You can't imagine how dizzying and refreshing that is to someone who's spent hours before a certification exam memorizing Oracle architecture diagrams.

Anyway, there are other simple database engines, of course, but sqlite has gotten a lot of attention recently (like an Open Source Award) for its efficiency and its support across many languages. There's an excellent and honest rundown of its powers and limitations here.

So, anyway - why sqlite to support Oracle? Well, one of my Oracle instances has a bunch of logic applied to it by a nightly batch job. For every data record, a series of decisions are made, and our users ask questions like, "For record #12945, why did it decide X instead of Y last Tuesday?" And I have to be able to answer, "Well, the seventh of the nine tests conducted on that record determined that, since column 'product' was 'lutefisk' and 'quantity_kg' was 22, blah blah blah..." So all those decisions need to be logged every night.

That generates a quantity of data far outweighing the application data itself. It can be discarded after a week or two, but all that inserting and deleting was causing out-of-control generation of archive log files and making a mess of my disk space allocation. By moving that data out into a sqlite database, it becomes a single simple file that can be moved or deleted as easily as any other file.

So, thanks to sqlite, I'm living happily ever after. Hooray!

Tuesday, September 13, 2005

more on XML

The problem with my XML-generating views is that they break down for rows whose XML is longer than 4000 characters.

Meanwhile, the troubles I cited with XMLELEMENT really are limited to SQL*Plus. I was deterred at first, because I want to experiment with unfamiliar features through SQL*Plus first, but when I swallowed that reluctance and remembered that there are other ways to experiment with ad-hoc SQL - TOAD's SQL window, for instance - I was OK.

But then, I wanted to create views that would store the particular combinations of XMLELEMENTs I wanted for various circumstances. Unfortunately, I found that a SELECT query that runs fine can't be used to generate a working view when the result is longer than - you guessed it - 4000 characters.

I'm having a little trouble figuring out where exactly the 4000 character problem kicks in - at first I thought CLOBS were simply converted back to VARCHAR2's whenever I used the concatenation operator |, but that doesn't explain why views fail when the equivalent bare SELECT statements don't.

Anyway, the end of my adventure was that I simply wrote Python scripts to get what I want. I'm beginning to get some nice conveniences built into my personal wrapper for cx_Oracle, and maybe I'll eventually float it around for others' use. There are plenty of tools like SQLObject out there, of course, but as far as I know, they're all focused on OO programmers who hate to handle SQL directly or to think of their data as "rows in a table" rather than "instances of an object". I, on the other hand, think very naturally in SQL statements and relational concepts, and just use some functions to streamline my use of SQL.

Finally, there's one cute little possibility for producing XML that I haven't explored. SQL*Plus can generate an HTML table as output really easily. That could be coupled with a XSLT to produce XML.

Wednesday, August 17, 2005

querying XML without fighting with XDB

My coworker and I want to query some nice, simple XML from my Oracle table. For example, from the table
CREATE TABLE pet (name VARCHAR2(22), species VARCHAR2(22), weight_kg NUMBER);
I want to query up results like
  <pet>
<name>Martin Luther</name>
<species>cat</species>
<weight_kg>6.2</weight_kg>
</pet>
<pet>
<name>Jordache</name>
<species>horse</species>
<weight_kg>450</weight_kg>
</pet>
My first thought was to use an XDB's XMLType View of my table. Okay... that looks... sort of intimidating. But I want to be able to query from it with the same old SQL I'm used to; I want to grab the XML for all rows
WHERE species = 'cat' OR (weight_kg > 100 AND name LIKE 'J%')
Now it looks really intimidating. I don't want to go back to school for a Master's of Science in XPath right now, and neither does my coworker.

Then I thought, "I know! I'll just use XMLForest, like this!"
SELECT XMLForest(name, species, weight_kg) FROM pet

On my platform (Windows 2003 / SQL*Plus 10.1.0.2.0 / Oracle 10.1.0.4.0 ), this produces
"Oracle SQL*Plus has encountered a problem and needs to close."
Attempted workarounds - like
EXEC SELECT XMLForest(name, species, weight_kg) INTO :petxml FROM pet
give me results like
"PLS-00801: Internal error [*** ASSERT at file pdw4.c, line 782; Cannot coerce between type 43 and type 30; _anon__2C9F5D70__AB[1, 7]]"
Okay, I give up - this XDB stuff is not mature enough for me today.

My handmade solution is fairly simple to use.
SELECT xml FROM pet_xmlvw WHERE species = 'cat' OR (weight_kg > 100 AND name LIKE 'J%')

Setting up views like pet_xmlvw is a headache and a half, but my ugly code will do it for you. (Can you hear PL/SQL crying? Can you hear it crying, "Somebody write Cheetah for me!"?)
CREATE OR REPLACE FUNCTION sf_columns_xml(table_name IN VARCHAR2)
RETURN CLOB
IS
result CLOB := '';
column_name VARCHAR2(30);
BEGIN
FOR c IN ( SELECT utc.column_name
FROM user_tab_columns utc
WHERE utc.table_name = UPPER(sf_columns_xml.table_name) )
LOOP
column_name := lower(c.column_name);
if length(result) > 0
then
result := result || chr(10);
end if;
result := result ||
' <' ||
column_name || '> '' || ' || column_name || ' || ''</' ||
column_name || '>';
END LOOP;
RETURN result;
END;
/

CREATE OR REPLACE FUNCTION sf_table_xml(table_name IN VARCHAR2)
RETURN CLOB
IS
BEGIN
RETURN
'CREATE OR REPLACE VIEW ' || lower(substr(table_name,1,24)) || '_xmlvw AS
SELECT ''
<' || lower(table_name) || '>
' || to_char(sf_columns_xml(table_name)) ||
'
</' || lower(table_name) || '>'' xml,
' || table_name || '.*
FROM ' || table_name;
END;
/
Now running EXEC execute immediate TO_CHAR(sf_table_xml('PET')) will generate the view pet_xmlvw, whose column xml contains what I want.

To generate such views for all the tables in my schema, I run
spool buildXMLViews.sql
select 'exec execute immediate to_char(sf_table_xml(''' || table_name || '''));'
from user_tables
where dropped = 'NO';
select 'exec execute immediate to_char(sf_table_xml(''' || view_name || '''));'
from user_views;
spool off
@buildXMLViews
(Hey, I didn't use q'| |')! Well, you may not all be on Oracle 10g.)

Sometimes, it's a sausage factory in here. Don't look at how it gets done, just pass the ketchup.

Wednesday, August 10, 2005

Geek event aggregator

Click me!

I've written up an aggregator script (Python, of course) that browses an assortment of event announcement webpages and parses out the name/city/state/date information. Then I made an HTML DB application to serve it up according to your state or province (sorry, non-North-Americans).

No, I haven't written a brilliant AI to figure this out. It's just a bunch of regular expressions.

To do:
  1. Mine from more event sources
  2. Refactor the source code so it doesn't embarrass me and I can post it
  3. Debug multiple hits for some events (InOUG especially)
  4. Add non-North-American region support
  5. Automate the mining (currently I kick it off and upload to HTML DB by hand)
  6. get somebody to host the app in some more prominent site
  7. Provide access to the pure XML as generated by the aggregator, so others don't have to go through the heck of parsing I do

Monday, August 08, 2005

Python for Oracle Geeks

IOUG posted my article, Python for Oracle Geeks, and referenced it in their 7/20/2005 "5 MINUTE BRIEFING: Oracle" e-mail. Ah, the fame, the adoring crowds, the paparazzi...

Friday, July 29, 2005

Tom Kyte

Tom Kyte was at OOUG in Columbus. He's even better in person than the is at AskTom.

Here's one little gem he gave that will change your life, if you're an oracle person. Or at least hurry you to upgrade to 10g, where it can be used.

The magic is q'| |'.

INSERT INTO my_tbl(str) VALUES
(q'|I don't have to worry when I write a string like
'INSERT 'r2d2' INTO droids'|');
INSERT INTO my_tbl(str) VALUES
(q'|It's almost as good as Perl!|');

SELECT * FROM my_tbl;

STR
-----------------------------------------------------
I don't have to worry when I write a string like
'INSERT 'r2d2' INTO droids'

It's almost as good as Perl!

All those single-quote single-quote, single-quote single-quote single-quote monstrosites we used to make to get quote marks into SQL strings are history. Let the people rejoice!

Tuesday, July 12, 2005

LibraryLookup

For Mozilla, Firefox, or Netscape, drag-and-drop any of these links to your toolbar to create a LibraryLookup bookmarklet. For Internet Explorer, use Add to Favorites... and create it in Links.

Public librariesCollege and university librariesNow, when you're browsing any site that describes a book by its ISBN (amazon.com, for example), clicking your new toolbar button or link will automagically look that book up in your library's catalog. Yay!

Hooray for Jon Udell, creator of LibraryLookup! (And a little bit of hooray for me, who slaved over a hot keyboard to make these public library bookmarklets.)

TODO: These bookmarks are written to extract the ISBNs from the URL, not the webpage text. Some sites, like O'Reilly, don't use ISBNs in their books' URLs - but still list them in the webpage text. A more sophisticated bookmarklet might be able to find the ISBN there, too. I'd like to write bookmarklets that could find and use those ISBNs, too.

Friday, July 01, 2005

the easy tools

  • If only I had known about tooltips months ago!
    A tooltip is what you get when you hover over a word like these ones. I went off looking for how to generate them, expecting to get myself hip-deep in JavaScript, and found that this is all you have to do:
    <span title="I love tooltips.">these ones<span>
  • If I had known about CherryPy months ago!
    With all due respect to other Python-based web application platforms like Zope, CherryPy is so much easier to get into, and soooooo satisfying. And it really does run much faster than Zope, too. Yum!

Thursday, June 23, 2005

Flaming about Curl and open source

Once upon a time, I read some interesting buzz about the Curl programming language. It sounded different, and it came from MIT (yeah, I know, alum chauvenism), so I had an inclination to check it out.

I'm a hopeless addict of the MicroCenter bargain rack, so when I spotted the Curl Programming Bible there for $5, I couldn't resist.

Wow, am I disappointed. It's no wonder Curl is apparently dying; it was doomed from the start.

One problem is the syntax. I didn't get very deep into it - just far enough to read
Notice that the entire herald is enclosed in curly braces ({ and }). All programming expressions in the Curl language are enclosed in these braces. They are used so often that the language itself was named after them!
Well, tee-hee-hee! Isn't that just precious? Maybe they should have named the language "Repetitive Motion Injury"! Okay, so I'm a Python geek; requiring gratuitous punctuation is an excellent way to send me reeling away in disgust..

But the real problem is the business model. The Curl plug-in is built so that, every time a user hits a Curl site, it notifies the Curl corporation to charge that site's owner. I don't care how small the charge per hit is, that's crazy. I'm going to get approval, in advance, to pay an unknown, usage-based fee, just to put up a site? For a tool that is (within our organization) totally unproven?

I think that open source really is the way to introduce a language these days. Very few languages are selected in advance by people with money-spending power. (Unless, of course, they are pushed by a megacorporation with the advertising budget to send flocks of salespeople to razzle managers and take them to lunch. But that's another discussion.) Rather, a developer or sysadmin or whoever finds a language she likes, starts using it informally for her own this and that, eventually builds something that her manager sees and likes, THEN gets permission to start working it into formal projects - the kind that have budgets. Then we'll start paying money for an IDE and training and maybe consulting and so on. At least, that's the way I've experienced it.

The sad thing is that the fate of Curl reminds me very much of the Water language - another MIT baby that interested me enough to sell me a book, but that is also more-or-less stalled. Again, I think the choice not to go open-source is a big part of that failure. (It probably doesn't help that Water is built on top of, and demands a little familiarity with, Java - so the ideal target audience is one that already has a preference for open-source.) Geeks just don't seize something and energetically play with it when they're told (1) This is ours, not yours and (2) Pay at the door.

I wonder if some foolish person in MIT's Office of Making Money Off Our Babies simply doesn't realize this, and is twisting arms to impose doomed business practices on MIT's babies. Ironic, for the institution that spawned the Free Software Foundation to go that way.

If anybody wants a barely-used Curl book, just ask.

Tuesday, June 07, 2005

HTML DB

A couple months ago, I built a D&D-suitable "random goods" generating website with HTML DB as an experiment. Parts of it are very frustrating, but overall it's an awfully powerful product. My example non-masterpiece is here.

Tuesday, May 31, 2005

making default column values work in PL/SQL's %ROWTYPE inserts

I'm going to look/post around the web for the answer to this puzzle...

I like PL/SQL's capability to insert a row all at once, using %ROWTYPE.

Example:

CREATE TABLE TEST
(a NUMBER,
b NUMBER DEFAULT 0 NOT NULL);

declare
t test%ROWTYPE;
begin
t.a := 22;
t.b := 0;
insert into test
values t;
end;
/

PL/SQL procedure successfully completed.

However, I'm trying to use it to do inserts on a table with many NOT NULL, DEFAULT 'N' columns, and I get ORA-01400 CANNOT INSERT NULL errors.

declare
t test%ROWTYPE;
begin
t.a := 22;
insert into test
values t;
end;
/

ERROR at line 1:
ORA-01400: cannot insert NULL into ("EQDBW"."TEST"."B")
ORA-06512: at line 5

I don't want to clutter my PL/SQL by manually setting all these columns to their default values; after all, reducing clutter was the whole idea behind using %ROWTYPE in the first place. Besides, if I hard-code the default assignment into my PL/SQL, then it will become out-of-date if I change the column defaults in the table specification later.

If life were perfect, the automatic behavior would be to use the default values for these columns when nothing is specified in the %ROWTYPE variable. I'd settle for having some manual way to specify that as the behavior I want, but I can't find it.

Ironically, there is a way to do this for a record type explicitly defined in PL/SQL - "To set all the fields in a record to default values, assign to it an uninitialized record of the same type". But that doesn't work for %ROWTYPE records (I tried).

reference:
PL/SQL User's Guide and Reference
5 Using PL/SQL Collections and Records
http://download-west.oracle.com/docs/cd/B14117_01/appdev.101/b10807/05_colls.htm#sthref655

Thursday, May 26, 2005

Win32::GuiTest

I have given laborious and painful birth to an automated GUI tester. I'm feeling a bit of postpartem depression, because it's nowhere near as clean as I'd like. I chose some complex ways of coding simply because the obvious ways produced errors or strange results that I didn't understand. It was also hard to get consistent results... things would stop and start working without apparent cause.

Bleh. I don't like programming voodoo.

This article says, "the first rule of GUI testing is: Don’t!", and it's really true. It would have been far better to get my developer to design his GUI app for testability in the first place, or at least get him to embed handy hooks in the code itself. Working totally from the outside, trying to write code that looks at and uses a GUI on the screen exactly as you do, is

But still, there are times when you do have to work as a complete outsider to the code. It is nice that there's a way, however painful. And here are some things I learned. Who knows if they will have any applicability to anybody else's Win32::GuiTest project? I'm sure it depends on a million quirks of the system you're testing.

1. Most things I saw on my screen could not be probed by Win32::GuiTest, as far as I could tell. About all I could see were the titles of windows and window elements, and not even all of those.
2. If you SendKeys, don't forget to bring your window to the foreground with SetForegroundWindow first. For me, the same GUI automatically seized the foreground on one machine, but on another machine, it stayed back, and the wrong window intercepted my keystrokes.
3. I wanted to know how long an hourglass wait lasted while a new window came up. I found no good solution. FindWindowLike succeeded too soon - it found the window immediately, long before the user could see it. What I settled for was to count the number of dependent items that GetChildWindows would find for the new window - and keep counting until the number was up to its maximum value. Then I knew the window was ready for business. Kludgey? Oh, yes!
4. Insert lots of sleep()s. They help reduce errors enormously.
5. I had trouble stringing several tests together into a single session. The first test in a single session would work fine, but inconsistent problems cropped up with subsequent tests. Eventually, I remembered that it's a machine and it doesn't get bored, so I made it close the application, get all the way out, and go all the way back in for each test.

Wednesday, May 18, 2005

timeout with threading

I'd gotten the impression that python's threading module was a very frightening thing, and had put off getting to know it. That's too bad, because when I finally looked at it, it provided a very simple way to add timeouts to some of my code. And - unlike os.fork(), etc. - it works on Windows!

t = threading.Thread(target=self.code)
t.start()
t.join(self.timeout)
if t.isAlive():
completed = False
self.msg = 'Timed out after %s seconds' % (self.timeout)
if self.killOnTimeout:
t._Thread__stop()

Tuesday, May 17, 2005

interactive perl

I'm writing a GUI tester, which involves me deeply in Win32::GuiTest. It's been a mix of joys and frustrations. Two things have saved my mind.
  1. I'm hooked on Python's interactivity, especially when exploring a poorly-documented module. I was missing that badly in Perl; Komodo's interactive mode is only a partial substitute. This tip saved me: http://www.devx.com/tips/Tip/17304?trk=DXRSS_WEBDEV
  2. Getting a module to export its functions was giving me fits. Just adding the function names to EXPORT_OK in the .pm file only gave me "Can't locate
    auto/... (function).al"
    errors. This is black magic to me, but it fixed it: http://groups.yahoo.com/group/libwww-perl/message/1470

Thursday, May 05, 2005

IOUG Live! sent me into rapturous joy. More on that later. I can't believe I was seriously considering not going.

Here's one of the problems that the conference set my mind to.

Some of my homemade procedures fill up some tables with huge amounts of somewhat ephemeral logging data. It's only needed for a couple weeks, after which I need to delete the rows. Doing it the conventional way generates cruel amounts of redo. I've wanted a good script to duplicate the effects of the imaginary command
DELETE FROM logtable WHERE blah blah blah NOLOGGING


The basic algorithm I've seen is
CREATE TABLE newlogtable AS
SELECT * FROM logtable
WHERE (criteria for keeper rows)
UNRECOVERABLE;
DROP TABLE logtable;
RENAME newlogtable TO logtable;

The trouble is that it's hard to get all the dependent DDL to move along. Especially since I don't want any names to change - I name all my constraints, down to the NOT NULLS. The script gets more complicated then.

I found out about DBMS_METADATA.GET_DDL and DBMS_METADATA.GET_DEPENDENT_DDL at Joe Trezzo's "New 10g & 9i features", and it looks like that might be a good basis. It's still tricky, though, because I can't create new dependent DDL until the originals have been dropped. It feels like juggling: generate the new DDL, drop the original dependent objects but not the original table, create the new table with its data based on the stored DDL, then finally drop the original table... hmm.

I'm thinking of relying on the recycle bin (more new IOUG knowledge) for some help here. Deleting objects in 10g really only renames them and designates them as being in a "recycle bin"; you can still access them, you just need to consult the right views to learn their new obscure system-assigned names. (They make an awful clutter in your schema viewed through TOAD, btw.)

Could I drop the table (with its objects) first, then build a new table based directly on the one in the recycle bin? That means all the old dependent object names would already be out of the way, and I can do it all in fewer steps.

Something seems scary about relying on the recycle bin that way, though. Like mailing checks against the deposit you expect to make while your envelopes are still in the mail. I guess Oracle makes no guarantees about the continued availability of objects in the recycle bin; they stay until they're manually purged or the tablespace fills up. But, since it is a very bulky table I'm working with, it's not too farfetched that the recycle-bin version could become unavailable while I'm halfway through this process. Hmm. Darn.

Thursday, April 21, 2005

O-verkill-racle

Oracle's had a philosophy for a while now that, "We're the database company; so anytime our software has to interact with any data, for any reason, we'll have it create an Oracle database for the purpose."

The trouble is, an Oracle database represents a LOT of overhead. For instance, an empty Oracle database takes ten times the disk space of an empty postgresql database. Worse, every new database is a new administration burden.

We wanted to do a little bit - a very little bit - with web-based Oracle Discoverer. That meant installing Oracle Application Server, and OAS created its very own infrastructure database. Now I've got this "infrastructure" database sitting around, doing next to nothing, soaking up almost as much disk space and memory as a small production database would. It's 9.0.1 - way behind my own databases - and I have to attend to its obsolete 9.0.1 security needs, or the network security scan gripes at me.

And the shame of it is - a use like this doesn't need the things an RDBMS provides. A store of configuration data for an application server doesn't need multi-user access, transactional consistency, etc. etc. - totally unnecessary. A few flat files would do fine.

It's like a trucking company that won't use a hand dolly because, well, they're a trucking company. So, of course, they need diesel pumps and regular oil changes and overhauls for the big rigs they use to get the copy paper down to the copy room, and maneuvering the 18-wheelers between the cubicles may be a little troublesome sometimes.