IOUG Collaborate! handouts

If you're on your way to Collaborate!, and getting annoyed at the way you need to search manually for each session just to download its session materials, this script may be handy. It lets you grab the session materials from your personal itinerary.

Be gentle, it was written in a huge hurry.

#!/usr/bin/python
"""Creates a version of your Collaborate! personal itinerary with links to
session materials.

To use:
0. Make sure your machine has Python.  www.python.org
1. Login to your personal itinerary at
http://iougew.prod.web.sba.com/displaymod/ITIntro.cfm?conference_id=44
2. Once your personal itinerary is showing, use Save As to save the webpage
to your hard drive.  Name it PersonalIT.cfm.html.  (This should be the 
default name.)
3. Put this script in the same directory with PersonalIT.cfm.html.
4. Run the script by issuing 'python makeLinks.py' at the command prompt.
5. Open the generated file PersonalIT.withLinks.cfm.html with a browser.
6. The (find materials) links for each title will search for session materials.

By Catherine Devlin (catherinedevlin.blogspot.com)"""
import re, urllib
titleRe = re.compile('(Title:</td>\s+<td.*?>(<a href.*?>(.*?)</a>))', re.DOTALL | re.MULTILINE)
f = open('PersonalIT.cfm.html')
contents = f.read()
newContents = contents
f.close()
sessions = titleRe.finditer(contents)
sessionLinks = [s.groups()[1:] for s in sessions]
for (wholeLink, title) in sessionLinks:
    withNewLink = '%s <a href="http://iougew.prod.web.sba.com/proceedingmod/SearchEvents.cfm?conference_id=44&searchType=2&title=%s">(find materials)</a>' % (wholeLink, urllib.quote(title))
    newContents = newContents.replace(wholeLink, withNewLink) 
newFile = open('PersonalIT.withLinks.cfm.html','w')
newFile.write(newContents)
newFile.close()

oraDifference.py

oraDifference.py - a tool for comparing items that differ between two Oracle schemas. The basic idea is to leverage the excellent graphical diff/merge tools available for file comparison and conveniently use them to inspect database object differences.

There are many programs that can compare two database schemas and tell you which objects are defined differently between them. That's really not good enough, though, because you then need to tediously dig into the definition of each (allegedly) differing object by hand, and perform any desired reconciliation by hand.

I wrote oraDifference.py to make comparing and reconciling schemas more convenient. For example, let's say you have the SCOTT schema in production and development instances. Stored function MYFUNC is defined in both, but the definition differs. View MYVIEW is defined only in development. Then running
python oraDifference.py scott@prod scott@dev
will generate the following batch files (Win) or shell scripts (*nix):

• oraDifferenceResults/FUNCTION/MYFUNC.bat, which will invoke a graphical diff/merge tool showing you precisely where MYFUNC's definition differs between the two instances
• oraDifferenceResults/FUNCTION/MYFUNC-copy-SCOTT-DEV.bat, which will write DEV's definition of FUNC into PROD
• oraDifferenceResults/VIEW/missingFrom-SCOTT-PROD/MYVIEW.sql, the definition of MYVIEW
• oraDifferenceResults/VIEW/missingFrom-SCOTT-PROD/MYVIEW-copy-SCOTT-DEV.bat, which writes MYVIEW into PROD

For now, you have to do the work of getting oraDifference.py (and sqlWrap.py) manually and putting them someplace appropriate. I do intend to wrap them up in a proper distutils distribution (maybe even with an .egg).

I'm posting this now because I find it really useful already. You may find some of my design decisions quirky - for instance, I mush all one-liner items (grants, synonyms) into big files by category, rather than making separate files for each grant or synonym. It's Python, though, so you should be able to tweak it to meet your tastes. Also, you can tweak the process oraDifference.py uses to decide whether two objects differ. I have always been annoyed that I can't stop TOAD's "Schema Compare" tool from turning up dozens of "differences" that I consider false hits. With oraDifference.py, you can just get in there and change it.

Eventually I hope to release something that will look polished and final, but for now, feel free to use it, and re-code any part that doesn't match your preferences - and let me know about any of your changes that you think should go into everybody's version.

For fairness, I'll mention some other options I found for schema comparison...

• LivingLogic's oradiff.py (part of ll.orasql) is the closest to oraDifference.py. It also compares the text for each object, but it outputs in unified diff format (or unidiff). If you can read unidiff comfortably - welcome, advanced extraterrestrial visitor! The oradiff.py source code looks tidy and well-organized, but it's still not obvious to me how to tweak it.
• schemaCompare, a Java program, was registered at SourceForge in June 2002, but has not yet released any files. I conceived oraDifference.py about two weeks ago. Not to suggest that this implies anything about the relative productivity of various languages. (jab, jab)

Python Core for Oracle

He put it in a comment, but it bears repeating:

Przemek Piotrowski has written up Python Core for Oracle, a set of instructions to put a top-to-bottom data-driven webserver stack on your machine in about half an hour. The installation is surprisingly straightforward. It's all fully functional, and it's all free.

This is the Golden Age!

Cheetah templating

Yesterday, Python's str.Template failed me, so it was finally time to learn Cheetah.

I wanted to use templates like
'my list has $len($myList) objects; the first is named $myList[0].name.upper()'
... but, of course, that sort of stuff is impossible with str.Template. I created a bunch of code to populate a dictionary to pass to str.Template, but that was clunky, and defeated the purpose of having a template that clearly describes its own contents. In Cheetah, it's perfectly straightforward.

from Cheetah.Template import Template
tmplt = 'my list has $len($myList) objects; the first is named $myList[0].name.upper()'
print Template(tmplt, [locals(),globals()])

The second argument is the list of dictionaries Cheetah will search for matches to variables in the Template. Using [locals(), globals()] is my way to cheat and say, "Look wherever the interpreter would".

Oracle XE and Ubuntu

WOW. I just installed Oracle XE on my Ubuntu machine. I absolutely cannot believe how easy it was. This is - honest to goodness - all I did.

1. Download oracle-xe_10.2.0.1-1.0_i386.deb
2. su - root
3. dpkg -i oracle-xe_10.2.0.1-1.0_i386.deb
   It ran for maybe thirty seconds - so short, I was certain there had been an error!
4. /etc/init.d/oracle-xe configure (it told me to do that)
5. pointed Firefox at http://127.0.0.1:8080/apex (it requested that, too)
6. Started using the database (plus its included Application Express).

The entire installation took less than five minutes. Unbelievable! "This is Oracle?"

The only glitches I've gotten so far were when using Python's cx_Oracle against XE, and I've puzzled them out. (I don't know whether other people will get these glitches; they could have resulted from some residue of the full-fledged 10.2 Oracle that was on the machine before.)

• import cx_Oracle gave ImportError: libclntsh.so.10.1: cannot open shared object file: No such file or directory until I set LD_LIBRARY_PATH=/usr/lib/oracle/xe/app/oracle/product/10.2.0/server/lib/
• conn = cx_Oracle.Connection('scott/tiger@xe') gave RuntimeError: Unable to acquire Oracle environment handle until I set ORACLE_HOME=/usr/lib/oracle/xe/app/oracle/product/10.2.0/server

[EDIT Oct. 9, 2007: Configuring cx_Oracle with Oracle XE has turned out to be harder than expected. See my new blog post.]

[EDIT Mar. 6, 2008: Great instructions for installing straight from Oracle's repository with apt-get here]

summary of Oracle/Python discussion

My OTN article on Oracle and Python was kept very brief, to be non-intimidating and to fit within OTN's preferred length. If you've come here, though, you're ready for the rest of the story! I'll use this post to summarize that discussion.

• Python+Oracle on other Linux distributions - see Andy Todd's blog entry
• alternatives to fetchone(): fetchmany(), fetchall(), and looping directly on the cursor - see my last entry and this comment
• Passing an argument to split(), to avoid errors on more complex init.ora parameters - see this comment

OTN article addendum

If you've read my new article at the Oracle Technology Network, Wrapping Your Brain Around Oracle + Python, thank you! I'd like to add a few more details about fetching rows with cx_Oracle that can make your code even cleaner.

Several times, I demonstrate getting rows from a cursor by means of the cursor's .fetchone() method. .fetchone() is used in loops like this:

curs.execute(<some query>, <bind variables>)
aRowOfData = curs.fetchone()
while aRowOfData:
    <commands>
    aRowOfData = curs.fetchone()

Another, more concise, alternative was not mentioned in the article. The cursor object itself can be iterated over, like this:

curs.execute(<some query>, <bind variables>)
for aRowOfData in curs:
    <commands>

The effect is the same, but it works with two fewer lines of code.

Finally, the .fetchall() method, bringing the entire result set into a list at once, was only briefly mentioned in the article, but it would probably be preferable to .fetchone() for the small result sets we'll find in places like v$parameter. Only when a result set is very large (or your computer is very memory-limited) do you need to worry about .fetchall()'s impact on your system's available memory.

Alice: corrupting the youth

I wish I could remember which PyCon delegate told me about Alice. It's a graphical environment for programming animations in a very kid-friendly fashion, yet full of solid, object-oriented programming goodness. This generation's LOGO, I suppose.

I was hoping it would fire our nine-year-old's interest in computers. I think it's working; Star Wars: Battlefront never produced such delighted shrieks and giggles. Maybe it works too well. He refused dessert to spend the extra couple minutes with Alice. Choosing code over food - I always thought of that as late-stage geekery.

Cheer: Alice works hard to be girl-friendly. Boo: Not available for Linux.

new and improved sqlWrap.py

After reading the Python Cookbook and attending PyCon, I have enormously improved sqlWrap.py, my Python module for handling DB-API 2.0 connections conveniently.

Also at PyCon, I learned that I could be accused of having duplicated projects like

• ll-orasql
• db_row
• dtuple

Like them, I allow tuple-like, dict-like, and object-like access to fields. I like mine better, though; it requires less preparation, and has really nice reporting methods. For example,

conn = sqlWrap.SqliteConnection('myDb.sqlite')
print conn.select('myTable').xml()

... is all you need to get an XML report on myTable. Similar reporting methods exist for

• tables in pp (prettyprint), xhtml, ReStructured Text
• transposed tables in pp, xhtml, ReStructuredText
• SQL INSERT statements

Hey, if everybody gets to write their own web app platform, why shouldn't I write my own DB-API wrapper?

PyCon2006

No, Guido doesn't know I have this... I snagged it from the badge reuse box. You keep your smelly old rock star T-shirts, I'm keeping this.

So... PyCon. Wow. It was wonderful, of course. What else could it be? Put 400 people with that much intelligence, creativity, and energy in one place, and it can hardly help but be wonderful.

What surprised me is that I found so many good ideas no matter what I was doing at PyCon. Whether I was in a talk that I expected to benefit hugely from, or a "Probably useless but I guess I'll go anyway" talk, or just chatting between or after sessions, I seemed to learn good stuff constantly just the same. The serendipity was worth more than the "planned" learning, and maybe that's what makes conventions so much more useful than formal books and classes and so forth.

Python Cookbook

This may seem silly, but I really have to sing the praises of the Python Cookbook, 2nd Edition. Everybody else has known about it forever, but I just got my copy a month ago.

I am blown away. I can't believe how good this book is, far beyond any other programming book I've ever known. I'm now completely embarrassed about the quality of the code I wrote without it, and tempted to stay up all night refactoring everything. In fact, I've already largely rewritten sqlWrap.py based on what I've learned.

I'm afraid I dawdled about buying my copy because I never found the recipes at ActiveState all that compelling - occasionally nice, but usually nothing to jump and scream about. I figured the Cookbook would just be a bunch of them, bound together. Not so - the careful selection and excellent discussion make it amazing. It's like having a master programmer at your elbow to guide you.

The Geek Event Aggregator is ready!

The Geek Event Aggregator is more-or-less ready for prime time! It now collects many more events. Better yet, it is very easy to feed more event sources, so it is set for even more growth. In other words,

PLEASE FEED THE AGGREGATOR NEW SITE SUGGESTIONS!

Some design notes:

After fooling around with complex regular expressions, Beautiful Soup, etc., I found a quick-and-dirty way that works better. The Aggregator downloads a page's HTML, replaces all tags with carriage returns, breaks the remaining text into lines, and checks those lines for ones that appear to contain recognizable future dates. The Aggregator assumes that is an upcoming event date. (There are many reasons dates get put on websites, but future dates almost always refer to meetings or events.)

Wow, human beings have many, many, many ways to write dates. Fortunately, python's dateutil module can recognize most of them - I mainly have to modify that to avoid false hits (like interpreting '.' as 'today', or '2006' alone as 'Jan 1, 2006').

Some sites already aggregate events from several groups and places together; for those, the Aggregator uses a slightly different algorithm. It finds the dates as above, then finds the location and event name by their line-number position relative to the date. (A human being (me) needs to provide relative line-number positions for those values in advance; for instance, one site may always list location on the line immediately after the event date, and that fact is recorded in advance.)

For the multi-event sites, the Aggregator has a decent but kludgey algorithm to parse city and region, despite the great variety of ways to write a location. Part of that relies on a list of recognized city names. It can be used for the single-event sites, too; if a recognizable city name is in the site's Title, or in the text in the form "Blah Blah City Blah Blahware Group", the Aggregator can find it. But if your event is in Athens, GA, the Aggregator thinks it's in Greece.

HTML DB makes it very convenient to build a web interface to the data. Oracle is also very gracious to host a free sandbox for HTML DB projects (which is where the Aggregator lives right now.) Unfortunately, as far as I can see, HTML DB doesn't support RESTful interfaces, or serving up pages as XML. That's a pity, because this cries out to be a REST web service. Maybe eventually I'll buy/find a place to host the web app in TurboGears or something.

For now, I've given up on feeding upcoming.org. Upcoming mandates an actual street address, which is just too hard to find automatically. I'd still like to pull from it, although I'll have to check their legal requirements, and - dare I say it? - I don't know if it will really have many relevant events I don't already have.

I'm sorry the events are so U.S./Canada-centric. It only has a handful of events from elsewhere, and it doesn't break down regions within other countries. (What's wrong with going from St. Petersburg to Novosibirsk for a meeting, anyway? Isn't that what the Trans-Siberian Railway is for?) You can help fix this by suggesting new sites to scan, and volunteering to introduce regional granularity for other countries.

Actually, because I've been the only one to feed the Aggregator so far, the events are Ohio-centric. You folks in benighted backwaters like California and New York are just going to have to feed it your own favorite sites if you want to fix that.

Some of the many things that produce misses and false hits:

• Dates without years. Somebody puts on the website, "Our next meeting is Nov. 19." The Aggregator can't tell that they haven't updated the site since 2004.
• Years must be on the same line as the rest of the date. If you say,
  2005 events:
  Feb. 14
  Aug. 12
  ... the Aggregator doesn't see the "2005", and believes there are events on Feb. 14 and Aug 12 of this year.
• Frames. Well, you can't blame it; frames mess up everybody. But if you can dig into the HTML source and puzzle out the URL of the frame with the data, then that can be read. That's what I did for the OKCOUG webpage, for example.

Geek Event Aggregator's future

The strangest thing happened to me this morning. The clouds split open, and a beam of light shone down onto me. (Since I was sitting in my cubicle, this in itself was odd enough.) A Voice spoke from Heaven, and said,

"Catherine, remember that somewhat cheesy Geek Event Aggregator you put on HTML DB a while ago?"

"Yes, Lord?" I said. (That seemed like the obvious response. That the Almighty would speak with hyperlinks didn't seem too surprising.)

"Well, I have heard the cries of my people. I want you to rewrite your aggregator to both consume and provide events in iCalendar format. Also, interface it with upcoming.org - both to download and to upload. Behold, Python libraries for upcoming.org have already been written for you. And I'm thinking that Beautiful Soup might help you pluck event descriptions from the God-awful HTML jungles you find them in."

"It sounds wonderful, Lord," I said. "But when am I supposed to do this? I mean, you did just drop custody of our Godson into our lap. Time is not exactly abundant right now."

There was a long pause, then the Voice said, "Let me get back to you on that."

cleanPath executable ready

An executable version of cleanPath is now available here!

You do need .NET Framework 2.0 to run this. It's pretty painless to download and install. You don't need IronPython.

Thanks to the newest IronPython release for making this possible! When I run

ironpythonconsole cleanPath.py

, the code runs, and it also generates cleanPath.exe without even being asked to. Until now, though, that .exe didn't actually work. In IronPython 1.0 Beta 2, however, released just Wednesday, it works great! I do need to put IronPython.dll in my directory before I compile... other than that, it's so simple, even a DBA can do it.

cleanPath.py: an IronPython utility

During my CodeCamp presentation, I introduced a very sketchy (and logically flawed) script to examine a PATH environment variable and clean out unwanted elements.

Since then, I created a much-improved version in IronPython which, I think, is actually worth using. Features:

• Conveniently view and delete path items
• Nonexistent directories and duplicate items are flagged, and can be purged with one keystroke
• works on PATH, CLASSPATH, PYTHONPATH, PERL5LIB, etc.

It's Windows-only, but only Windows makes editing your PATH such a pain, anyway.

The code is here.

Right now, you need IronPython, and use "ironpythonconsole cleanPath.py" to run it. I need to figure out how to generate an .exe, since getting set up with IronPython is too much to ask just to clean up your PATH.

[EDIT: An executable is ready now. See my next blog entry.]

Smurfette syndrome

One thing about the CodeCamp, though: severe Smurfette syndrome. Maybe 10% of the attendees were women, but probably closer to 5%.

I don't think it's the CodeCamp's fault; I'm afraid they may have gotten an accurate sample of the programmers in the area, and that's the scary thought. Possibly things would have been a little better on a weekday; men might feel more comfortable in taking a day away from home duties for their professional development. Still.

I know, it's been talked to death, and there are a million theories and stuff, but... yikes. It's just scary that one year goes by after another, and the situation persists.

On the other hand, the GCOUG meeting was about half women, maybe even a majority. At a small meeting, granted, but it's still a bright spot.

Dayton-Cincinnati Code Camp review

If you missed the CodeCamp, don't miss the next one. It was great. I'm not even into .NET, but even the minority of the conference that was non-.NET was well worth my time. Plus, sampling a bit of the .NET take on a few topics was good for my general education.

I almost wish I were involved in .NET, purely for the sake of the Dayton .NET group, which has a vitality that you just don't see that much of in the Dayton area IT community (at least, in my tracks thereof.)

Anyway, a few things I learned...

• There really are Python-experienced people in the area; I'd just never found them before.
• Ruby on Rails supports Oracle. Uh-oh. It's not that I'm unwilling to learn Ruby; it sounds great, and (of course) simple to learn. It's just that I really want to use TurboGears, but it uses SQLObject, and Oracle support in SQLObject is the sword in the stone. Many have tugged, but none have completed it, and maybe I should stop waiting and just use RoR for now.
• Speaking of Ruby, never miss seeing Jim Weirich speak. I already knew that, but maybe you didn't.
• GCOUG really needs to work on getting more people involved. There were 15-20 people at Jimmy Brock's talk on Oracle Developer Tools for .NET - none of whom had more than the haziest awareness of GCOUG's existence.
• Speaking of Jimmy's talk, Oracle Developer Tools has a great and handy way to bring .NET assemblies from Visual Studio all the way into Oracle, where they can be used like stored procedures.
  So, one glorious day, this year I hope, Visual Studio will support IronPython, and I'll be able to (essentially) write my Oracle stored procedures in Python instead of PL/SQL. And choirs of angels will sing.

I'm sure I'll have more to say about the Camp as I think about it.

Presenting (with) Bruce

When I present at a meeting, I like to use plenty of live demos. I can get away with this, because I type very, very well. (I did get into IT via the secretarial pool, after all.) Still, even I get a little nervous about whether I can keep up typing speed and accuracy with the pressure of a whole audience staring at me.

Fortunately, This blog post by Andy Todd (You do read his blog, don't you?) pointed me to Bruce, a wonderful presentation tool freshly created by Richard Jones.

Bruce lets you wrap up decent-looking slides, graphics, and Python interpreter sessions all into a single presentation that scrolls smoothly along at your command. (Simply being spared the fuss of flipping between Presentation slides and a command-prompt window is a blessing.) For demos, you can pre-enter the text you want to type, then Bruce sends it to the interpreter one character at a time. You type 'dfjkasfjdska;dfsafd', but 'def plusOne(x): return x+1' appears on the screen, and the audience thinks you're a keyboard goddess.

My one disappointment in Bruce - as I understand it (I just found it, you know) - is that the wonderful interactive session can only be used with a Python session. I'll need to dig into Bruce and PyGame to verify that, and to see if it can be extended elsewhere. I dearly want to use it when I'm demonstrating SQL.

Until Bruce, incidentally, my best bet was TPP, the Text Presentation Program. I've seen presentations done with it, and it looks OK, except it's written in Ruby and has dependencies (ncurses) that need C compiling - not very practical where I am, unfortunately.

Search engines are not Zen

I decided that I should make myself a list of questions I've never been able to resolve, or that were very difficult to research, because they don't "search" well. That is, in everything from Metalink's Search bar to Google, it's hard to come up with relevant results.

For instance,

- I'm trying to use RMAN DUPLICATE, and it does indeed produce the data files on the auxiliary database, but it doesn't create a controlfile. The documentation clearly says it should. Searching on terms like RMAN DUPLICATE controlfile not generated only brings up more pages saying that it should.

- All sorts of questions like such-and-such service DOESN'T start, program WON'T run, logfile ISN'T created. Negation - "not" - is clearly a failing of a typical search. Searching for green eggs no ham will find you 10,000 pages on green eggs AND ham, with the one page on "My green eggs came without ham!" buried anonymously as result #4,312.

I'm not sure what I'll ultimately want to do with this list. But I began to wonder, more broadly, if someone out there had already started assembling issues of unsearchable questions. Ironically, Can't Find On Google is very easy to find on Google.

Dayton - Cincinnati Code Camp

Hooray! - to the Dayton .NET Developers Group for putting together the Code Camp on Jan. 21, 2006 - a whole day of community-provided seminars on a variety of programming topics. For free! I'm looking forward to it; in fact, I'm planning to present there (on Python, naturally).

If you're anywhere near southwest Ohio, come and enjoy. Better yet, come and share! Whatever cool programming tricks you have up your sleeve - come and show the rest of us!