Ohio LinuxFest slides & code

Thanks to everyone for your attendance and feedback at my OLF talk! I had a great time!

A tarball with everything you need to duplicate my presentation is here.

It's pretty complex, so it includes a README file. (I used s5 slides generated by rst2s5, a Socrates file from Bruce, and eight directories of code files. Don't worry, though, it's very lean - just 60K.)

I'm speaking on Python at OLF

What's this? You haven't registered for LinuxFest yet?

But I'm (absurdly honored to be) one of the speakers!

Introduction to Python and/or Solar System Demolition
An introduction to Python programming. Demonstrates the language by developing a graphical solar system gravitation simulation in real-time.
September 29th, 2007, 3:50 - 4:40 PM

Greater Columbus Convention Center

400 North High Street

Columbus, OH 43215 USA

And I'm bringing my own projector this time, just in case... (plus I've got nVidia drivers installed, which should help...)

Ohio LinuxFest: register now!

Drop everything and register for




It's free - but the registration deadline is Sep. 14 - this Friday! Move, move, move!

Ohio LinuxFest
September 29th, 2007

Greater Columbus Convention Center

400 North High Street

Columbus, OH 43215 USA

Microformats talk

Have I mentioned that the Dayton Web Standards Meetup is a great group? It's brimming over with good ideas for every sort of web writer. Very highly recommended!

This coming Thursday, I'll be speaking on Microformats! Microformats are a wonderful way to make your information available to automated tools, while mandating nothing about what else you do on your page. They're great fun, they're catching on, and I'm looking forward to it!

5:30, Sep. 6: Microformats talk at Dayton Web Standards Meetup
at Panera Bread on 1203 Brown Street. Dayton OH 45409 USA (937) 223 6700 microformats web meetup See you there!

Okay, I admit, I wrote this post partially to practice writing a microformat. Install Operator into your Firefox. Now, with one click (each), you can grab a vCard for Panera, get a map there, or pop the event inot your calendar program. Oooooh!

Dayton-Oracle is for real

The Dayton-Oracle mailing list I set up a couple years ago is bearing fruit. The Dayton Oracle User Group is real! We live! We had an organizational pre-meeting last night with five great people who were willing to take up the burden of making this happen.

We had a lot of rich discussion about the way we want to approach running the group. In many ways, we're going to break from the traditional format and content of Oracle User Groups and adopt some practices better-known from the open-source world: more interactive, user-driven content; zero-cost bring-your-own-food arrangements instead of catered lunches with meeting fees and sponsored content; evening rather than daylong meetings; etc. I think these are good practices on their own merits. Also, there are already good traditional user groups in Columbus, Cincinnati, and Indianapolis, so if we want a successful niche between them, it makes sense for us to differentiate; we're not likely to be a better OOUG than OOUG, after all. I hope Dayton people who do already attend OOUG or GCOUG will continue to do so, and will enrich themselves yet further with our meetings.

Watch this space for further announcements, including our first meeting time (sometime early October) and a link to our super-awesome website (once it's built).

openSUSE and the bleeding edge

I fear I'm about to sound both boastful and ungrateful... but my boss refreshed my work laptop with this monster I have dubbed "DellZilla". Cool, huh? Definitely - except that you need a pack mule to actually move it. Once upon a time, laptops could actually fit in your lap.

The other problem is that I haven't found a Linux distribution that comes with the right drivers for it. Ubuntu Feisty was hopeless - all incarnations (u-, ku-, and xu-) were flummoxed by the screen and failed to start xwindows, or gave an utterly black empty screen.

Next I tried openSUSE, and was in some ways pleasantly surprised. It copes OK with whatever video voodoo is required for this monster screen. It automatically noticed that I had a Windows Vista partition and included it as a boot option. In fact, it includes "Restart and boot directly into Windows" as a menu option, so you can go get a cola while switching OS's instead of hovering over the keyboard waiting to catch the boot loader. Considerate! And it comes with all the cutting-edge goodies like Beagle. (And freeciv! Eek!)

One significant complaint is that, if you don't have wireless connectivity while you're doing the openSUSE installation, YAST won't even record the URLs of the repositories you'll need for software. When you need to download the good stuff later on, you'll need to hunt around for the URLs (psst, they're here) and add them to your list of sources by hand. For a SuSE newbie, that's a pain.

But I got through that. The real problem is that I've got no support for the mac80211 wireless card. Trying to get a driver installed for that has been an adventure in frustration. There's a driver here, but it appears to my amateurish eye as though YAST installs the Linux kernel-source into locations where the mac80211 makefile absolutely does not expect it, and trying to hack them into agreement is a step too far for me. It's like being back in the Bad Old Pre-Ubuntu days when Linux was something I only wished I could use.

So here I am, typing on a zillion-dollar laptop and tethered to the wall like it's 1994, and trying to figure out my next move. This says Sabayon 3.4 supports mac80211. I hadn't even heard of Sabayon before this week, and I've read good things about it, but I wonder if it's just too fancy. It installs from DVDs; I've never even tried burning one of those. And I can't even get its blasted homepage to come up under Firefox or Konqueror. But whatever; I need wireless. If you don't hear from me for a while, it's because I need to compile my own drivers for the keyboard, or maybe the AC adapter.

birthday list

Pardon my long absence from the blogosphere. One cause has been that my workplace blocked access to Blogger, all Blogger-hosted blogs, all mailing lists anywhere, etc. I'm becoming thoroughly frustrated. The message it seems to send is, "We don't desire the fruit of your work as much as we value our infantilizing mistrust of your professionalism. We pay for the services of skilled adults yet believe that we got delinquent children instead."

But anyway. That's not why I'm posting. I've been fairly nomadic lately, migrating my work from machine to machine and experimenting with various operating systems, and it's made me crave some good organizational software. Since my birthday is coming up, I thought I would ask the Universe to produce it and see what happens.

1. I need help keeping track of the difference between files I've worked on - stuff I want to keep when I move or rebuild - and stuff I can leave behind because I just downloaded it from somewhere and I'll just download it afresh on a new system anyway. Then it'll be easy to back up and transfer my work. This software will note when I modify a file with a text editor, IDE, image editor, OpenOffice, or whatnot, and will flag it as my own personal work - a "keeper". Stuff that comes from downloads, from expanding zipfiles, automated processes and loggers, etc. will all be flagged "ephemeral".

2. I need help reorganizing my folder structure. I'd like something that can scan my drive and notice that my photos are scattered in four different places, short text notes are all over, I've got four copies of the same file in four directories under three names, etc., and propose and facilitate a consolidation plan.

3. A nice tag-based file access system That could actually replace #2 and be a key component of #1.

If you know of this stuff already existing, speak up! If you want to write it, hooray! If you want to wait several years until I have the skill and the time to do it myself, well... keep reading my blog, I guess. If you are saying, "You wouldn't need any of this if you exercised organizational discipline when creating your files", then please note that I am Sticking My Tongue Out At You.

get a job, sha na na na

If you're reading this and you're interested in switching jobs, let me know...

... because I'm constantly being asked these days whether I know people looking for work. It's a good feeling, but it would be a better feeling if I had somebody to refer them to. And I'm sure I'm not the only one getting these inquiries, so if you've felt stuck writing boringCode for a pointy-haired boss, it's a great time to let your friends know.

This applies to Python people as well as Oracle. That's a neat change; when I got involved in Python a few years ago, the assumption was that most Python programmers had day jobs using more boring languages, and could only use Python in their passion-driven midnight coding sprees. Now, employer interest is very high - it seems employer interest has actually grown faster and overtaken the population of Python coders. That's especially noteworthy when you consider how quickly someone can become a Python programmer.

Anyway, good times for Py and Ora geeks. Woo hoo!

Kubuntu

For Feisty, just to learn something new, I thought I'd try Kubuntu (KDE Ubuntu) instead of Ubuntu this time around.

And... and... I think I'm a believer.

For what I've done so far, the extra information KDE provides is very well thought-out, just what I need to know when I need to know it. For instance...

• When I want to remove a USB drive or somesuch, but something is still using it, Ubuntu (like Windows, including Vista) only tells me that something is using it. Kubuntu tells me what is using it.
• Alt-Tabbing through my open windows gives more information than just the icon of each running program - enough so that I can distinguish between my eight different command prompt windows.
• Hovering over an icon pops up the file properties. Nice! I mean, why else would I be hovering over an icon, except that I was fumbling for the right-mouse button so I could find out more?
• Clicking my networking icon provides all the information I need to choose at a single glance, and then one more click connects me. Also, Kubuntu seems to jump to conclusions and do the right thing for networking - when there's only one wireless network around, Kubuntu just connects me without waiting to be asked.

The downsides I've found so far...

• I really liked Ctrl-+ to increase the font size of GNOME windows. That's important when trying to read laptop screens in adverse environments (like outside). I haven't found a KDE equivalent.
• The built-in text editor, Kate, bungles the spaces/tabs in Python files. That's one good reason to install Komodo (free editor or the real thing) immediately.

Real gurus can discuss the relative strengths of KDE and Gnome for hours, of course, and I thank God that I've never been caught in such a debate. I'm just saying that KDE is making a really good first impression on me, and that's after years of Gnome experience.

Extensible CheckBox List

Introducing a brand-new widget for TurboGears: Extensible CheckBox List!

To use it:

1. easy_install xcbl
2. In your controller.py,
   from xcbl.widgets import ExtensibleCheckBoxList
3. In your form's WidgetsList,
   pets = ExtensibleCheckBoxList(options=['cats','dogs','turtles'])
   (exactly the same as you would for an ordinary CheckBox List widget)

The result - a CheckBox List with an "other" field that lets the user hand-enter new options - as many as they want.

Unfortunately, the new items will appear beneath the "Other:" box, not above it, unless you upgrade your MochiKit to 1.4. Details on how to do that and more are here.

Developing it was lots of fun - it really shows how easily a relative amateur can contribute to TurboGears. I hope to write up a "Developing and Contributing TurboGears Widgets for Dummies" document soon, based on the experience.

cleansing table and column names

What kind of joker thought "BASE POC'S" would make a good column name? Sheesh, MS Access. (Not to mention apostrophe abuse. See Bob's Quick Guide to the Apostrophe, You Idiots.)

To my surprise, I googled "script rename invalid columns oracle" and got nothing useful. So here's a chance to play with Oracle regular expressions.

You may be thinking, "If they're illegal column names, how can they get in the database in the first place?" With "quotation marks", that's how. In fact, you can

CREATE TABLE "!!!My Rockin' Table!!!"
  ( "*^@(@  @!!!" VARCHAR2(30) 
                  CONSTRAINT " -- d00d --" PRIMARY KEY,
    "miXiNG CASE is TEH KEWList" VARCHAR2(30) ); 

and, not only will all your hideous names be used verbatim, but they will be case-sensitive. If you take advantage of this feature, you should be slapped. I guess the reason Oracle makes this possible in the first place is so that an import can be accepted from a source - like MS Access - regardless of how awful its names are.

Anyway, here's a PL/SQL cure for bad table and column names.
(link to code)

/*
Renames tables and columns to be Oracle-legal.

SET LINESIZE 200 before running to avoid unwanted page breaks.
Run from an account with privileges to
  - access v$reserved words 
  - alter the appropriate tables

Thanks to Eddie Awad for his description of v$reserved words:
http://awads.net/wp/2007/01/10/what-does-reserved-y-really-mean/

Catherine Devlin, catherinedevlin.blogspot.com, 11 May 2007, v0.1
*/
CREATE OR REPLACE PROCEDURE cleanse_names IS
  new_obj_name VARCHAR2(30);
  CURSOR tbl_names IS
    SELECT owner,
           table_name
    FROM   all_tables
    WHERE  dropped = 'NO';
  CURSOR col_names IS
    SELECT owner,
           table_name,
           column_name
    FROM   all_tab_columns
    JOIN   all_tables USING (table_name, owner)
    WHERE  dropped = 'NO';

  PROCEDURE do(this IN VARCHAR2) IS
  BEGIN
    DBMS_OUTPUT.PUT_LINE(this || ';');
    -- Uncomment if you're feeling decisive
    -- EXECUTE IMMEDIATE(this);
  END do;
  FUNCTION legalize(name IN VARCHAR2, replacement_letter IN VARCHAR2)
  RETURN varchar2
  IS
    new_name VARCHAR2(30);
    is_reserved NUMBER(11,0);
  BEGIN    
    -- must begin with a letter
    new_name := REGEXP_REPLACE(name,'^[^[:alpha:]]',replacement_letter);
    -- and contain only letters, numbers, #,_,$
    new_name := REGEXP_REPLACE(new_name,'[^[:alnum:]#_$]','_');
    -- and mixing case is evil
    new_name := UPPER(new_name);
    -- and we must respect Oracle reserved words
    SELECT count(*)
    INTO   is_reserved
    FROM   v$reserved_words
    WHERE  keyword=new_name
    AND    (reserved='Y' OR res_semi='Y');
    IF is_reserved > 0
    THEN
      new_name := SUBSTR(new_name, 1, 29) || '_';
    END IF;
    RETURN new_name;
  END legalize;
BEGIN
    DBMS_OUTPUT.PUT_LINE('set scan off');
    FOR col IN col_names
    LOOP
      new_obj_name := legalize(col.column_name, 'C');
      IF new_obj_name != col.column_name
      THEN
        do('ALTER TABLE ' || col.owner ||
           '."' || col.table_name ||
           '" RENAME COLUMN "' || col.column_name ||
           '" TO ' || new_obj_name);
      END IF;
    END LOOP;
    FOR tbl IN tbl_names
    LOOP
      new_obj_name := legalize(tbl.table_name, 'T');
      IF new_obj_name != tbl.table_name
      THEN
        do('ALTER TABLE ' || tbl.owner ||
           '."' || tbl.table_name ||
           '" RENAME TO ' || new_obj_name);
      END IF;
    END LOOP;
END cleanse_names;

Larry Wall, God, Lois McMaster Bujold

Part of what got me thinking about my last post is that I'm suffering a little crisis over Larry Wall. I'm really admiring his spirit. For instance, I like his way of expressing his Christianity.

As if that weren't enough, he's into The Lois, as quoted here:

Note how we still periodically hear the phrase "serious literature". This is literature that is supposedly about Real Life. Let me tell you something. The most serious literature I've ever read is by Lois McMaster Bujold. Any of you read her? It's also the funniest literature I've ever read. It's also space opera. "Genre fiction," sneers the Modernist. Meaning it follows certain conventions. So what? Nobody in the world can mix gravity and levity the way Bujold does in her Vorkosigan books. It's oh so definitely about real life. So what if it follows space opera conventions. Sonnets follow certain conventions too, but I don't see them getting sneered at much these days. Certainly they were always called "serious".

How long till Bujold becomes required reading in high school? Far too long, in my opinion. Horrors. We wouldn't want our students actually enjoying what they read. It's not--it's not Real Life.

As if the Lord of the Flies is real life. Feh.

Yes, I'm a Lois fan, too. I once drove hundreds of miles to an SF con purely because she would be there...

So, anyway. Yaaaay, Larry. But there's no getting around this: I prefer Python. Perl is great, Ruby is great - all dynamic languages contribute to each others' success, I think, because the prominence and respect won by each rubs off the others, so we can get our necktie-level blessing. But, when it comes down to what I use, I can't write anything else these days without wishing it were Python.

getting a bit philosophical

Have you had the experience of struggling with a problem for hours, going to ask for help with it, and then realizing the answer to the problem while you're in the middle of asking the question?

Of course you have. It's an amazing phenomenon. It really shows how much the brain depends on language to think clearly. It's one reason I'm such a junkie for user groups and conferences! And it's not good enough to just hear it or read it - you have to speak it or write it to really engage your brain. Similarly, sometimes it seems like you never really understand a subject until you have to teach it to somebody else.

Meanwhile, it's become an axiom of conversational politeness that politics and religion are to be avoided in most contexts. Which means... which means that we're forbidding ourselves to use our most powerful means of stimulating clear thought... for some of the most important ideas we have to think about. Hmm...

Microformats

As I work more on the Geek Event Aggregator and similar projects, I'm getting really fascinated by movements like microformats and POSH - efforts to make web content more semantically meaningful and machine-readable, without trying to radically reinvent the web. Just small tweaks to the HTML we've been writing all along can make a world of difference - indeed, often just using HTML best practices that have been around (and largely ignored) forever. Interesting stuff! I'll probably be blogging about it more in the future.

One related concept is blogchalking, so here goes.

This is my new blogchalk:
United States, Ohio, Dayton, Brookville, English, German, Catherine, Female, 31-35, Python, environment. :)

That's for machines to read. Humans, avert your eyes.

Let's see, I have about three more months before I have to update that age category.

Unicode error with TurboGears and Oracle

Thanks to kap_ravi's comment, I tracked down a surprisingly big and poorly-documented stumbling block for folks using TG with Oracle.

I've updated the original TG+Oracle blog post to reflect it, but the gist of it is that you're likely to get

SQLError: (NotSupportedError) Variable_TypeByValue(): unhandled data
type unicode

errors unless you include

sqlalchemy.convert_unicode = True

in your .cfg files.

Configuring Oracle for proper Unicode use may avoid the need for this; I'm not sure. My dreadfully mononational working environment has prevented me from developing any Unicode experience.

good errors vs. bad errors

One of the things I love most about PostgreSQL is its beautiful error messages. For instance, let's say I've inexplicably forgotten to add my home state to a table of states, thus violating a foreign key constraint when I try to insert a Minnesotan city. Here it is in Oracle:

SQL> insert into city values ('International Falls','MN');
insert into city values ('International Falls','MN')
*
ERROR at line 1:
ORA-02291: integrity constraint (MYSCHEMA.FK_CITY_STATE) violated - parent key not found

OK, that doesn't suck. In this case, it's pretty easy to figure out the problem, especially since I gave my constraint a meaningful name. However, there are cases where the foreign key is not so obvious - maybe it involves multiple columns - and I'm sometimes sent off to look up details on what FK_CITY_STATE involves. Since DESCRIBE won't work on a constraint, that means either an intricate query joining several V$ tables, or a graphical tool like Toad or SQL Developer.

Now I'll make the same mistake in psql.

mydb=# insert into city values (2,'International Falls','MN');
ERROR: insert or update on table "city" violates foreign key constraint "fk_city_state"
DETAIL: Key (state)=(MN) is not present in table "state".

Now that's clarity! It's one of the things I love about PostgreSQL. After all, the database has all this detail easily accessible to it; why shouldn't it deliver it to me along with the error message? It's even more important when an error like this is thrown from somewhere deep inside some code, where it may not be obvious to me which statement or which row is responsible for it - having the offending value displayed is incredibly convenient. It's considerate programming - no more, no less.

I'll say this for Oracle, though - Oracle's standardized ABC-12345 error codes are really handy for looking them up, for searches on Metalink or Google.

In any case, either of these SQL environments have good error reporting. But what about Oracle Enterprise Manager?


Where am I supposed to go with this? Shall we go search on Metalink for "An error has occurred"? Does someone at Oracle imagine that this message is, in some mysterious way, useful? Maybe there's some logfile somewhere on the server that will tell me more... maybe... somewhere. Maybe. Somewhere.

Seriously, folks. OEM has been around for a long time now. It still throws lots of errors, often for no apparent fault of the DBA (I got this one simply by trying to access the "Administration" tab.) But I would forgive throwing lots of errors, if the errors were clear, and if they included information to facilitate finding out more. This, however, I cannot forgive, and it is all too typical of my experiences with OEM. (Well, a few years back, the errors tended to be screenfuls of Java error-stack vomit; now most of those seem to have been "cleaned up" into these poker-faced non-messages. That's progress? It's like Windows 3.1 all over again.)

So, if you see me rolling my eyes next time an Oracle rep shows a slide telling us how wonderful lovely OEM makes DBAs' jobs so much easier... this is why.

Hello Penguicon

I'm having a fantastic time at Penguicon so far...

except, of course, for being unable to connect my laptop to the projector for my talk. Aaaaaaaaaah!

If you were in my audience, 1) thank you for bearing with me, 2) I hope you managed to get something out of it, and 3) I'm so glad you came here!

The full tarball of talk material is available here. You can use it to look through the plain-text version of the talk (independently available here), or install vpython on your system and run the demos (I recommend running 'python solarSim4.py', then looking at the code), or actually install Bruce on your system so you can run 'socrates.py pyIntro.soc' and see the presentation the way you should have seen it at the con.

Thanks for being a great, involved - not to mention forgiving - audience!

Excel reports from TurboGears

Last week, I had the chance to design, build, and deploy a TurboGears application to outside customers for the first time ever. It was awesome! It was a simple survey form - well, simple in concept, but various questions were dependent in various ways on the answers to other questions, so I did have to get into the JavaScript, praise be to MochiKit.

The application owner wanted to see his survey results in (hmm, can you guess?) MS Excel. "No problem", I said, and then was surprised at how much trouble I had figuring out how to do that straight from TG. AFAIK, TurboGears doesn't have a handy way to output CSV. I could certainly produce a webpage with CSV data, and even use the default method in my controller to serve it with a .CSV extension ("http://myserver/report.csv"), but it still had webpage-type headers and thus Excel still didn't know what to do with it. And no, I wasn't about to tell him to cut-and-paste.

What I eventually hit on was to provide the results as a simple HTML table, then take advantage of the fact that Excel can open a webpage that contains (only) a table rather nicely. Then I gave him a batchfile that simply said

"c:\Program Files\Microsoft Office\Office11\excel.exe" http://myserver/report

... double-clicking it gets a live view of the data.

If he'd been cooler, the batchfile could just as well have said
"c:\Program Files\OpenOffice.org 2.2\program\scalc.exe" http://myserver/report
- I tried that, too, it works.

Actually, I didn't serve it under http://myserver/report, I served it under https://myserver/reportWithHideousGUIDblahblah4242rtfm22hike. That was my quick-and-dirty way to provide some basic security - the report URL is unguessable, so unless he shares the link, it should be safe. On the other hand, anybody who can sniff his request can extract the magic URL and use it. Can anybody comment on how much of a risk that is? There's no real sensitivity to the data in this case, but it would be nice to know if this the-URL-is-the-password scheme is a worthwhile shortcut or is terribly dumb.

Presenting at Penguicon

Well, two weeks from Friday, I'll be presenting on Python in Michigan... wearing a Starfleet uniform, if the vendor area opens in time.

Yes, I'll be at Penguicon. It's an open-source convention - it's an SF convention - it's a dessert topping! Troy, MI, April 20-22. I've never been there before, and I expect to enjoy it. The schedule included two higher-level Python talks, so it seemed to me like somebody should introduce Python so newbies could get the background they need to absorb the higher-level stuff. Wish me luck!

Introduction to Python, April 20 7:30 - 8:30 PM, Catherine Devlin

Django the Python framework for rapid web app development, April 21 9:30 - 10:30 AM, Dan Scott

Patterns of Data Access implemented in SQLAlchemy, April 21 6:00 - 7:00 PM, Mark Ramm

I also hope to make it to

PostgreSQL & Explaining Explain, April 20 10:00 - 11:00 PM, Aaron Thul

and to sit on the Constructed Languages panel (I'm an Esperantist)... and not to sleep, apparently.

My presentation will be more or less a repeat of the one I gave at Dayton Perl Mongers - it went well there, and the planetary theme should be even more appropriate for Penguicon. In fact, now that I've got a good talk to give, I'm starting to look around for any old place, any old audience in the area I can introduce Python to. Suggestions are welcome! centralOH@python.org is gaining membership pretty nicely, but we're still going to need more to turn it into a really thriving in-person group. Hopefully I can lure a bunch of new people into the joys of Python.

Feeling the love from Oracle

The Oracle Technology Network has rolled out two new features deeply gratifying to the PyOraGeek...

• The Scripting Language Technology Center is devoted to articles about Python and Ruby. Keep an eye on it - more articles are in the pipeline! (Including one by me... oops, now I mentioned it publicly, now I really have to get cracking on it.)
• They established an OTN Forum page just for Python.

Thank you, OTN! This attention is important for more than just the benefits to those of us already hooked on Python. Oracle's interest carries a lot of weight; when Oracle embraced Linux several years ago, it was a big step forward in establishing Linux's credibility in the corporate world. I think we can expect a boost, too.