Unicode error with TurboGears and Oracle

Thanks to kap_ravi's comment, I tracked down a surprisingly big and poorly-documented stumbling block for folks using TG with Oracle.

I've updated the original TG+Oracle blog post to reflect it, but the gist of it is that you're likely to get

SQLError: (NotSupportedError) Variable_TypeByValue(): unhandled data
type unicode

errors unless you include

sqlalchemy.convert_unicode = True

in your .cfg files.

Configuring Oracle for proper Unicode use may avoid the need for this; I'm not sure. My dreadfully mononational working environment has prevented me from developing any Unicode experience.

good errors vs. bad errors

One of the things I love most about PostgreSQL is its beautiful error messages. For instance, let's say I've inexplicably forgotten to add my home state to a table of states, thus violating a foreign key constraint when I try to insert a Minnesotan city. Here it is in Oracle:

SQL> insert into city values ('International Falls','MN');
insert into city values ('International Falls','MN')
*
ERROR at line 1:
ORA-02291: integrity constraint (MYSCHEMA.FK_CITY_STATE) violated - parent key not found

OK, that doesn't suck. In this case, it's pretty easy to figure out the problem, especially since I gave my constraint a meaningful name. However, there are cases where the foreign key is not so obvious - maybe it involves multiple columns - and I'm sometimes sent off to look up details on what FK_CITY_STATE involves. Since DESCRIBE won't work on a constraint, that means either an intricate query joining several V$ tables, or a graphical tool like Toad or SQL Developer.

Now I'll make the same mistake in psql.

mydb=# insert into city values (2,'International Falls','MN');
ERROR: insert or update on table "city" violates foreign key constraint "fk_city_state"
DETAIL: Key (state)=(MN) is not present in table "state".

Now that's clarity! It's one of the things I love about PostgreSQL. After all, the database has all this detail easily accessible to it; why shouldn't it deliver it to me along with the error message? It's even more important when an error like this is thrown from somewhere deep inside some code, where it may not be obvious to me which statement or which row is responsible for it - having the offending value displayed is incredibly convenient. It's considerate programming - no more, no less.

I'll say this for Oracle, though - Oracle's standardized ABC-12345 error codes are really handy for looking them up, for searches on Metalink or Google.

In any case, either of these SQL environments have good error reporting. But what about Oracle Enterprise Manager?


Where am I supposed to go with this? Shall we go search on Metalink for "An error has occurred"? Does someone at Oracle imagine that this message is, in some mysterious way, useful? Maybe there's some logfile somewhere on the server that will tell me more... maybe... somewhere. Maybe. Somewhere.

Seriously, folks. OEM has been around for a long time now. It still throws lots of errors, often for no apparent fault of the DBA (I got this one simply by trying to access the "Administration" tab.) But I would forgive throwing lots of errors, if the errors were clear, and if they included information to facilitate finding out more. This, however, I cannot forgive, and it is all too typical of my experiences with OEM. (Well, a few years back, the errors tended to be screenfuls of Java error-stack vomit; now most of those seem to have been "cleaned up" into these poker-faced non-messages. That's progress? It's like Windows 3.1 all over again.)

So, if you see me rolling my eyes next time an Oracle rep shows a slide telling us how wonderful lovely OEM makes DBAs' jobs so much easier... this is why.

Hello Penguicon

I'm having a fantastic time at Penguicon so far...

except, of course, for being unable to connect my laptop to the projector for my talk. Aaaaaaaaaah!

If you were in my audience, 1) thank you for bearing with me, 2) I hope you managed to get something out of it, and 3) I'm so glad you came here!

The full tarball of talk material is available here. You can use it to look through the plain-text version of the talk (independently available here), or install vpython on your system and run the demos (I recommend running 'python solarSim4.py', then looking at the code), or actually install Bruce on your system so you can run 'socrates.py pyIntro.soc' and see the presentation the way you should have seen it at the con.

Thanks for being a great, involved - not to mention forgiving - audience!

Excel reports from TurboGears

Last week, I had the chance to design, build, and deploy a TurboGears application to outside customers for the first time ever. It was awesome! It was a simple survey form - well, simple in concept, but various questions were dependent in various ways on the answers to other questions, so I did have to get into the JavaScript, praise be to MochiKit.

The application owner wanted to see his survey results in (hmm, can you guess?) MS Excel. "No problem", I said, and then was surprised at how much trouble I had figuring out how to do that straight from TG. AFAIK, TurboGears doesn't have a handy way to output CSV. I could certainly produce a webpage with CSV data, and even use the default method in my controller to serve it with a .CSV extension ("http://myserver/report.csv"), but it still had webpage-type headers and thus Excel still didn't know what to do with it. And no, I wasn't about to tell him to cut-and-paste.

What I eventually hit on was to provide the results as a simple HTML table, then take advantage of the fact that Excel can open a webpage that contains (only) a table rather nicely. Then I gave him a batchfile that simply said

"c:\Program Files\Microsoft Office\Office11\excel.exe" http://myserver/report

... double-clicking it gets a live view of the data.

If he'd been cooler, the batchfile could just as well have said
"c:\Program Files\OpenOffice.org 2.2\program\scalc.exe" http://myserver/report
- I tried that, too, it works.

Actually, I didn't serve it under http://myserver/report, I served it under https://myserver/reportWithHideousGUIDblahblah4242rtfm22hike. That was my quick-and-dirty way to provide some basic security - the report URL is unguessable, so unless he shares the link, it should be safe. On the other hand, anybody who can sniff his request can extract the magic URL and use it. Can anybody comment on how much of a risk that is? There's no real sensitivity to the data in this case, but it would be nice to know if this the-URL-is-the-password scheme is a worthwhile shortcut or is terribly dumb.

Presenting at Penguicon

Well, two weeks from Friday, I'll be presenting on Python in Michigan... wearing a Starfleet uniform, if the vendor area opens in time.

Yes, I'll be at Penguicon. It's an open-source convention - it's an SF convention - it's a dessert topping! Troy, MI, April 20-22. I've never been there before, and I expect to enjoy it. The schedule included two higher-level Python talks, so it seemed to me like somebody should introduce Python so newbies could get the background they need to absorb the higher-level stuff. Wish me luck!

Introduction to Python, April 20 7:30 - 8:30 PM, Catherine Devlin

Django the Python framework for rapid web app development, April 21 9:30 - 10:30 AM, Dan Scott

Patterns of Data Access implemented in SQLAlchemy, April 21 6:00 - 7:00 PM, Mark Ramm

I also hope to make it to

PostgreSQL & Explaining Explain, April 20 10:00 - 11:00 PM, Aaron Thul

and to sit on the Constructed Languages panel (I'm an Esperantist)... and not to sleep, apparently.

My presentation will be more or less a repeat of the one I gave at Dayton Perl Mongers - it went well there, and the planetary theme should be even more appropriate for Penguicon. In fact, now that I've got a good talk to give, I'm starting to look around for any old place, any old audience in the area I can introduce Python to. Suggestions are welcome! centralOH@python.org is gaining membership pretty nicely, but we're still going to need more to turn it into a really thriving in-person group. Hopefully I can lure a bunch of new people into the joys of Python.

Feeling the love from Oracle

The Oracle Technology Network has rolled out two new features deeply gratifying to the PyOraGeek...

• The Scripting Language Technology Center is devoted to articles about Python and Ruby. Keep an eye on it - more articles are in the pipeline! (Including one by me... oops, now I mentioned it publicly, now I really have to get cracking on it.)
• They established an OTN Forum page just for Python.

Thank you, OTN! This attention is important for more than just the benefits to those of us already hooked on Python. Oracle's interest carries a lot of weight; when Oracle embraced Linux several years ago, it was a big step forward in establishing Linux's credibility in the corporate world. I think we can expect a boost, too.

Intro to Python presentation

I've put the presentation for tonight's Intro to Python talk here.

It's not exactly the "slides" - it's a Bruce presentation. Bruce is great for live demos, especially for Python. Unfortunately, there's no "slide deck" that you can review statically - if you want to use it, you have to install Bruce and run it. The TAR I posted has a README with some basic instructions.

Since many conference organizers want a slide deck, one nice addition to Bruce would be a way to generate a slide deck. (No, I'm not volunteering... yet.) The other wart I ran into was the difficulty of changing the font sizes for the interpreter. Setting values in Socrates !CONFIG directives and config.py didn't work; I had to dig down into baseinterp.py and hard-code new font size values. That one, I probably could patch, if I get time.

Anyway, I did end up using Visual Python to do some fun solar system simulations. That should hold audience attention, I think...

Dayton, meet Python. Python, Dayton.

I'm giving an introduction to Python at the next Dayton Perl Mongers meeting:

March 14th 7pm to 9pm 2007 —
Introduction to Python at Dayton Perl Mongers—
at 119 Valley St. Dayton, Ohio, USA

Introductory talk on the Python programming language

This hCalendar event brought to you by the hCalendar Creator.

(Caution: the hCalendar creator didn't work for me with Opera or IE, only Firefox.)

If you're in the area, and still haven't learned Python (how have you put up with this blog?), please drop by!

I think that, to keep some visual interest up, I'll use Python's turtle module for some of my demonstrations. Not exactly snooty enterprise stuff, but turtle graphics are cute. Either that, or possibly vPython, which is certainly more impressive - but a bit more complex and less intuitive, possibly wrong for an introduction.

(I haven't forgotten my promise to post about PyCon. I'm waiting for presentation links to be published so I can refer to them.)

Geek Event Aggregator design notes

I finally created a central page for the Geek Event Aggregator - mostly it's design notes, but it's also a home less ephemeral than a bunch of blog posts:

http://catherine.devlin.googlepages.com/geekEventAggregator.html

Thanks to Google

... the Geek Event Aggregator is working again. Shortly after my post about the upload to gCal not working, I got an email from Ryan Boyd, of Google's Calendar team:

Someone brought the following post to my attention and I thought I'd follow up and see if I can be of any assistance... The problem that you are experiencing (500 error: Cannot access the calendar you requested) is due to the large size of (number of events in) the calendar.

Wow! Thanks to whoever passed my whining on to him... let me never speak ill of this social web stuff!

With that information, I was able to get it working again, so go ahead and check out that APL conference near your uncle in Kalamazoo.

Better yet, I was able to give my Lightning Talk on it, and it was very well received. The Lightning Talks are so popular that PyCon doesn't even schedule anything opposite them anymore, so basically the entire conference body of 593 people (!!!) was there; I've never spoken before so many in my life. I'm glad that didn't occur to me until afterward.

In the talk, I briefly outlined the architecture. At some point I'll post about it here. The basic principle is: HTML interpretation is hard, let's go datetutils.parser.parse()ing!

Yeah, PyCon! I just got back, and I've got so much to say; expect a flurry of blog posts as soon as I can get to it. I'm as happy as a clam. As several clams! As several unusually happy clams!

Geek Event Aggravator

I swear to you, the code that gathers data for the Geek Event Aggregator is running great.  It's harvesting over 5,000 events, with ever-improving completeness and accuracy.  

Unfortunately... the code that ships the harvested data up to Google Calendar - which worked perfectly at first - is now falling flat, with gCal returning 
"Error: Cannot access the calendar you requested." 
when I try to ship the new data up, or to clear out the old data - thus the redundant entries.  (No, it's not a simple password problem, I did check that...) 

I suspect that I triggered something at gCal that said, "Hey, there's way too much activity on this calendar, it must be some evil bot, lock it down."
  
I guess this is the problem with mashups.  It's great to springboard off the work of total strangers - but you're at the mercy of a black box.  Maybe I can buttonhole somebody from Google at PyCon next week (holy cow!  PyCon next week!) and ask for a contact somewhere in the bowels of the gCal team who might be willing to help me troubleshoot.  The sad thing is, this pretty much sinks my plan to do a Lightning Talk on it.

So anyway, my apologies if you've been trying to use the Aggregator and have found only stale and redundant information.  My box is overladen with fresh, hot event data; I just have to figure out a way to get it out to you.  I tried Swivel, but can't seem to get the data there, either.  And the old-fashioned solution - the app running in the Oracle Application Express sandbox - can only receive data by manual uploads, and then only in small chunks; I would have to do about fifty mouse clicks every time I wanted to refresh the Aggregator's data, and I'm sorry, but I'm not a clicker.

The ultimate solution, of course, is to build my own app on my own web server, or on a web host with permissive policies on installing software.  I don't have those resources right now, unfortunately.  (sigh)

[I apologize; Blogger has gone loony on the line breaks for this post, and I can't seem to fix it. Grumble. All my web resources are letting me down today.]

Presenting at Dayton-Cincinnati Code Camp

The following topics you submitted have been selected to be included in this year’s Code Camp:

· Data in Python, from ASCII to ZODB

Yay! Be in West Chester, OH (i.e., Northern Cincinnati, basically) on March 24 for a fun, free day of Code Camp. Most of the content is .NET, but they're very gracious about encouraging other content, too.

My talk is going to be a broad survey of ways you can get data of various sorts into and out of Python programs - not just RDBMS, but everything from the simplest to the most elaborate of data infrastructures. It's a lot to take on in an hour, but I think it'll be good for raising awareness of the breadth of possibilities - even for non-Python programmers (you poor things). I know I've learned quite a bit already just outlining the talk.

held for ransom by Vista

I attended an MSDN event here in Dayton last fall, where I was given a beta CD of Vista. I finally got around to putting it on my Windows partition last month. The install went pretty smoothly - it had an option to upgrade the XP partition, preserving the applications and data already present.

I didn't really play with it long enough to form a firm opinion on Vista, but the experiment ended rather badly. Vista popped up a window telling me that the trial period had expired, and it was time to pay up. I had to go to work at that point, so I shut down the machine, brought it into the office, and booted it up again. Then, the "trial period expired" window and a browser window - supplied so that I could buy a license at the MS site - were the only functionality that Vista would enable.
I couldn't go to my desktop, Windows Explorer, a command prompt, or anything that would enable me to access my hard drive. Files that I had created under my paid-for XP were now inaccessible to me; no way was provided to copy them off before reinstalling XP. My hard disk was, in essence, being held hostage; paying for a Vista licence was the ransom.

I tried to reinstall XP from its CD while preserving the data on the partition, but I got some frightening error messages in the process and aborted before risking the whole partition.

This could have been a serious crisis, but for two things -

• I do all my real work on the Linux partition, and didn't really have anything to preserve from the Windows side. I just wanted to take a look around to verify that.
• The Windows NTFS partition was perfectly accessible from Ubuntu, so I copied what I needed that way - didn't even need a thumb drive, just dragged icons onto the Ubuntu desktop. Hooray! I hadn't realized that read-only access to NTFS was so foolproof from Linux.

But what if I hadn't been a Linux freak? This could have been a real sackcloth-and-ashes event.

[If it hadn't been a dual-boot machine, I think I could have accomplished the same thing with an Ubuntu install CD, booting from it and copying the files to a USB drive.]

I don't think MS planned it this way. They just failed to plan a soft landing for their software's trial period ending. I'll remember that next time I play with a demo.

Microsoft Expression Web

Let me say two things in praise of Microsoft today:

1. They sponsored CodeMash.

2. I really, really like their new HTML editor, Expression Web. I like the split-screen, with the cursor that simultaneously shows each element in the WYSIWYG and HTML panes; I like the ease of generating in-line styles and moving them easily and cleanly to a separate CSS, I *love* the clean, uncluttered HTML it generates (*so* unlike FrontPage, ptui)

And, to my amazement, it's not Microsofty at all. It doesn't generate IIS-dependent code, you can preview the rendering in Firefox, etc. as easily as in IE. It's... respectful!

I'm officially asking my boss to buy us a copy. The one problem is... I prefer to work on Linux. Yes, I have a dual-boot machine, but going back and forth is a pain. So - can anybody recommend their favorite Linux-compatible HTML editors? Has anybody else tried Expression Web? Can you speak to how its features compare? Honestly, I've pretty much stuck with a text editor all this time, and I don't know if Expression Web's features are unique or routine.

centralOH@python.org

There's a new mailing list, centralOH@python.org, for Python users in and around Central Ohio. If this describes you, please join us!

http://mail.python.org/mailman/listinfo/centraloh


The Cleveland area already has a wonderful Python users' group - clepy. Those of us too far from Cleveland have been out of luck, though. We hope to use the new mailing list to gather together a bunch of Python users from the area, then begin to plan some live events. Eventually, perhaps we'll have a full-fledged, regularly meeting user group. First, though, we need to contact each other!

So, please, if you know anybody within shouting distance of central Ohio, let them know about centralOH@python.org !

Pardon, your template is showing.

Forgetting to apply data to your template is a pretty understandable mistake. But why are quantities like $baseCommittmentNumber queried live, while "6th movie" and "7th movie" are hard-coded? I hope they weren't just too lazy to generate ordinal forms of $baseCommittmentNumber+1 and +2 on the fly!

Incidentally, when all is said and done, the Columbia House deal works out to a minimum of ($0.50 X 5) + ($15 X 1) + ($20 X 3) + ($2 X 9) = $93.50 for 9 DVDs.

photos on business cards

Joe Wirtley (a CodeMash attendee) blogged an appeal for folks to include a photo of themselves on their blogs. I think it's a great idea - this weird bloggy world of electronic chatter is the most useful when we keep it linked to the world of flesh-and-blood. It's a shame to think that I've probably walked right past people at conferences whose writings I knew well.

I'm thinking of taking the appeal a step further. I love being able to recognize people and call them by name. Unfortunately, I'm a nerd, and I'm not very good at it. I struggle to find techniques to empower my socially deficient brain. Often, after I get someone's business card, I rush away to jot notes on the back - notes about what they're involved in and what we talked about, but also notes on their appearance. This is a desperate attempt to enable myself to recognize them the next time we meet, probably months or years in the future. The trouble is, a few phrases like "tall; curly hair" or "beard, glasses" really aren't enough.

I fantasized today about buying a cameraphone, snapping a photo of each person I meet, then finding a tag-based filing tool so that I could store the photos with the people's names and keywords about where I'm likely to see them again. Then, before arriving at an event, I could review the photos of people I'd be likely to see there; when I arrived, I'd call out to them by name and feel so cool.

Now, just imagine if somebody gave me a business card with their picture on it. Wow! I'd be so grateful! It would short-circuit this entire dilemma!

It's almost time for me to get new cards printed, and I could lead this incredibly useful trend by putting my own photo on them. Except... I'm nervous that it will come across funny, like I'm a wannabe model or something. Traditionally, geeks are supposed to imagine themselves as not really even having faces, as just being disembodied intelligences free-floating in the ether. The thing is, that's really not a useful myth. Talking to each other - face-to-face when we can - has very real value.

Maybe I should go ahead and do it anyway, and not worry about what people think. Maybe just being "the weirdo who puts her picture on her business cards" isn't so bad - for networking purposes, being weird isn't as bad as being forgotten. It's not "goofiness", it's "branding", right?

Hmm. Opinions? Would anybody join me on this?

CodeMash summary

Wow, CodeMash was FUN! It was chock-full of the kind of inquisitive, energetic people who make you glad to be a geek, both in the formal sessions and the informal sociable stuff.

I'll give just a few highlights, hopefully ones that haven't already been blogged to death by other CodeMash bloggers...

Even for someone who already uses TurboGears a bit, Kevin Dangoor's TG talk was great. Almost too high-energy to follow, but I got great ideas like using an in-memory SQLite back end for development testing (thus not worrying about repairing the database after testing);

It was nice to have Kevin's co-author Mark Ramm there, too. Questioning the authors beats leafing through a book any day... I just ordered my copy of the TG book for $26.99 from Nerdbooks. Amazon can slightly beat that price ($25.64), but at Nerdbooks I could toss in mildly outdated books on Unix and DB2 for $2-3 each.

David Stanek's talk on "Python Web Services" was actually more important, I thought, for the language-independent description of the relative merits of REST, XML-RPC, and SOAP-based services than it was for the Python-specific content. If you were off at some .NET talk instead, and you've always been a little fuzzy about how those work and relate to one another, go read his slide deck.

As a data geek, I really perked up at Scott Guthrie's keynote on LINQ. It's a new language component that basically extends the SQL idea (describe what data you want, let the computer worry about how to get it) beyond just RDBMS and into flat files, XML, web services... wherever the data may be squirreled away, it'll be a way to go get it with a single consistent syntax. Cool! I'll be eagerly looking forward to IronPython implementations of LINQ. So, Microsoft - thank you for LINQ, thank you for being one of the CodeMash sponsors, and I forgive you for Vista overwriting the GRUB boot loader on my dual-boot machine this morning (but only because I found a tutorial on how to repair it).

James Ward demonstrated ooooh-aaaah stuff you can do with Flex, which wasn't surprising. More importantly, I learned that the Flex/Flash world has really opened up ("opened" as in "open source"). When I looked into it a few years ago, it seemed that you needed a $mucho-K Flash server to even dip in. That's not the way I do things; I build a minor app or two - something that will really be used, not a development-only toy - before I know whether I really want to run with a technology. Now, most of the stack is open-source and free; only a couple optional (admittedly very convenient) components are commercially licensed. Adobe's new mostly-open-source business model for Flex permits my kind of experimentation; good for them!

Kevin Dahlhausen's talk on Test-Driven Development filled an embarrassing gap in my understanding of TDD - I'd heard of mock objects, but had never really figured out what role they play in TDD.

I took part in two Open Space discussions - one on women in IT, one on social networking for nerds - that were important enough that I think I'll give them their own blog posts in the next few days.

I heard Martin Fowler's name come up so many times that I finally screamed (internally), "Okay OKAY I'll read his stuff, already"!

In a week or so, I'll update this post to include hotlinks to the talk slide decks; most of them aren't posted yet.

Anyway... so much glorious fun. With CodeMash in January and PyCon in February, who dares call winter a blah time?

CodeMash

I'm here at CodeMash in Sandusky! I'd thought about blogging the whole conference real-time, but I think I'll only be doing a little of that... my laptop is fearfully heavy, and dragging it around all day would seriously inhibit my enjoyment of the Hallway Track (the stuff you learn chatting informally with people between sessions).

Anyway, so far, CodeMash is AWESOME. Very professionally run, and with excellent content. It began last night with a great expert panel discussion on languages - the sort of experts who seem to know a dozen different languages in more depth than you know your very favorite. It feels so good to have something like this here in Ohio.

A couple notes from the language panel...

• As a Pythonista, I was very gratified that the whole panel had a lot of knowledge of and respect for Python; it seemed to be mentioned more than any other single language!
• Tha panel took up the question of "which language should new computer science students learn?" The consensus was that sticking to any one language is a real mistake. Every language carries its own limitations and patterns of thought; learning new languages broadens the ways you have to think about a problem. Teaching students only one language risks hard-coding the assumptions and limitations of that language right into their brains. The rule of thumb "learn a new language every year" came up.
• As for specific languages, Python was the first reccommended... and assembly ended up also very highly reccommended.

Anyway, it's going to be a great conference. I'll try to update tonight. There will be a lot of tough decisions about which talks to go to, and I may decide against some of the Python talks - partly because I might be already beyond some of them in skill, and partially to honor the cross-language spirit of CodeMash.

PyCon and CodeMash

I'll be at PyCon, Feb. 23-25 in Dallas!


I'll also be at CodeMash in Sandusky!



I've mentioned both these conferences, but this is a good time to remind you of both of them, because the early-bird PyCon registration is almost upon us (Jan. 15) and CodeMash itself is almost upon us.

After two conferences in two months, my employer will probably manacle me to the server for the rest of the year, but it'll be worth it. I'd been coy up until now as to whether I'd actually make it to these events; my personal schedule has been wildly chaotic. But now I've got reservations in hand, so I'll see you there!