managing installation requirements by Python version

I always feel guilty using this blog as my LazyWeb, but it works really well, so here we go again...

I'm trying out Oracle Enterprise Linux 5 - basically a clone of RHEL - and trying to get sqlpython installed on it. OEL5 comes with Python 2.4; sqlpython 1.6.5.1 needs Python 2.5. I could take out the 2.5 dependency in sqlpython, but it uses pyparsing. Pyparsing dropped 2.4 compatibility as of 1.5.2.

[EDIT: It turns out that pyparsing 1.5.2 still works on Python 2.4. It emits a scary error message, during easy_install under Python 2.4 -

except ParseException as err:
^
SyntaxError: invalid syntax

but it installs successfully nonetheless. Thanks to Paul McGuire to pointing that out. So my job in this case is simply to release a 2.4-compatible sqlpython 1.6.5.2.]

I could change install_requires=['pyparsing>=1.5.1'] to install_requires=['pyparsing==1.5.1'] in setup.py, but that locks everybody into an obsolete pyparsing whether they need it or not.

I'd like to install_requires=(['pyparsing>=1.5.1'] if python.version >= '2.5' else ['pyparsing==1.5.1']), but that's absolutely imaginary syntax.

Obviously, I could just download a newer Python and install it on my machine, but I'm trying to make sqlpython usable for DBAs who don't have that kind of authority, or who fear to muck around with their environment that way.

I wonder what the smart people do in situations like this?

SpeakerRate

I've created a page at speakerrate.com for myself:

SpeakerRate profile

... and populated it with my two upcoming talks.

If you want to use SpeakerRate to give feedback to a speaker, you don't actually have to wait for them to create their own profile - you can enter it yourself.

I don't know if SpeakerRate will grow into the ultimate connecting-to-speakers webtool, but it might. I do know that they support microformats, which is a big plus in my book.

Hope to see some of you at Penguicon (this weekend) or IOUG Collaborate (next week)!

right to complaint

(prompted by discussion of porn use at GoGaRuCo - see here, here)

Quick thought: It's not that the community needs to ensure offensive content never happens, or that the community needs to find a single standard of what is appropriate.

The key is the right to complain safely. When complaints are predictably met with accusations of "overreacting", "political correctness", and "intolerance", the resulting message is: Be like us, be silent, or leave.

If you reject the criticism, then try something like, "I think you're wrong, but I accept your right to complain." Complaint is feedback, it's a legitimate part of a community's communication.

(Let me clarify that I've had mostly really good experiences in the software communities I participate in!)

the expanding reStructuredTextiverse

It seems I'm always coming across new uses for reStructuredText, the plaintext format that goes everywhere. (Really, more of a "set of plaintext conventions" than a format as such.) I'm beginning to imagine a talk reviewing them all for next Fall's Ohio LinuxFest, or maybe a magazine article.

The places you can go with reStructuredText - am I missing any? (I haven't checked these all for viability)

• HTML, XML, pseudo-XML, LaTeX, and S5 (PowerPoint-like presentations), with the conversion utilities in docutils
• rst2a - reStructuredText 2 Anything - docutils conversion utilities, online
• rst2pdf
• rest2web - webpages with templates, navigation structure, etc.
• rst2man - Unix man pages
• Sphinx - easy, multi-page sites for docs or other static content
• epydoc - Automated API documentation
• nabu - ReStructured Text to RDBMS
• rst2odt - Produce files for OpenOffice Writer.
• wiki pages
• Blog clients
• django-rstify - ReStructuredText in Django templates
• Editor support

Not everything has been done yet, however. Here are a couple projects yet undone - so far as I know. Comment with your own ideas... or take this as a challenge and implement something!

• rst2word - this would really be the holy grail, for communicating to the unwashed masses. (We in the know can use rst2odt and convert within OpenOffice, but rst2word would get my boss on board.)
• Fuse more templating engines to rst (perhaps not a good idea, violate the readability principle?)
• ReST lexer for Scintilla - this would allow ReST support in WingIDE, too.

[EDIT: Thanks Michael Foord for info about rest2web, rst2pdf.]

chocolate

We interrupt this blog with a special message from our sponsor.

My friend James has a new business, Two Bears Chocolates, handmaking organic chocolates. They are crazy-good... I've never been a food snob, but James's chocolates have spoiled me to the point where even "gourmet" mass-produced chocolates seem waxy and bland by comparison. They're a diet aid! I have to eat these chocolates so that I won't be tempted by lesser chocolates!

Anyway, they're expensive, but worth it, and a good gift (see: Mother's Day)... or a good way to maximize your yummy-per-calorie ratio.

He's particularly gung-ho to do custom orders. Go ahead, ask him to make a gooseberry-chocolate truffle in the shape of the state of Michigan. He'll love you for the challenge.

So, as James says, "If you love someone, send Two Bears chocolates. If not, bummer."

calm down

Yes, yes. Oracle is buying Sun, which owns MySQL and Java. No, this is not the end of MySQL. You're being silly.

Oracle is about as open-source friendly as a huge proprietary software company can be, and has been since before it was cool. Oracle adores Linux, and started pushing it vigorously since about, hm, 2002? Oracle has been Java-crazy since that time, too. Oracle's marketing strategy has long been against lock-in - it wants to plug easily into a thriving open-standards economy, not to enclose and lock a walled garden. It's also been very easygoing about licensing, eager to see casual, non-paying users gaining familiarity with its products, knowing that those are the seeds that later big-money sales will come from. It doesn't try to catch and squeeze little fish, it feeds them fish food and waits for them to grow into whales. In short, you over there, slapping MySQL on your Linux box for your brother's home business? Oracle doesn't want to shut you down. Oracle loves you, has always loved you, and wants your love and trust for when you get big.

In fact, if anything, I'm a little disappointed that Oracle's (superb) marketing power, name recognition, and corporate respect will all benefit MySQL and Java... which is all fine and good, except that I'd rather see that gust of wind behind PostgreSQL and Python. (OTN's PyCon sponsorship warmed my heart, to be sure, but I wish there was a way to make ORACLE + PYTHON stop-the-presses news all around techland.)

#amazonfail

If you haven't heard about #amazonfail, this article will catch you up quickly.

1. It's a good reminder that giving market dominance to one company in a crucial role in steering our culture is probably unwise. Let's not have a market where a single company can relegate books to obscurity, intentionally or not. Patronize a variety of booksellers.

2. A proper apology would look something like, "Mr. Doofus Middle Manager didn't think through what he was doing, and company management failed to supervise it properly. We feel humiliated and commit to new efforts to keep book culture diverse and uncensored." Amazon's lame, mealy-mouthed "glitch" non-apology suggests that it has fallen prey to Big Corporatosis. Unless, of course, Amazon really has discovered a homophobic computer glitch, in which case this is huge news for artificial intelligence researchers.

3. This is not the only case where the label "adult content" has had a strangling effect. Actions taken under the justification of "adult content" should never be blandly accepted, but should be carefully examined for accuracy, necessity, and bias.

[ EDIT: A final complaint: Labelling a bad policy, sloppily implemented and poorly supervised, as a "glitch", is blaming the company's technologists for a mistake of its management. It says that Amazon's management does not trust, understand, or respect its technologists. In a technology company, this is a strong signal of decline.)

Penguicon 7.0

If you're anywhere near Michigan, you need to consider Penguicon. It's an open-source software conference! It's a science-fiction con! It's two great tastes that really do taste great together. There's always a great deal of excellent technical content, and the SF people lend a really healthy sense of relaxation and creativity to the whole thing. Where else can you learn CSS and belly dancing in one weekend?

I'm giving a talk at Penguicon this year: "sqlpython: SQL is fun again". It's sort of a preview of my upcoming SQL*Plus Alternatives talk at IOUG Collaborate... but without the stuff about Oracle-only tools, and with more focus on sqlpython's rapidly developing cross-RDBMS powers, and a healthy plug to pull more people into the project.

In fact, I'm leaving directly from Penguicon to Collaborate. Yikes! It'll be a fun week.

PyOhio: Call for Proposals

The PyOhio Call for Proposals has been issued!

PyOhio 2009 takes place July 25-26, 2009 at the Ohio State University in Columbus, Ohio. Much like a mini-PyCon, it includes scheduled talks, tutorials, Lightning Talks, Open Spaces, and room for your own unique ideas. If you can make it to Ohio this summer, please consider participating.



PyOhio 2009, the second annual Python programming mini-conference for Ohio and surrounding areas, will take place Saturday-Sunday, July 25-26, 2009 at the Ohio State University in Columbus, Ohio. A variety of activities are planned, including tutorials, scheduled talks, Lightning Talks, and Open Spaces.

PyOhio invites all interested people to submit proposals for scheduled talks and tutorials. PyOhio will accept abstracts on any topics of interest to Python programmers.

Standard presentations are expected to last 40 minutes with a 10 minute question-and-answer period. Other talk formats will also be considered, however; please indicate your preferred format in your proposal. Hands-on tutorial sessions are also welcomed. Tutorial instructors should indicate the expected length

PyOhio is especially interested in hosting a Beginners' Track for those new to Python or new to programming in general. If your proposal would be suitable for inclusion in the Beginners' Track, please indicate so. Organizers will work with speakers and instructors in the Beginners' Track to help them coordinate their talks/tutorials into a smooth, coherent learning curve for new Python users.

All proposals should include abstracts no longer than 500 words in length. Abstracts must include the title, summary of the presentation, the expertise level targeted, and a brief description of the area of Python programming it relates to.

All proposals should be emailed to cfp@pyohio.org for review. Please submit proposals by May 15, 2009. Accepted speakers will be notified by June 1.

You can read more about the conference at http://pyohio.org

If you have questions about proposals, please email cfp@pyohio.org. You can also contact the PyOhio organizers at pyohio-organizers@python.org.

geekspeakr.com

geekspeakr.com: connecting tech women speakers with event organizers

Many organisers of technical conferences, meetups, and dinners want to have more gender-balance in their lineups, but they don't know where to find technical women speakers.

Enter geekspeakr.com, a simple directory and connections system to help technical women speakers and event organisers to find each other.

I'm really glad to see this. I'm even more glad to see, browsing through speakers, that they really are technical women. I've... um, there's no good way to say this... seen other "tech women" groups that quickly became dominated by women networking for their multilevel marketing careers. It's pretty understandable, since they have a much more obvious need to network than us geeks - but, you know, it's really not the purpose.

Anyway. geekspeakr's are the real deal. w00t! Need more Python and Oracle speakers there, though.

Note to self: just as soon as the PyOhio CFP is out (very soon), do not neglect to spam Pythonistas on geekspeakr! At least, the ones vaguely near Ohio.

Five minutes at PyCon change everything

I gave a five-minute Lightning Talk on sqlpython on Saturday. I hoped it would pique the interest of some people who sometimes use Oracle, and give them a neat example of yet another cool thing being done with Python. It certainly did that, and I got lots of gratifying feedback.

I knew people would ask when it would be available for non-Oracle databases, so I said, tongue-in-cheek, that this was my distant-future ambition for "sqlpython 3000". What I didn't expect was that several of the people buttonholing me over the next two days would ask to collaborate to get multi-RDBMS support in place. Help? Uh, yeah... I guess help would help... I honestly hadn't even been thinking about that...

Brian Dorsey in particular wanted to see the code face-to-face with me, so I put a card on the Open Space board, just in case anybody else wanted to show up, and twittered about it one bare hour in advance.

Nine people came, all of them eager to get going on writing code, bringing great ideas to get started. All this for a project that was basically personal 36 hours before.

Noooo, now other people are going to be exposed to my squiggly code! Now I know what embarrassment-driven development really is.

If I'd had $1 million of startup funding to hire a staff to work on sqlpython, I couldn't have gotten a team that large or that talented. I figure that gives me better than a 1000-to-1 return on my PyCon investment. :)

So anyway, I'm setting up a mailing list for cooperation on sqlpython, and it looks like the far-future dream of multi-RDBMS sqlpython has suddenly become imminent. Stay tuned!

sqlpython lightning talk follow-up

This morning's lightning talk on sqlpython was an example of what's so great about PyCon - it instantly really good suggestions about how to go onward with sqlpython development, and offers of collaboration. Awesome!

Except that I forgot to show the instant graphs using \b / \l terminators. Rats! That's the most eye-catching part!

I wasn't prepared for the common question, though: "Where's the repository?" Well, it's here:
https://www.assembla.com/wiki/show/sqlpython It is crazy-unstable, and if you're actually trying to use it, use the PyPI version instead.

I mean to get a link to that into the docs as soon as possible, but I don't have Sphinx configured right on this machine, so that may have to wait.

Quick review of the lightning talk:

* Unix-like powers: cat, ls, grep, >, |
* Python interactive session; access to resultsets (`r`) and bind variables (`binds`)
* Special output formats with alternate terminators (see `help terminators`)
* The magic that makes sqlpython work:
- cmd
- cmd2
- pyparsing
- code (for the embedded Python interpreter)
- cx_Oracle

For further reference, see the sqlpython docs, particularly the comparative review of sqlpython vs. SQL*Plus vs. gqlplus vs. Senora vs. YASQL.

Ada Lovelace Day: Tech Women I Admire

I will publish a blog post on Tuesday 24th March about a woman in technology whom I admire but only if 1,000 other people will do the same.

I can't stop at one; sorry if that breaks the rules.

• As a frequent speaker at conventions and as co-author of the Python Cookbook, Anna Ravenscroft has a great talent for helping people understand problems, even the brain-bending ones.
• Dianne Marsh helps run a company that vigorously fosters good, innovative programming, both among their own developers and throughout the entire region. Her energy and dedication have been crucial to CodeMash, one of the most innovative things to happen in our area, and to several user groups in Michigan.
• Sarah Dutkiewicz, aka the Coding Geekette, is a programmer from northeast Ohio who's done great things in pulling together the geek community and teaching them new technologies (like IronPython on Mono) - often the sort of stuff that requires venturing into dragon-infested realms where the documentation is scattered or nonexistent.
• In her long-time leadership of the OOUG, Coreen Walker
has helped make it one of the best Oracle groups around.

PyCon begins...

... well, the tutorials and summits begin tomorrow. But some of the organizers are already in place, getting things set up. I wish I were there! I won't arrive until Thursday afternoon - I had to sacrifice all but the core conference days so that I could make the separate trip to IOUG Collaborate. For me, it feels like arriving at a family Christmas halfway through the gift unwrapping.

Have you seen the list of PyCon sponsors this year? It's amazing! It's practically a Who's Who... if your company isn't sponsoring PyCon, your corporate headquarters is probably roofed with thatch. :) It's a tribute to Van Lindberg, PyCon's sponsorship coordinator, but even more to the growing corporate recognition that Python has become a pillar of the IT world.

Where to host docs?

I'm working on some preliminary sqlpython docs using Sphinx, which is really really nice. No guarantees that they will remain there, though - they need a home without a raw IP address for a URL!

I'm trying to figure out a permanent home for the docs. Here are the possibilities I know of.

• Host the files on my own server. Buy a domain name (but not like last time). This is obviously the best option... unless I ever decide to quit paying for the domain, or the server, or forget to, or get hit by a bus. So really, I'd prefer something that wasn't so dependent on, well, me.
• Sourceforge, Assembla, Google Code, and probably everybody else who hosts projects also let you create and host wiki-style documentation. As far as I know, though, there's no way to upload Sphinx documentation into any of them; the wiki form and the Sphinx form are not compatible.
• Google Pages gives you free webpages with a reasonable URL - but no folders. Sphinx depends on a folder structure - it creates folders "html", "doctrees", "_sources", "_static", and maybe more once I get serious.
• Creating a Google App Engine to host the docs - now this is an intriguing possibility. I'm going to check it out. Still, it would be nice if posting the docs for a technical project were not, itself, a technical project.

Any possibilities I'm missing? A convenient way to host Sphinx docs, freely available, would be a really nice service to FOSS developers. I wonder if the Google App Engine approach could be extended to provide that... hmm hmm hmm... just what I needed, one more project...

sqlpython 1.6.1 puts the "python" in "sqlpython"

I've been asked several times, "Why is it called sqlpython?"

My answer used to be, "Ask Luca [Canali]; he wrote it."

Not anymore. Now, witness the power of this fully armed and operational hybrid SQL/Python working environment.

[EDIT: As of sqlpython 1.6.2, you use quit(), exit(), or Ctrl-D/Ctrl-Z to return from
interactive Python mode. See docs.]

0:testschema@eqtest> select title, author from play;

TITLE           AUTHOR
--------------- -----------
Timon of Athens Shakespeare
Twelfth Night   Shakespeare
The Tempest     Shakespeare
Agamemnon       Aeschylus

4 rows selected.

0:testschema@eqtest> py import urllib
0:testschema@eqtest> py current_season = urllib.urlopen('http://cincyshakes.com/').read()
0:testschema@eqtest> py
Now accepting python commands; end with `end py`
>>> r[-1]
[('Timon of Athens', 'Shakespeare'), ('Twelfth Night', 'Shakespeare'), ('The Tempest', 'Shakespeare'), ('Agamemnon', 'Aeschylus')]
>>> for row in r[-1]:
...     print '%s by %s' % (row.title, row.author)
Timon of Athens by Shakespeare
Twelfth Night by Shakespeare
The Tempest by Shakespeare
Agamemnon by Aeschylus
>>> [row.title for row in r[-1] if row.title in current_season]
['Timon of Athens', 'Twelfth Night']
>>> binds['nowplaying'] = [row.title for row in r[-1] if row.title in current_season][0]
>>> end py
0:testschema@eqtest> print
:nowplaying = Timon of Athens
0:testschema@eqtest> select title, author from play where title = :nowplaying;

TITLE           AUTHOR
--------------- -----------
Timon of Athens Shakespeare

1 row selected.

A history of result sets from each query is exposed to the python session as the list `r`; the most recent result set is `r[-1]`. Bind variables are exposed as the dictionary `binds`. All variables are retained each time the python environment is entered (whether interactively, or with one-line `py` statements).

Resultsets in `r` are read-only, but `binds` can be written as well as read, and will be working bind variables in the SQL environment.

Oh, the possibilities...

Senora 1.0.1, and mutual FOSS goading

Martin Drautzburg just published version 1.0.1 of Senora, a command-line client for Oracle that makes a very attractive alternative to Oracle's SQL*Plus.

The timing - seven weeks before I present a review of Senora, sqlpython, and others at IOUG Collaborate - is not a coincidence. I asked Martin to review my draft paper submission for the conference. He graciously did, and supplied several crucial corrections and additions regarding Senora.

He also mentioned that he'd been continuing Senora development for in-house use, but that it had been years since he'd released the updates. My upcoming review spurred him to do the release.

And what a release it is - it's full of awesome! Mighty new flag options! Multiple sessions! Automatic generation of Senora commands from SQL scripts!

It sent me to scheming about how I can copy this fresh goodness to sqlpython.

Plus, of course, the paper reminded me painfully of some of sqlpython's shortcomings. I'm starting to use the sqlpython trac seriously now, as a token to myself of my goodwill. This will probably lead to a spurt of Embarrassment-Driven Development before the conference.

FOSS people know what the ancient Romans knew: money is a feeble motivator compared to glory. (Half the time, people only want money so they can buy things that will make them feel glorious...)

using TurboGears 2 model outside TurboGears

Until/unless this ticket gets incorporated into the TurboGears 2 docs, I (for one) need a reminder about how I can make use of a sqlalchemy model, set up from a TurboGears 2 instance, for projects that don't actually start or use TurboGears. (I want a single, canonical sqlalchemy model for my database, for its web-based and non-web-based applications alike.)

Including this function in my model/__init__.py does the trick.

import paste.deploy, os.path
def externally_usable_session(configfile = 'development.ini'):
    tg_home_directory = '/path/to/tg2instancehomedirectory'
    conf_dict = paste.deploy.appconfig('config:%s' % os.path.join(tg_home_directory, configfile))
    engine = create_engine(conf_dict['sqlalchemy.url'])
    init_model(engine)
    return DBSession()

sqlpython 1.6.0 with Wild SQL

I just released sqlpython 1.6.0.

SELECTing a limited, but large, set of columns from a table is a real pain. What if you could use wildcards in the column list of the SELECT statement itself? Wouldn't that be wild?

OK, then, let's SET WILD ON.

jrrt@orcl> cat party

NAME    STR INT WIS DEX CON CHA
------- --- --- --- --- --- ---
Frodo     8  14  16  15  14  16
Gimli    17  12  10  11  17  11
Legolas  13  15  14  18  15  17
Sam      11   9  14  11  16  13

4 rows selected.

jrrt@orcl> set wild on
wildsql - was: False
now: True
jrrt@orcl> select *i* from party;

INT WIS
--- ---
 14  16
 12  10
 15  14
  9  14

4 rows selected.

You can also call columns out by number...

jrrt@orcl> select #1, #5 from party;

NAME    DEX
------- ---
Frodo    15
Gimli    11
Legolas  18
Sam      11

4 rows selected.

... or use ! as NOT.

jrrt@orcl> select !str from party;

NAME    INT WIS DEX CON CHA
------- --- --- --- --- ---
Frodo    14  16  15  14  16
Gimli    12  10  11  17  11
Legolas  15  14  18  15  17
Sam       9  14  11  16  13

4 rows selected.

... and you can mix it all together.

jrrt@orcl> select n*, !#3, !c* from party;

NAME    STR WIS DEX
------- --- --- ---
Frodo     8  16  15
Gimli    17  10  11
Legolas  13  14  18
Sam      11  14  11

4 rows selected.

A bunch of limitations:

• Wild SQL is not yet a widely-accepted industry standard. Actually, I just made it up. If ANSI hears about it, they will hunt me down with dogs. That's why you need to SET WILDSQL ON to turn it on.
• Wild SQL only works on the column list - the part between the SELECT and the FROM. It doesn't work in the WHERE clause, or in subqueries.
• Wild SQL only works in SELECT statements. What, you were thinking about using it in DML? Are you crazy?
• Do I really have to say that it's very alpha? Well, it is. Expect a trickle of bugfixes over the next few months.

sqlpython 1.5.3, with version control

There are several barriers to use of version control tools for Oracle DDL (Data Definition Language: table structures, etc.)

• Tools from Oracle, Quest, etc. may have VC capabilities, but you never know whether you'll have the tool available in any particular situation, and you never know when the workings of the tools will change beyond recognition. (I'm looking at you, Oracle.)
• The tools are separate from the mainstream of version control in software development - you're learning quirky specialty tools instead of widely-known industry standards.
• They lack some of the modern capabilities of distributed version control.

We're better off using standard software development VC.

But, of course, those tools are meant for text files, so there needs to be a handy way to get this stuff from DDL inside the database to version-controlled text files.

It's easy with sqlpython 1.5.3

The new sqlpython commands svn, bzr, and hg all do the following:

1. Create or update a directory tree, beginning at your current working directory, containing text files with the DDL for all the objects in your schema
2. Put these files under version control and commit

If you don't want all the DDL, you can limit the dump using the same arguments the ls command takes.

For example,

testschema@orcl> !pwd
/home/catherine/oracle_vc
testschema@orcl> ls

NAME
--------------
INDEX/XPK_PLAY
TABLE/PLAY

2 rows selected.

testschema@orcl> bzr
added testschema
added testschema/index
added testschema/index/xpk_play.sql
added testschema/table
added testschema/table/play.sql
Committing to: /home/catherine/oracle_vc/
added testschema
added testschema/index
added testschema/table
added testschema/index/xpk_play.sql
added testschema/table/play.sql
Committed revision 1.
testschema@orcl> alter table play add (opening_night DATE);

Executed

testschema@orcl> bzr
bzr: ERROR: Already a branch: ".".
Committing to: /home/catherine/oracle_vc/
modified testschema/table/play.sql
Committed revision 2.

testschema@orcl> alter table play add (performances NUMBER(5,0));

Executed

testschema@orcl> create index xif1_play on play (opening_night);

Executed

testschema@orcl> bzr index/
bzr: ERROR: Already a branch: ".".
added testschema/index/xif1_play.sql
Committing to: /home/catherine/oracle_vc/
added testschema/index/xif1_play.sql
Committed revision 3.


"Writing about Python" at PyCon

PyCon early-bird registration deadline is TOMORROW (Saturday)! No time to lose - go register!

Most Open Spaces are not scheduled until the very day they are held, and that's good. Some are done with some advance planning, though, and that's good too. Doug Hellman is already preparing a "Writing about Python" open-space session at PyCon; I'm eager to take part.

If you're not familiar with conferences like PyCon, you may not realize that the formal schedule, goodie-packed though it is, is not the whole story by a long shot. People use the Open Spaces for a huge variety of things; last year, for instance, I got a lot out of a group organizers' freeform open space discussion, and even more out of the now-famous "Teach Me Twisted" session. If you think the published schedule leaves you in fits of indecision, wait until you see the Open Space board. There's a sort of joyful despair in seeing that you would need three months of PyCon to take part in all the PyCon you want.

Open Source wrecked the economy

Well, not really, but it did get your attention.

This is a really interesting New York Times article about VaR ("Value at Risk"), a mathematical tool. Over-reliance on this tool was arguably the reason the world's financial experts and geniuses spent the last several years acting like morons. This paragraph jumped out at me.

What caused VaR to catapult above the risk systems being developed by JPMorgan competitors was what the firm did next: it gave VaR away. In 1993, Guldimann made risk the theme of the firm’s annual client conference. Many of the clients were so impressed with the JPMorgan approach that they asked if they could purchase the underlying system. JPMorgan decided it didn’t want to get into that business, but proceeded instead to form a small group, RiskMetrics, that would teach the concept to anyone who wanted to learn it, while also posting it on the Internet so that other risk experts could make suggestions to improve it. As Guldimann wrote years later, “Many wondered what the bank was trying to accomplish by giving away ‘proprietary’ methodologies and lots of data, but not selling any products or services.” He continued, “It popularized a methodology and made it a market standard, and it enhanced the image of JPMorgan.”

I thought this was a fascinating summary of open-source advantages: prestige, benefiting from community-contributed enhancements, creating a standard, all without the effort and expense of attempting to market a proprietary technology.

OK, so it's not really a feather in Open Source's cap, seeing as the economy did end up wrecking over it 'n all, but it does demonstrate the power of openness to popularize a tool. Open-sourcing a tool can make it very popular, but it's up to us not to make a tool into a god.

(Of course, when it's applied outside the software world, we really ought to point out that the principle never came from software at all. It's simply openness, the same ancient innovation that happened in the move from alchemists guarding their secrets to scientists publishing their work.)

Recent developments in the doom of humankind

xkcd foresees it

skynet 0.1

Multi-ton, clawed semi-autonomous Python-powered robots

PyCon talks

You want an awesome badge like this for your blog, right? You can choose from a variety of badges at the publicizing PyCon site. PyCon depends on the user community (that's you) to spread the word. Bringing in a bigger and broader community is how PyCon keeps getting more exciting.

Anyway, even if you are certain you can't attend PyCon, it's worth it to browse the list of accepted PyCon talks - it's a great window into some of the exciting things going on in Python-land. It's already helped me learn about several useful packages and ideas.

Hope to see you in Chicago!

blogging about business, for once

Today my employer benefited through me, but almost despite me.

We're facing an exhausting slog through yet another incarnation of the Air Force's process for getting permission to continue to operate an IT system. My (Air Force) boss had heard somehow that Mark, one of my fellow employees at Intellitech, was shepherding some projects through this process, and asked if he'd be willing to give some advice. Mark came and spent a couple hours giving some desperately needed information, despite being warned that my boss has no prospects of funding to take on additional contracting help.

Everyone there was enormously grateful, because living, breathing survivors of the process are almost unheard of, and the available training material is of very little use. What's more, my boss had made a last-minute impromptu invitation to a friend whose project is also facing the process; she was just as happy to attend, and just might have the funding for some help. And everybody there is going to spread the word that they now know a source of much-needed information on this process.

The problem? I was passive. I hadn't thought to suggest to my boss to tap Mark's experience; good thing he knew about it and thought to ask. I hadn't thought to suggest inviting others to the meeting, either; again, credit to my boss. I really need to be more alert to this sort of thing.

The other problem? If Intellitech does end up with more work thanks to today, it's the kind of hellishly bureaucratic work that makes you want to chew your leg off to escape. Great for the company bottom line, but a blasphemous waste of a living human soul.

complexity

Fire up a program like Eclipse or Visual Studio and start a new project. Blam! It creates a dense forest of code files, text files, configuration files, XML files, directories, subdirectories, assistant directories, deputy directories, acting deputy subdirectories, and special liason to the ad-hoc subcommittee directories.

Java and C# people seem to think this is great. "Look at all this work the tool does for you!"

I disagree. "What IS all this stuff?" I wail. "What's it for? What's it doing? My program is already hopelessly complex and I haven't even started writing it yet! Guido, take me away!" A tutorial may tell me to start on one file and ignore everything else, but that's completely unsatisfying psychologically. I feel like I'm learning nothing because I'm at the mercy of so much auto-generated stuff I can't even hope to understand.

This, I think, is my biggest barrier to learning the "enterprisey" languages.

I admit, starting a project in TurboGears or Django does something kind of similar, and I tolerate it there. I think that's because I know Python well and can handle a limited amount of temporary mystery. It's when I'm trying to dip into a brand-new language and a brand-new environment that I want a small, digestible bite of mystery to get my mind around.

[ADDENDUM]

I'm beginning to think that what might get me over this hurdle is a C# tutorial that specifically avoids Visual Studio and similar IDEs... that gives projects and examples in old-fashioned code files in a text editor - and as few files as possible - even if that seems crazy to VS junkies, if it means the examples are a lot more painstaking and a lot less impressive. But that's what I need - I need to feel like the code I'm writing really encompasses the problem, isn't just a little flourish atop a vast understructure that I didn't write and don't understand. Then, after a while of that, I can start dipping into the IDEs and perhaps appreciate the code generation, rather than dreading it.

Any pointers to a C# tutorial along those lines would be most welcome.

Reinteract

Shortly after semi-finishing pyparsing_helper, I realized that it's really not needed, because there's a tool called Reinteract that can fill the same role, plus much more.

Reinteract is a graphical Python session that lets you tinker with your code at any point and get an immediate recalculation. That's what pyparsing_helper does, too, but pyparsing_helper is specialized for pyparsing use, whereas reinteract is suitable for any Python. Here's reinteract applied to a pyparsing example.

Reinteract is also more sophisticated and versatile, so I'm afraid pyparsing_helper's short time in the sun has been eclipsed. I'm happy to have found this new tool, though!

New Year's Resolution (blogging while angry)

From now on, before spending any significant money on anything, I will test their customer service phone number. If I cannot get to a human being who is reasonably helpful, I will not become a customer.

Understand, I'm all for saving money by automating customer service where possible. But many companies (AT&T Wireless, for example) now make ad hoc, human-interaction support literally unavailable. A demonstrated intent to avoid customer contact is a sign of a bad business partner, and I will not support such bad business practices anymore. (steam, steam, steam)

coworking for the rest of us

Ah, so many ways to learn from each other.

• Formal conferences and meetings. "Eyes-front" presentations.


• Getting together with the geeks for chat.


• Unconferences and open spaces. Lots of potential for multi-directional learning, yet shaped around specific topics.


• Sprinting: gathering to code together with people outside your usual circle on specific projects. GiveCamps.

And yet, there's a new one: hanging out and geeking out with other geeks. Programming with, or around, people who aren't your usual co-workers. Like sprinting, sort of, but on no predefined topic. Questions, advice, and ideas bubbling around unpredictably while you code.

That's basically what the CodeJam at CodeMash was, and I loved it. Even if you already work every day with a wealth of fiercely creative, inquisitive, knowledgable co-workers who are delighted to pitch ideas into your project - and not all of us do - it's still a benefit to mix it up with some new minds.

That happened just a week ago, yet suddenly, I'm seeing it everywhere. In Cincinnati, they're doing an evening called "bitslingers". In Ann Arbor, they're doing a daylong Code Retreat.

Solo from-home workers developed this a while ago; they call it "coworking". Now we cubicle monkeys are getting some chances at it, too. I'd love to see this carried to its logical extreme - a custom of getting out to work in new places with new people, say, one day a week. I think the payoff would be enormous.

pyparsing_helper

I'm back from CodeMash!

I got a lot of great ideas, and I especially liked Wednesday's CodeJam, where I started building pyparsing_helper. Hacking on code while surrounded by sharp people to absorb energy and advice from is a great way to get things done! I finished it on the way home, so pyparsing_helper is officially a child of CodeMash!


This is what I had in mind when imagining "Kodos for pyparsing" last week.

One big bad fly in my ointment: pasting into pyparsing_helper isn't working. I don't know why. That's top priority for fixing for v0.1.1; I wonder if I'll have to switch out of Tk to get it to work.

easy_install pyparsing_helper

[EDIT:] Paste does work under Windows, and it looks like it's an inherent Tkinter flaw... what's up with that? Tkinter's only been around for, oh, 15 years or so - and this core function is still unfixed? Maybe that's why all the cool kids quit using it years ago.

[EDIT #2:] OK, paste did work in *nix, but only in its Shift-Insert variety. pyparsing_helper 0.1.1 (available now) makes the more familiar Ctrl-V pasting available, too.

new baby

Items shipped on December 30, 2008:
Delivery estimate: January 6, 2009
1 package via USPS

1 of: One Laptop per Child XO Laptop (Give a Laptop, Get a Laptop) Sold by: OLPC Foundation (seller profile)

Before or after I leave for CodeMash? I MUST KNOW!

Well, even if the new baby doesn't arrive on time, I'm very much looking forward to CodeMash. It's very hard to decide what to do, though, particularly with the Precompiler day. I like the CodeJam idea best, but I'll have to see if the particulars of the project appeal to me - and I'd want to do it in a platform I don't yet know, and I'm not sure whether I could be effective that way. And a whole day of absorbing Ruby from Jim Weirich would certainly be an entertaining alternative. But maybe I should do the .NET tutorial... knowing some .NET would really help me get going in IronPython. Decisions, decisions... what a nice dilemma!

domain name woes

I bought the domain name for nerdstogether.org from 1and1.com, but I'm now realizing that domain name ownership isn't as straightforward as I thought.

Apparently, the Domain Name Servers of the world have not been alerted that nerdstogether.org now points to 66.35.48.8, the IP address where I'm hosting nerdstogether. Instead, they believe that it points to a server at 1and1, which then is responsible for passing the request along to my machine.

1and1 offers two choices for how this can be done.

HTTP forwarding

This simply sends the user's browser off to 66.35.48.8. There are two drawbacks:

• The browser's URL bar shows 66.35.48.8; "nerdstogether.org" would be much prettier.
• 1and1 truncates the URL before sending it on; this makes RESTful access impossible, since http://nerdstogether.org/dayton goes to 66.35.48.8/ instead of 66.35.48.8/dayton

Frame redirect

In this case, 1and1 hosts a webpage which simply contains a single frame; the contents of this frame are requested from my server. This looks better, since the user's URL bar continues to display "nerdstogether.org". Furthermore, 1and1 does attach the remainder of the requested URL, so RESTful access remains possible.

The problem? I'd like to provide not just visible HTML access from this domain, but a JSON web service as well. JSON data should be returned raw, as "content-type: text/json", not as an HTML frame embedded in an HTML webpage. No web service consumer can digest that! So I need to distribute a separate URL, with my raw IP address, to web service consumers.

Virtual servers

Finally, both HTTP forwarding and frame redirect also make virtual servers impossible. I'd like to serve multiple unrelated websites from my machine, which I can do by configuring my webserver to react differently based on whether the URL requested was for nerdstogether.org or for a different domain name. Unfortunately, either of 1and1's options remove this information from the request before it is passed along to my machine; my machine only sees a request with its IP address. The fact that nerdstogether.org was the domain name requested is not passed along.

It's still possible to host multiple sites, by specifying a separate directory within 66.35.48.8 for each separate domain. Virtual servers would enable a cleaner separation between sites, though, with no possibility of navigating back up the directory hierarchy.

As far as I can tell, there is no ideal solution for me, aside from buying my domain name from a different provider, one who would actually propagate my domain name ownership out through the worldwide DNS network. I don't even know what the term would be for that kind of "full ownership" of a domain name.

This is my first trip through this wilderness, and I'd be delighted if I've missed some better solution that someone wise can point out to me.

nerdstogether.org

Happy New Year! And a happy new name for The Application Formerly Known As Geek Event Aggregator (or geekeventfinder). Say hello to

nerdstogether.org

It's your one-stop resource for finding out, "How can I get together with some nerds in <insert place here>?"

nerdstogether.org comes with an exciting new feature: it actually works now. No, really. Go try it.

I achieved this by using some cutting-edge technology called "CGI scripting" and a "database". The problem of gathering the data has been pretty much solved for literally years, but I kept trying to find Web 2.0-ish solutions for getting the data to you: Oracle Application Express, Google Calendar, Google App Engine. All of them seemed great at first, but eventually frustrated me to tears.

The one nod to Web 2.0 is the very helpful use of Yahoo and Google web services for interpreting location information.

Anyway. I was so proud I actually bought a domain name for it. So go use and enjoy, and suggest missing events for it, and I'll see you at the VIC-20 club in Kalamazoo.

Kodos for pyparsing

"Kodos for pyparsing"

Perhaps if I say it, with earnest conviction, it will come into existence. Perhaps if I intone it repeatedly; light a candle, maybe.

Or perhaps I will get around to creating the modification to Kodos myself. It might not even be too hard - it just can't be a priority project right now...

Kudos for pyparsing, too, of course.

speaking at IOUG

Whoo! My talk got accepted at IOUG Collaborate!

Congratulations! Your Technical Session (60-minute) proposal below has been accepted by the IOUG Conference Committee for presentation at COLLABORATE 09 IOUG Forum. You are scheduled to present at the following date/time.

405: Long Live the Command Line: SQL*Plus and Alternatives
Thursday, May 7, 2009 from 11:00 AM until 12:00 PM

Yay! I even got a good time slot!

This will be kind of like the talks I've given at Oracle user groups around here... but more polished, of course - as SQLPython itself is more polished.

account rot

I feel a gradually growing sense of nervousness and guilt over the growing trail of accounts I have opened on various web-based services, then abandoned. You know how it is... you try out a service, but 95% of the time you lose interest or find a better one eventually. Then your account just sits there forlornly, quietly counting the months since your last login.

I'd back up and delete the old accounts, but many of them don't provide convenient (or any) account deletion options, and most of them I have just plain forgotten.

What's the harm? Well, I do have a very young nth cousin named Catherine Devlin who is probably going to curse me in 15 years for squatting on her account name EVERYWHERE. Assuming she doesn't want to be "WebkinzBellybuttonLint" or something. Worse, there are just a couple sites where I got the username "catherine" - which is awesome, except when I ended up not using them, leaving a choice username to gather digital dust.

Plus, there's this feeling that any of those accounts is potentially crackable, creating risk of a very mild form of identity theft. OK, I don't care that much if somebody's leaving comments as me on IMDB, but it still doesn't seem quite right.

This is one thing I love OpenID for - my account at liquidID.net has a nice listing of everyplace I've used it. Alas, that represents maybe 0.1% of the services I've signed up for in the past 10 years.

Anybody have a solution? I should probably start a centralized record of all my newly created accounts, but what about the ones I've already forgotten?

Planet CodeMash (Yahoo! Pipes)

When a product is described as, "Easy to use - requires no programming!", run. Run, and don't look back.

That's been my rule, anyway. After my first Yahoo! Pipes project, I'm willing to flex it... a little bit. It's been a blend of agony and ecstasy.

Dozens of CodeMash attendees have listed the URLs of their (human-readable) blogs on a webpage. I wanted to browse their blogs, but not by drilling into each one manually. I wanted to read them in one place, with a Planet CodeMash.

(Would you believe that I couldn't find an online "planetizing" website that would simply take a list like that and generate the planet for me? The closest I found was RSSMix, which wanted me to find the RSS URLs myself - which I did, with Python - but then it couldn't read Feedburner links.)

Enter Yahoo! Pipes. Voilà.

Planet CodeMash

Yahoo! Pipes is good for problems like this where you get web-published data (not only RSS feeds, as I first believed), apply a series of transformations, and publish it.

You design your project by dragging logical blocks around a 2-D graphical flowsheet, with pretty curvey lines connecting them. The modules are well-chosen to provide a fair amount of versatility despite their limited number. I was happy to find a Regex module, as well as Feed Auto-Discovery, which drills into a page and finds its RSS link - perfect for this project.

There are problems, however. The biggest one is that, since the editor runs in your web browser, there are some... annoying... ... lags. Every time you click on anything, it takes a second or two for your click to register. That's more than annoying; when you need to click-and-drag, it's nearly crippling. If you can sit there with your finger on the mouse button, wondering, "Has my click registered? Can I drag yet?", again and again, and not curse somebody, then you're a wonderful person.

Of course, every new environment seems a bit confusing and wrong until you've had a chance to poke all the buttons and twiddle the knobs - but this painful interface pretty much prevents playful exploration, so the environment will remain alien and full of surprises for a long time,

Oh, and the error messages stink. "Oops: System error. Problem parsing response." Gee, thanks.

Still, I did get it done, in a halfway reasonable amount of time, and the result is undeniably pretty.

I will consider Yahoo! Pipes for future projects like this. And if I ever find a way to run its editor locally, it will probably become one of my favorite toys.

gift idea and hot investment tip

Yay for Python 3.0! You knew I had to say that. I'm not actually using 3.0 or even 2.6 very seriously yet, though Brandon Rhodes showed me how to use comprehension for dictionaries, and that's just awesome. So I'm going to try to switch as soon as cx_Oracle and pyparsing are ready.

Anyway, I do have one more nontechnical thing I've just got to talk about. I want to do some free advertising for Microplace, a microcredit investment site. I discovered them a few months ago and was delighted with how smooth and easy they made it to place a microcredit investment. It was as professionally done as a good bank's site, and there is enormous choice in where you want your investment to go.

They're also doing a promotion right now where they'll give you a handmade piggy bank! I got one unexpectedly about a month ago, and I absolutely love it.



The picture really doesn't do it justice - it's delightfully cute. More, the tactile and audio sensation of plunking a coin into it is just... astonishingly satisfying. It's just exactly the way a piggy bank should look, feel, and sound, the Platonic ideal of "piggy bank". I find a nickel under the couch cushions and I practically dance over to the bank to plunk it in. I'm going to get another for my nephew for Christmas. This is what you should get somebody for Christmas (or any other gift-giving occasion).

Obviously, from an investor's point of view, microcredit sucks. The interest rate probably won't even keep up with inflation. That's OK. This is not serious capitalism; this is more like making a loan to a friend in need. I like it that way. Generally, what we hope to buy with money is satisfaction anyway - buying stuff we hope will satisfy us. If I put $200 in a standard investment for a couple years, maybe it would earn me enough to go see a movie, and maybe the movie would be satisfying. But the thought that the money is actually hard at work in somebody's life - that's vastly more satisfying, in my book. (I also find it a handy way to squirrel away little self-earmarked savings funds. If I intend to put aside $100 toward a future bathroom remodeling, but I put it into my regular savings account, I am really bad at keeping track of my specific intention for that money; it just vanishes into the ebb and flow of funds. But it's easy to remember that the $100 investment to Tanzania has a specific purpose.)

a viral Bible campaign

A different topic today, though still with an open-source tie-in.

I've never seen anybody offering free audiobook CDs of the Bible. The only audiobook Bibles I've ever seen were actually kind of expensive.

How come? Unless you live on Mars, somebody has thrust a free paper Bible into your hands at least once in your life. Yet, if you drive a car, you have sometimes stooped to listening to painfully worthless crud on the radio just to alleviate the boredom of driving. If you'd had an audiobook Bible, you probably would have listened to it, if only for your cultural education.

Yet a bit of websearching suggests that, holy cow, this actually hasn't been done yet. Am I wrong?

Probably one reason is that the only well-known translation that isn't strictly copyrighted is the King James Version, which is perfect if you intend to do outreach to 17th-century England. No CD players there, though, so it's not so useful for this purpose.

Alas that all the well-known translation committees apparently preceded or were unaware of the open-source/creative-commons movement. Especially since, you know, Jesus did kind of INVENT the GPL for crying out loud. And explicitly applied it to his teaching. Hello. "Freely you have received; freely give".

Anyway, there is, fortunately, one English translation that is freely distributable, the World English Version, and one site that's done voice recordings of it, audiotreasure.

So here's a plan for a viral campaign.

1. Make up a master audio CD from audiotreasure's MP files. I'm really not ambitious enough to try for more than one CD's worth, so I hope at least the Gospel of Mark will fit. If not, make a new recording with someonewhotalksreallyfast. Or something.
2. Compose a nice CD label. Include the URL (#4)
3. Write a nice letter asking permission to distribute the CDs without charge from places where you find bored drivers: fast-food restaurants, truck stops, etc.
4. Set up a website with the files from #1-3, and instructions on how anybody else can do step #5, becoming another viral site of distribution. It looks like you can make a homemade, home-labelled CD for $0.25 or less, so anybody with a computer can make a hundred or two with a modest commitment.
5. Burn some CDs, label them, get permission and distribute them.
6. Hopefully, others join in and the campaign spreads...
7. Profit! Oh, wait, I guess not.

I'm excited. I plan to do this. I wonder if I can get CDs out there by Christmas? However, if you move faster than me and want to set up the seed site, go for it!

I'm sure the Gideons would have started this years ago, if only their grandkids had taught them how. (I'm teasing! Hey, if any Gideons or similar groups should read this, yes of COURSE you can use the idea.)

Python 3.0: what to do?

In recent months, many people considering taking up Python have asked me what Python beginners should do about the release of Python 3.0. Here's your answer.

nothing

To be more specific,
1. Install Python 2.6
2. (optional) Learn Python 3.0 syntax, and use it in Python 2.6.

It's been confusing, I admit; people hear about Python 3.0 and naturally wonder if Python 2.6 is now obsolete. No, Python 2.6 is specifically designed to make a smooth transition by accepting both Python 3.0 and 2.5 syntax.

The problem with installing Python 3.0 right away is that it will take a while before all the juicy third-party modules are available for 3.0. They should be available immediately for 2.6, though, since 2.6 can run them exactly as 2.5 does.

If you learn and use Python 3.0 syntax, then the code you're writing now in Python 2.6 will work in 3.0 in the future when your third-party modules are ready and you want to convert.

And if you'd rather ignore it all? That's fine, too. The differences are really pretty slight, and if you put off updating all your Python code for a year or two, it will be a very minor endeavor.

In short, it's really nothing to lose sleep over. And you absolutely don't need to hold off on learning Python until the situation is "more stable" - Python 2.6 provides you with a stable platform right now that bridges Python's past and future nicely.

Sprechen Sie GNU/Linux?

My boss has been supportive of my love for open-source, but I've never really been able to make him understand what makes it commercially viable. I think I finally blundered across the right analogy, though.

Simon and Schuster could develop a proprietary written language for their authors to write fiction in. Then, they could try to persuade people to learn Simon-and-Schusterese in order to read their books. In fact, if they did a fantastic job of marketing, they might persuade people to buy the right to learn Simon-and-Schusterese, then buy the books. Meanwhile, their legal department would chase down anybody using Simon-and-Schusterese without proper payment and licensing. Random House, HarperCollins, etc. could do the same with their own proprietary corporate languages. It's vastly more practical, though, for everybody to write in - and sell in - a language that nobody owns, everyone can use, and everyone contributes to.

Of course, in the real world, shared human languages came before publishing companies. But if, somehow, things had happened the other way around, you can see how there would be some initial skepticism (What? Give away our language for free? When we could charge money for it?), but companies that started working with a shared language would eventually dominate, and society would be much richer and more literate for it.

cmd2 0.4, now with testing

cmd2 0.4 has been released. Its biggest new feature is transcript-based testing. Basically, you can write a test suite for your cmd2-based application just this easily:

1. Add to your application script (suppose it's myCmd2App.py):

from cmd2 import Cmd2TestCase
class TestMyAppCase(Cmd2TestCase):
    CmdApp = CmdLineApp
    transcriptFileName = 'exampleSession.txt'

parser = optparse.OptionParser()
parser.add_option('-t', '--test', dest='unittests', action='store_true', default=False, help='Run unit test suite')
(callopts, callargs) = parser.parse_args()
if callopts.unittests:
    sys.argv = [sys.argv[0]]  # the --test argument upsets unittest.main()
    unittest.main()
else:
    app = CmdLineApp()
    app.cmdloop()

2. Run a session of your application. Run all the commands you want to test.

3. Cut-and-paste your entire session into exampleSession.txt.

4. Run python myCmd2App.py -t

Ta-da! cmd2 runs runs your app, issues all the commands saved in exampleSession.txt, and verifies that they produce the same output as in your transcript. Now you can change your app fearlessly without bugs sneaking in.

Finally, cmd2 is now available for Python 2.4 through 2.6.

Monty Python Fluxx

We are so going to play this game at PyCon.

Maybe I'll go to a talk or two, too. Between games.

sqlpython: getting crowded in here

I've come across two other SQL command-line clients lately, both written in Python:

pysql, like sqlpython, is for Oracle only.

sqlcmd is intended to work across all sorts of database backends (Oracle, postgreSQL, MySQL, etc.) seamlessly.

I intend to publicly review them soon. In the meantime, thought I'd let you know of their existence.

speaking at IEEE

In my ongoing campaign to teach Python to every multicellular organism in Ohio, I'm bringing my Smash, Crash, Kaboom Course in Python (you know, the one with the exploding planets) to this Tuesday's meeting of the IEEE-Dayton Section Computer Society Meeting. The chapter invites non-members to attend, too; no charge, but $3 to buy into the pizza.

The talk materials are here.

October 21, 2008 11:30 – 12:30pm
Introduction to Python

Lockheed Martin Corporate Sales

2940 Presidential Drive, Suite 290

Fairborn, OH, 45324 USA

Tags: python programming introduction class meeting

This hCalendar event brought to you by the hCalendar Creator.

Counting votes: You're doing it wrong

I have to admit, the first time I heard about people mistrusting computerized voting, I just felt amused. Silly Luddites, I thought. After all, how hard can it be for a computer to count?

Really hard, it turns out, if you start with the stupid premise that you ought to write every byte of the software from scratch, incorporating no preexisting software of known and verifiable quality. That's the approach that proprietary vending machine makers have taken - presumably to lend credibility to their patents.

There's a new interview with the Ohio Secretary of State, Jennifer Brunner, where she talks very frankly about the serious problems Ohio has found with its voting software. I'm glad Ohio now offers a paper ballot option. I'm going to use it. I truly have no idea what happened to my 2004 vote.

There's a perfect solution waiting to be used: PVote by Ka-Ping Yee, one of the Python community's greats. In his interview with NPR's Science Friday, he describes how voting software should and can be written: as a minimal, readable, high-level program that relies on existing open-source components of thoroughly-verified quality. Ka-Ping, my vote is for you.

PyOhio's table at Ohio LinuxFest

This year, PyOhio decided to buy table space in the nonprofit exhibitors' area of Ohio LinuxFest to publicize PyOhio (as well as PyCon and Python in general).

It was a really good decision! The "hallway track" is one of the most interesting parts of any conference, of course, but when you have a table of interesting stuff to draw people in and start conversations about, it goes to the next level. I barely made it into any formally scheduled events at all, and had lots of fun meeting people from the big and growing open-source community. I tried to preserve my voice, but I was half-hoarse by the time my 4:00 talk started.

• Ponyshow was a big hit - it caught peoples' eye and drew them over to the table. I'd still like to add more flashy graphics to it for next year, though - I ran short on time, and had trouble installing pyglet. Tables in the nonprofit zone lacked electricity, but I used two laptops to get around that - one on display at the table and one recharging at an outlet elsewhere.
• Python stickers donated by PyCon were an even bigger hit. If 1/2 the people who took a sticker are using or will use Python, we've got a very healthy community here!
• I whipped up a homemade PyOhio banner that went pretty well. I projected our logo onto a wall, traced the outline, used an Exacto knife to make a stencil from the pattern, then used some fabric spray paint.
• For next year: bring candy. We may see if we can do a swag raffle of our own, too.
• A rerun of last year's Python introduction went well. The Python Beginners' Hackathon was good, but small. We'll have to think about what might need changes there.
• I really like the idea of PyOhio running something on the Friday of next year's LinuxFest. Join the pyohio-organizers mailing list to help kick around ideas for that.

There was a lot of interest among the attendees. There were more people already actively using Python than I expected, and virtually everyone else knew Python as something they wanted to learn more about. I think we'll see that reflected in an even bigger and more intense PyOhio next year.

Ohio LinuxFest 2008 - talk materials

Thanks to my Ohio LinuxFest audience for learning some Python with me! The talk materials are here.

Here's the "Resources for Python Learners" handout.

I had a great time at OLF, particularly in the "hallway track"; we have a really fun and growing open-source community in Ohio. Go, us!

I can plot that data in two keystrokes

(plus a carriage return)

One of the ideas I most gleefully stole from YASQL for sqlpython is special terminators, sequences like \g and \c that replace a SELECT statement's ending semicolon. When a query ends with a special terminator, the output is specially formatted: \c gives CSV, \h gives HTML, \t gives transposed (columns as rows / rows as columns), etc. Type help terminators for details.

sqlpython 1.5.0 is out today, with the most demented special output format yet: CHARTS! Instant ad-hoc grapical goodness direct from your query, no tedious mucking around in spreadsheets or exporting to another program. Just terminate your query with \l (line graph), \L (scatter graph - no lines), \p (pie chart), or \b (bar graph).





Also, as of 1.5.0, it's pretty easy to define your own special terminators and formats. Just install sqlpython in uncompressed form (easy_install -UZ will do that), open up output_templates.py, and follow the pattern.

ponyshow: showing off in Python

PyOhio is getting a table at Ohio LinuxFest to advertise PyOhio and Python in general. We're going to set up a computer running demonstrations of eye-catching Python tricks - stuff passing geeks can look at and think, "Hunh! That's pretty cool! I'm going to try this Python thing."

To run the demo, I've written a little script called ponyshow. You can use it yourself (on *nix) - install Mercurial, then
hg clone http://hg.assembla.com/ponyshow ponyshow

I need suggestions for what to put in the show! If you had a few lines of code to show why you love Python, what would they be? Importing modules is fine - I'm certainly going to show off vPython and pyglet, for example. What would you show?

more area events

Ohio LinuxFest isn't the only major geeky event coming up in our area. Also check out:

• the Miami Valley Joint Societies meeting: more geeky groups than you can shake a USB drive at, all coming together for their annual dinner meeting. Sep. 25 in Dayton.
• .NET University: a free 101-level education in .NET. Nov. 8 in Columbus.
• 50 Years of Computing at UC: the big picture. Nov. 10-11 in Cincinnati.

Geek Event Finder: now working

It's working! The Geek Event Finder on Google App Engine! Go play!

In some ways, the Google App Engine is a dream. Not thinking about the app server is wonderful. Deploying couldn't be easier.

I hate the GAE datastore with the passion of a thousand blazing suns, however. 90% of the work of this project has been trying to figure out workarounds and kludges for its bizzare limitations, like

• There is no mass delete. None. No truncate. No way to get rid of a large number of records at once.
• Bulk upload exists, but it always appends to the datastore - never replaces - which brings you right back to the "no mass delete" problem.
• No long-running operations - anything that would take more than a couple seconds dies - so you can't loop over all your records to do something (like delete them).
• Countless unexpected restrictions on queries. Can't filter on one property while sorting on another. Can't do inequality filters on more than one property. Can't filter on a string property if it is multiline (has \n's) or is longer than 500 characters (type Text). Can't use any function calls or arithmetic within a WHERE clause. Queries that fetch a large number of records die instead of completing (so I fetch in LIMIT 20 batches and assemble the results on the app side... crazy).

So if you look at the code and see some incredibly stupid stuff going on with data access - trust me, I tried fifteen different sane ways first. I am so not buying the buzz about this being "the future of databases". Fighting for hours to try to kludge your way to your data... that's the Bad Old Days, not the future.

My workaround for mass deletion was to write pages that would delete one record, then invoke it in a loop from my client computer. That has to run all night to clean out the datastore when new data is uploaded.

But anyway. I'm still very happy. It works!

Ohio LinuxFest

Ohio LinuxFest is Oct. 11 - one month from today!

It's free, it's fabulous. Missing it would be like missing your own birthday. Go get registered!


The PyOhio gang is going cook up some good stuff to do - a table in the midway, a Python workshop in the Open Spaces, etc. Let me know if you have ideas and/or if you'd like to help staff the table.

BigTable blues

This was supposed to be the blog entry where I would announce the Geek Event Aggregator's successful port to Google App Engine.

(sigh)

I've read an awful lot of buzz about how GAE's BigTable is the Next Big Thing in data, makes RDBMS obsolete, etc. Maybe I'm just doing it wrong, but right now I am utterly unimpressed.

The Geek Event Aggregator needs to search its database of 5000 or so events for events whose longitude and latitude are close enough to the user to be of interest. Does that sound so impossible?

I couldn't do it in GAE. First, "Inequality Filters Are Allowed On One Property Only" - so I can filter for longitude or latitude, but not both. I had to filter only for longitude, pull all resulting records into the application, and finish boiling the ocean in my app. It was slow, in the local application environment, but I hoped it would run faster once uploaded to the actual GAE production servers.

In production, though, it doesn't run at all - "Timeout: datastore timeout: operation took too long.". Querying from 5000 records - too much for the mighty BigTable, apparently. Dropping the filters on longitude (to do all the filtering in the app, in case inequality filtering is just so poisonous) didn't help, either.

Oh well. I still enjoyed working with GAE at first, and maybe I'll use it again for something with very light data demands. For the Geek Event Aggregator, I do have a server available where I can host in TurboGears - it'll just take a bit of rewriting. Later this week, hopefully.

recursive dowload with wget

I often need to download documentation for offline use. Online docs typically come as a finely branched tree of separate .html files; sometimes a one-page or PDF version is also available, but those aren't as convenient to use.

I'd gotten sick of using "File/Save As..." from my browser again and again, but Firefox shortcuts like DownThemAll flattened the directory structure, broke the internal links, and didn't crawl them to get sub-pages.

The answer, of course, was in my /usr/bin all along: wget.

catherine@Elli:~/docs$ wget --recursive --convert-links \
> --page-requisites --no-parent \
> http://www.postgresql.org/docs/8.3/interactive/index.html

or, shortcut form,

catherine@Elli:~/docs$ wget -rkp -np \
> http://www.postgresql.org/docs/8.3/interactive/index.html

pulls down the everything beneath the index page, and it turns out perfectly, with a single command!

--convert-links adjusts all the internal hyperlinks so they'll work perfectly in your downloaded version.

Without --no-parent, wget would find the link to "Home" and download everything under that.

--page-requisites makes sure you get the images, stylesheets, etc. for your pages to work properly.

Now bookmark the downloaded top page in firefox, give the bookmark a keyword, and voila! Now just typing your keyword into the Firefox URL bar gets you to your perfect offline mirror.

Regex skirt review

I got my xkcd regex skirt as a birthday present (funny what subtle hints like "this is exactly what I want for my birthday please please please" will do), so I thought I'd give a review. An hReview-microformatted review, of course.

gloriously geeky, if flawed

Aug 22, 2008 by
Catherine Devlin
(sam18xgy4fz@liquidid.net)product

XKCD Regex Cheat Skirt

Rating: 4 of 5

We've all got a drawer full of geeky T-shirts. But what about a geeky skirt? Thanks to XKCD, it finally exists!

The text is regular expressions syntax reminders, oriented upside down so you can read it while you sit. Simple stuff, like:

Must escape: \|()[{^$*+? 
^ start of string 
$ end of string 
[^abc] ^ means not 
\n Newline 
{N,M} N to M

The font is large, comfortable to read while sitting. I wish the lines weren't double-spaced - I think 1.5 spacing would make the text look more like a trendy decorative hem accent.

The sizes run big. I wear 10's or 12's, ordered the medium, and it is more than big enough, even without the stretchiness. It's a soft, stretchy fabric, stretchy enough to do Rockettes high-kicks if that's what you do (but programming pays better).

The hem falls a bit above the knee, but not at all "you are not leaving the house like that young lady" high.

I'm a little disappointed in the craftsmanship, to tell the truth, for the $35. It kind of looks like a skirt made by T-shirt makers - it's two simple trapezoidal panels sewn together. The text leaves a fairly broad gap at each edge of the panels, making it really obvious where the side seams are. (I guess that could be considered a fashion choice, but I prefer hiding the seams.) The print is that slightly raised plasticky print, and I have zero confidence in its longevity. I will obviously not only have to launder it gently as advised, but also refrain from wearing it every single day. Save it for geeky occasions where it will be properly appreciated.

Nonetheless, it is far and away my favorite skirt, because it is far and away the winner of the "geekiest skirt ever" contest, and that trumps everything in my book.

0.3

This hReview brought to you by the hReview Creator.

taking FOSS to the people

It's all very well to invite newbies to come to your Linux group, but you're not going to get really new people that way. Coming to a meeting or an Installfest is a commitment that demands strong, specific curiosity at the very least. Chances are, if someone knows enough about Linux to know that they want to know Linux, they're going to learn it, helpful usergroup or not. That's especially true now that the entry barrier is so low with super-easy distributions like Ubuntu. But what about those who don't know what they're missing?

That's why some of us from Dayton Linux User Group are considering renting a booth at the Montgomery County Fair. We can set up several computers and let people touch, try, and see just how easy it is - people who had never even heard of Linux or open-source, people who had no idea there was a choice, who thought that Microsoft was just how computers run. Step right up, folks. Surf a webpage, write a document, edit a spreadsheet. It's real, and it doesn't bite.

"Do you know that Tux loves you, and has a wonderful plan for your computer?"

(WOW, if only I had cartooning talent, I would make THE BEST mockup of a lowbrow evangelical tract there has ever been.)

(Wait a minute. Since when did that require cartooning talent?)

(Incidentally, I really am a hardcore Christian. Seriously. Maybe that's why this sort of thing appeals to me so much. "Freely you have received, freely give"; Jesus wrote the GPL!)

The only thing that may stop us is the booth fee: $270. It's reasonable, all things considered, but it's still a sizable hat to pass around for a small group.

Anyway, whether we get this together in time for the fair or not, I'd love to get other ideas for where to take a show like this. The fair is perfect because it has a whole lot of people milling around looking for something interesting. Where else?

Good Kids and Open Spaces

Not everything at PyOhio was perfect, of course. In particular, we were frustrated that we didn't get more participation in the Open Spaces. Some of the attendees pretty much took a spot in the auditorium, watched the scheduled talks, and went home. I think they were satisfied, but those attendees missed out on a lot.

I wasn't too surprised, since even PyCon wrestles with how to encourage Open Space participation. Many PyOhio attendees were inexperienced conference-goers with absolutely no experience with Open Spaces. But I don't think unfamiliarity was the only barrier.

Back in high school, I was one of the Good Kids. I was Born to be Mild. I went to class early, sat down, readied my notebooks, and waited for the teacher to start. I did not hang around in the hallway gossiping. I certainly did not cut class!

I think most people who become programmers started out as Good Kids, and that's part of why this is hard. We can put a preachy message in the program guide singing the virtues of Open Spaces and the hallway track, of informal, collaborative learning - but they were Good Kids, and they're not buying it. When we say, "get into unplanned discussions, skip some scheduled talks in favor of Open Spaces", it sounds to them like "cut class and hang out." No wonder it's a tough sell! Closing their ears to that is exactly how they became educated enough to be at a programming conference in the first place. In high school, the things your peers could teach you would probably get you arrested, hospitalized, and/or pregnant.

So we've got some ideas to get people over this psychological hump for PyOhio 2009. We're going to use panel discussions to ease people from listening into talking, try a schedule that forces people to get up and move around and bump into each other, and leave time slots where there are no scheduled talks.

Honest, everybody, it's OK. One Good Kid to another.

[EDIT: Let me make clear that this wasn't the only thing keeping people in their seats. We had a jam-packed schedule - gave in to the "fit everything in" temptation, I'm afraid; the Open Space rooms were distant from the main lecture room; and there was no natural transition to a hanging-out space when the library closed at 6. We'll certainly be looking hard for ways to improve those for PyOhio 2009.]

PyOhio: wow

Wow. PyOhio went great. I have so much to say, I hardly know where to start. Expect multiple posts.

The hope was to duplicate some of the learning and atmosphere of PyCon on the small scale. I really think we achieved it. The talks were excellent, attendee enthusiasm was high, and plenty of people stepped up to the tasks that needed doing.

THANK YOU to everybody who was involved!

Not everything was perfect, of course. I'll devote a full post to glitches and mistakes. But in general, I'm extremely happy. We could have sat around wondering whether this area could really support a good regional conference, but instead, we just went ahead and did the experiment, and the answer is clear: yes, it can and it did!

Check out some of the neat swag donation we got from WearPython and Apress!

PyOhio: one more week

PyOhio is Saturday, July 26!

Please register right away - it helps us schedule talks properly and guarantees you entry, food, and swag.

What sort of Open Spaces do you want to take part in? Do you want to do a poster? How about a Lightning Talk?

See you there!

serendipity

I just met with Amanda Laucher at the Blue Moose Cafe in Morgantown, WV.

I don't live here; she doesn't live here; we weren't at a conference; we didn't expect to meet. We were both in town on family visits, and both seeking out an internet fix. (I'm visiting my father-in-law at Ruby Hospital - but writing Python there, hah.)

She's a serious geek-community addict like me - she's organizing the DevLink Bus. She gave me great advice for PyOhio and tipped me off to other upcoming events, and I gave her some ideas from PyCon. It was great!

That's just really cool. Someday, when wireless access is ubiquitous, geekish nomads will no longer concentrate themselves in a handful of WiFi hotspots, and unforeseeable meetings like these will become more rare. That will be the downside of the future.