recursive dowload with wget

I often need to download documentation for offline use. Online docs typically come as a finely branched tree of separate .html files; sometimes a one-page or PDF version is also available, but those aren't as convenient to use.

I'd gotten sick of using "File/Save As..." from my browser again and again, but Firefox shortcuts like DownThemAll flattened the directory structure, broke the internal links, and didn't crawl them to get sub-pages.

The answer, of course, was in my /usr/bin all along: wget.

catherine@Elli:~/docs$ wget --recursive --convert-links \
> --page-requisites --no-parent \
> http://www.postgresql.org/docs/8.3/interactive/index.html

or, shortcut form,

catherine@Elli:~/docs$ wget -rkp -np \
> http://www.postgresql.org/docs/8.3/interactive/index.html

pulls down the everything beneath the index page, and it turns out perfectly, with a single command!

--convert-links adjusts all the internal hyperlinks so they'll work perfectly in your downloaded version.

Without --no-parent, wget would find the link to "Home" and download everything under that.

--page-requisites makes sure you get the images, stylesheets, etc. for your pages to work properly.

Now bookmark the downloaded top page in firefox, give the bookmark a keyword, and voila! Now just typing your keyword into the Firefox URL bar gets you to your perfect offline mirror.

Regex skirt review

I got my xkcd regex skirt as a birthday present (funny what subtle hints like "this is exactly what I want for my birthday please please please" will do), so I thought I'd give a review. An hReview-microformatted review, of course.

gloriously geeky, if flawed

Aug 22, 2008 by
Catherine Devlin
(sam18xgy4fz@liquidid.net)product

XKCD Regex Cheat Skirt

Rating: 4 of 5

We've all got a drawer full of geeky T-shirts. But what about a geeky skirt? Thanks to XKCD, it finally exists!

The text is regular expressions syntax reminders, oriented upside down so you can read it while you sit. Simple stuff, like:

Must escape: \|()[{^$*+? 
^ start of string 
$ end of string 
[^abc] ^ means not 
\n Newline 
{N,M} N to M

The font is large, comfortable to read while sitting. I wish the lines weren't double-spaced - I think 1.5 spacing would make the text look more like a trendy decorative hem accent.

The sizes run big. I wear 10's or 12's, ordered the medium, and it is more than big enough, even without the stretchiness. It's a soft, stretchy fabric, stretchy enough to do Rockettes high-kicks if that's what you do (but programming pays better).

The hem falls a bit above the knee, but not at all "you are not leaving the house like that young lady" high.

I'm a little disappointed in the craftsmanship, to tell the truth, for the $35. It kind of looks like a skirt made by T-shirt makers - it's two simple trapezoidal panels sewn together. The text leaves a fairly broad gap at each edge of the panels, making it really obvious where the side seams are. (I guess that could be considered a fashion choice, but I prefer hiding the seams.) The print is that slightly raised plasticky print, and I have zero confidence in its longevity. I will obviously not only have to launder it gently as advised, but also refrain from wearing it every single day. Save it for geeky occasions where it will be properly appreciated.

Nonetheless, it is far and away my favorite skirt, because it is far and away the winner of the "geekiest skirt ever" contest, and that trumps everything in my book.

0.3

This hReview brought to you by the hReview Creator.

taking FOSS to the people

It's all very well to invite newbies to come to your Linux group, but you're not going to get really new people that way. Coming to a meeting or an Installfest is a commitment that demands strong, specific curiosity at the very least. Chances are, if someone knows enough about Linux to know that they want to know Linux, they're going to learn it, helpful usergroup or not. That's especially true now that the entry barrier is so low with super-easy distributions like Ubuntu. But what about those who don't know what they're missing?

That's why some of us from Dayton Linux User Group are considering renting a booth at the Montgomery County Fair. We can set up several computers and let people touch, try, and see just how easy it is - people who had never even heard of Linux or open-source, people who had no idea there was a choice, who thought that Microsoft was just how computers run. Step right up, folks. Surf a webpage, write a document, edit a spreadsheet. It's real, and it doesn't bite.

"Do you know that Tux loves you, and has a wonderful plan for your computer?"

(WOW, if only I had cartooning talent, I would make THE BEST mockup of a lowbrow evangelical tract there has ever been.)

(Wait a minute. Since when did that require cartooning talent?)

(Incidentally, I really am a hardcore Christian. Seriously. Maybe that's why this sort of thing appeals to me so much. "Freely you have received, freely give"; Jesus wrote the GPL!)

The only thing that may stop us is the booth fee: $270. It's reasonable, all things considered, but it's still a sizable hat to pass around for a small group.

Anyway, whether we get this together in time for the fair or not, I'd love to get other ideas for where to take a show like this. The fair is perfect because it has a whole lot of people milling around looking for something interesting. Where else?

Good Kids and Open Spaces

Not everything at PyOhio was perfect, of course. In particular, we were frustrated that we didn't get more participation in the Open Spaces. Some of the attendees pretty much took a spot in the auditorium, watched the scheduled talks, and went home. I think they were satisfied, but those attendees missed out on a lot.

I wasn't too surprised, since even PyCon wrestles with how to encourage Open Space participation. Many PyOhio attendees were inexperienced conference-goers with absolutely no experience with Open Spaces. But I don't think unfamiliarity was the only barrier.

Back in high school, I was one of the Good Kids. I was Born to be Mild. I went to class early, sat down, readied my notebooks, and waited for the teacher to start. I did not hang around in the hallway gossiping. I certainly did not cut class!

I think most people who become programmers started out as Good Kids, and that's part of why this is hard. We can put a preachy message in the program guide singing the virtues of Open Spaces and the hallway track, of informal, collaborative learning - but they were Good Kids, and they're not buying it. When we say, "get into unplanned discussions, skip some scheduled talks in favor of Open Spaces", it sounds to them like "cut class and hang out." No wonder it's a tough sell! Closing their ears to that is exactly how they became educated enough to be at a programming conference in the first place. In high school, the things your peers could teach you would probably get you arrested, hospitalized, and/or pregnant.

So we've got some ideas to get people over this psychological hump for PyOhio 2009. We're going to use panel discussions to ease people from listening into talking, try a schedule that forces people to get up and move around and bump into each other, and leave time slots where there are no scheduled talks.

Honest, everybody, it's OK. One Good Kid to another.

[EDIT: Let me make clear that this wasn't the only thing keeping people in their seats. We had a jam-packed schedule - gave in to the "fit everything in" temptation, I'm afraid; the Open Space rooms were distant from the main lecture room; and there was no natural transition to a hanging-out space when the library closed at 6. We'll certainly be looking hard for ways to improve those for PyOhio 2009.]

PyOhio: wow

Wow. PyOhio went great. I have so much to say, I hardly know where to start. Expect multiple posts.

The hope was to duplicate some of the learning and atmosphere of PyCon on the small scale. I really think we achieved it. The talks were excellent, attendee enthusiasm was high, and plenty of people stepped up to the tasks that needed doing.

THANK YOU to everybody who was involved!

Not everything was perfect, of course. I'll devote a full post to glitches and mistakes. But in general, I'm extremely happy. We could have sat around wondering whether this area could really support a good regional conference, but instead, we just went ahead and did the experiment, and the answer is clear: yes, it can and it did!

Check out some of the neat swag donation we got from WearPython and Apress!

PyOhio: one more week

PyOhio is Saturday, July 26!

Please register right away - it helps us schedule talks properly and guarantees you entry, food, and swag.

What sort of Open Spaces do you want to take part in? Do you want to do a poster? How about a Lightning Talk?

See you there!

serendipity

I just met with Amanda Laucher at the Blue Moose Cafe in Morgantown, WV.

I don't live here; she doesn't live here; we weren't at a conference; we didn't expect to meet. We were both in town on family visits, and both seeking out an internet fix. (I'm visiting my father-in-law at Ruby Hospital - but writing Python there, hah.)

She's a serious geek-community addict like me - she's organizing the DevLink Bus. She gave me great advice for PyOhio and tipped me off to other upcoming events, and I gave her some ideas from PyCon. It was great!

That's just really cool. Someday, when wireless access is ubiquitous, geekish nomads will no longer concentrate themselves in a handful of WiFi hotspots, and unforeseeable meetings like these will become more rare. That will be the downside of the future.

cx_Oracle and Oracle XE on Ubuntu

Simplifying options have appeared since the last time I posted about installing cx_Oracle against Oracle XE on Ubuntu, so I thought I'd produce an updated, simplified summary.

1. Install Oracle XE from Oracle's own repository of .deb files
2. export ORACLE_HOME=/usr/lib/oracle/xe/app/oracle/product/10.2.0/server/
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH
3. sudo apt-get install python-dev python-setuptools
sudo -E easy_install cx_Oracle
   The -E flag carries your environment variables into your sudo session; if you omit it, you may get a cannot locate an Oracle software installation error.
4. Test:
$ python
>>> import cx_Oracle
>>> cnxn = cx_Oracle.connect('system/mysystempassword@xe')
>>> crsr = cnxn.cursor()
>>> crsr.execute('SELECT * FROM dual')

cmd2 0.3.4

I just keep having fun with cmd2 (and with sqlpython, which is built on cmd2 and inherits its goodies). The latest:

• If you use > to redirect output and omit a destination filename, your output will go into the paste buffer. Then you can Ctrl-V and paste it into any window.
• You can use the pipe symbol (|) just like in Unix, to redirect output into any shell command.

geographical anonymity

On the Internet, nobody knows you live in Dogtown.

I've volunteered to do the publicity for PyOhio. I guess it's going really well. Two months out, we've already got 80 people signed up for the PyOhio announcements mailing list. If we assume that most of them will come to PyOhio, plus some others, and that the numbers will keep growing... maybe I'd better relax before we max out the library's capacity.

Still, bootstrapping a regional community is an interesting challenge. It's easy to go out on the internet and find a bazillion people who use Python - but how many of them are in or near Ohio? The wonderfully free-floating nature of the Internet and the open-source community, where geography is irrelevant, can become frustrating when geography actually is relevant.

Finding local companies is even harder. Most companies' websites try to give the impression that they have no geographical location whatsoever - wouldn't want to put off a potential client just because they're on another continent, after all. They also try to avoid mentioning the languages they use, again trying not to appear limited. (In a sense, that's a very good thing, because end users should shop for brains, not tools - it's just difficult for this application.) Finally, with a technology like Python, many of us users sneak it into our jobs without our higher-ups even knowing it, much less promoting it as a strength of their companies.

Any ideas? Googling for "Python Ohio" gets... well, me. Looking for "Python" plus the domain names of Ohio universities is helping a bit. (Universities, unlike businesses, do publicize their physical locations and the technologies they use.)

Anyway, if you've been thinking of submitting an abstract for PyOhio, quit thinking and write; our deadline is June 1 (this Sunday). And don't forget to blog...

That number again:

PyOhio

Regional Python programming miniconference

July 26th, 2008

Columbus Metropolitan Library

96 S. Grant Ave.

Columbus, OH 43215
USA

Tags: python open-source programming conference ohio

(Side note: Blogger's captchas are very near the threshhold of legibility, and getting worse. If I quit posting, I guess you'll know why.)

slides from SQL*Plus Alternatives

I've posted my slides from today's GCOUG presentation here.

Thanks for being a great audience! Feel free to contact me to ask questions about SQLPython or request new features.

cmd2 0.3.0

I'm speaking on SQL*Plus alternatives again on Wednesday at GCOUG, and trying to polish up sqlpython a bit in preparation for that. That required polishing cmd2; I've been increasingly embarrassed that cmd2 used homemade code (flagReader) to interpret --options --like --these, instead of using a perfectly good module from the standard library (optparse).

Optparse is not designed for iterative cmd-type apps where you remain at an application prompt after issuing each command - it keeps issuing sys.exit()s to kick you back to the shell. That was my excuse for not using it earlier, but once I decided to fix it, a little subclassing took care of it.

Next I got a little deeper into decorators than I'd ever been, to provide a concise way to specify those options:

@options([make_option('-k', '--ketchup', action="store_true", help="Ketchup on that?"),
         [make_option('-r', '--relish', help="relish (sweet or dill)")])
def do_hotdog(self, arg, opts):
    if opts.relish:
        self.stdout.write('One dog with %s relish\n' % opts.relish)
    if opts.ketchup:
         self.stdout.write('Hope you're not wearing white!\n')

That was pretty fun. Took some experimentation and this article, though, to figure out how to write a decorator that would accept an argument.

I didn't throw away the homemade ("flagReader") code, for backward compatibility; I issue a DeprecationWarning instead. I did throw away the guts and reimplement it as a wrapper around optparse. I managed to do that part on a single 20-minute bus ride, thanks to good doctests on flagReader. I feel so grown-up!

Finally, inspired by Brandon Rhodes' article in the March 2008 Python magazine, I tried supporting buildout deployments this time. I'm not sure I'm getting it quite right yet; expect it to actually work in cmd2 0.3.1.

PyOhio Call for Proposals

PyOhio: Call for Presentations

PyOhio, the first annual Python programming mini-conference for Ohio and surrounding areas will take place Saturday, July 26, in Columbus, Ohio. The conference is free of change and will include scheduled presentations, Lighting Talks and unconference-style Open Spaces.

You can read more about the conference at http://pyohio.org

PyOhio invites all interested people to present scheduled talks. All presentations are expected to last 40 minutes with a 10 minute question-and-answer period. PyOhio will accept abstracts covering any area of Python programming. A classroom area with computers will also be available for possible hands-on sessions.

All presentations proposals should submit abstracts no longer than 500 words in length. Abstracts must include the title, summary of the presentation, the expertise level targeted, and a brief description of the area of Python programming it relates to.

All proposals should be emailed to for review.

The submission deadline will be June 1, 2008. Accepted proposals will be notified by July 1.

choose the tool last

Yesterday, someone asked me if I could design "an MS Access database" for him.

Growl.

As I reflected on it, I realized that the problem is not really the particular technology specified - the problem is that the technology was specified at all before the need was described.

If I offered to make you anything you wanted for dinner, would you ask for "something made with a Teflon spatula?" If you did, I'd tell you, "No no no no no. Describe what you want to eat, and let me figure out what I'm going to use to cook it."

My potential client didn't literally mean that he wanted MS Access to be used, he meant that he wants a database with a GUI interface. He thinks of "Access database" as a way to describe that, because it's the only thing he's ever seen used to build products vaguely resembling the one he's imagining. It's nothing more than a limitation of vocabulary and experience.

The problem is that, so often, that kind of pidgin is taken literally. Tools are pre-selected far too early - while the business users are imagining their needs, before the geeks themselves are even involved,

As a result, there are a lot of geeks out there trying to beat eggs with Teflon spatulas.

My story ends happily, however. My potential client got it when I explained this, and we're going to meet so I can find out what he wants. Then - and only then - will we start discussing which tools to use.

Penguicon talk materials

Ooh - sorry, I got distracted! Here, at last, are the materials for my talk last Saturday, "Into the Dayta-b'hase: A First-Level Introductory Adventure".

A tarball is here with the slides (dnd.html), queries run (scripts.sql), and a big SQL script to set up the database (dnd.sql)

tax rant

ABSTRACT

Citibank spends a lot of time imploring their customers to go to all-electronic statements... but they don't improve their electronic statement availability to make them practical. Ohio Dept. of Taxation spends a lot of time imploring people to file electronically... but they can't be bothered to test their forms, or to provide a feedback button so that end users can notify them of errors they find.

This is a bad pattern. If you want people to go electronic, focus on getting it right, not on pro-electronic cheerleading.

* * * * *

Ohio has a "use tax" - a law requiring that you add up all your mail-order and interstate purchases from out of state so you can retroactively pay Ohio sales tax on them with your tax return. I think six or seven of us actually obey this law, and the folks at the bureau of taxation laugh, blow a kazoo, and scrawl "PATSY!" on the return whenever they come across someone who paid it. Oh, well.

Digging up records on all these purchases sucks. Citibank, though I love their customer support*, didn't help. (You can download your last six statements, one at a time... or, after March, you can request a PDF report of the whole last year - please allow two business days for your request to be processed. Excuse me? Some people store information on devices called "computers"? A year of credit card statements is about 50 KB... I think you can afford to keep that on disk. Ask Google if you're hard up, they'll give you a few gigs.)

So anyway... I finally, FINALLY, splice electronic records and sift paper records and total up my internet purchases for the year... open up the Adobe form from tax.ohio.gov... and...

...and the line to enter your quantity of out-of-state purchases is not editable. It is pre-set to zero and locked.

One good thing has come out of tonight: I found wesabe, a really nice site for slurping up records from your various accounts automatically. Next year I won't need to fight Citibank, because my Wesabe account will keep itself updated all year long. Wesabe, I'm happy to say, gets it right. Free services generally do, as opposed to people I pay, who generally get it wrong. It's a strange era we live in.

(* - When my credit card was stolen years ago, Citibank was absolutely wonderful to me. It'll take a lot of blundering to erode the goodwill they earned from me then.)

PyOhio

It's official! We have a place, we have a date... now all we need is you!

PyOhio

Regional Python programming miniconference

July 26th, 2008

Columbus Metropolitan Library

96 S. Grant Ave.

Columbus, OH 43215 USA

Tags: python open-source programming conference ohio

Alas, Mendelson's

Mendelson's is planning to shrink their store and relocate. This is very sad news. I can only hope they don't shrink too much or move too far. Fortunately, they have no timetable for this move yet, and if we're lucky a giant asteroid will hit the earth and wipe us all out first.

If you don't know Mendelson's, the best way I can describe it is "a Jawa sandcrawler parked in downtown Dayton". It's a giant industrial salvage shop, the most amazing pile of thingies and whatsits you can imagine. I consider it the most entertaining location in our fair city. If you can walk through Mendelson's without saying, "MY robot will have THIS!", then you are already dead.

Beautiful Visual Python examples

The best thing that can happen, when you try to teach something, is to have somebody learn it and do something that amazes you. That's exactly what happened after my demo of Visual Python at CodeMash... Doug Mair started to play with vPython and came up with two really beautiful examples: a (working!) Rubik's Cube and a set of magnetized disks.

Thank you, Doug!

PyOhio

a regional Python miniconference

Ohio had a good representation at PyCon, and we agreed that the time has come.

http://pyohio.org

announcement-only list: http://mail.python.org/mailman/listinfo/pyohio

planning list: http://mail.python.org/mailman/listinfo/pyohio-organizers

Almost everything is up in the air right now... all we know for certain is that we're aiming for one day, presumably a Saturday, in the Columbus, OH area. Join us, and help the plan take shape!