%sql to Pandas

After getting %sql magic for IPython working, my next big goal was to figure out how to get those results into Pandas.

Er, OK, not such a big goal. Even with zero Pandas experience, it took about five minutes of skimming the first page of documentation to figure out:

In [1]: %load_ext sql

In [2]: data = %sql postgresql://will:longliveliz@localhost/shakes select * from work

In [3]: import pandas as pd

In [4]: s = pd.DataFrame.from_records(data, columns=data.keys)

This is not the only way to move data from an RDBMS to Pandas (there's pandas.io.sql, for example), and I don't know enough about Pandas to know if it's the best way. But I bet it's the easiest way.

released: %sql magic for IPython

Inspired and informed by discussions with the IPython developers at PyCon 2013, I've released ipython-sql, a %sql magic for IPython.

With this, I really think the IPython Notebook will become the most amazing database tool ever. In fact, virtually every computing problem will become a lot more workable when manipulated via the IPython Notebook - you can remember, inspect, and annotate all the steps to investigate whatever your issue is. All hail the Notebook! The FSF really chose well in choosing Fernando Perez for their 2012 award.

The Canadian menace

Incidentally, some people have been asking, "Wait a minute - PyCon-US in Canada? How does that work? Wouldn't it be more correct to call it PyCon-NA for North America?"

It might, if this were a case of nations cooperating to share PyCon. However, that is not the case. You will notice that the Canadians haven't cancelled their own PyCon. Rather, they have seized PyCon-US by ruthless volunteerism and are even now dragging it off to their stronghold on the St. Lawrence, to hoard it along with the PyCon they already have. That's right, they want ALL THE PYCONS.

Pythonistas of the world, be warned! When friendly faces from the North arrive to lend a hand, watch them carefully! Or you'll soon be flying to Vancouver for PyAr and Winnipeg for EuroPython.

Just to clarify: I am giddy over having PyCon in Montreal. I'm so excited that they'll probably need to name a Montreal Syndrome to go along with Jerusalem Syndrome and Paris Syndrome.

post-PyCon post

You might be sick of me saying after each PyCon, "That was the best PyCon ever!", but it's not my fault if it's true.

I hardly know where to start summing up the highlights...

• PyPGDay was a great addition! I've had virtually no exposure to the PostgreSQL community before, so this was very valuable to me. Evan Klitzke from Uber gave a talk on migrating to PG from MySQL that is going to save me a ton of time, and Jeff Davis from Aster showed off the huge usefulness of range types.
• Naomi had to talk me into the Education Summit, but I'm glad she did - I got a lot of great ideas and inspiration that will help in teaching future workshops.
• One of these ideas was the use of Matt Davis' fantastic ipythonblocks, which will let us do graphical exercises right within the IPython Notebook - an amazing, intuitive, seamless learning experience.
• Speaking of the IPython Notebook, this PyCon was really its tour de force. Seemed like everyone was using it to do and show just amazing things. If you haven't seen some talks on it yet, go watch a bunch of videos immediately and then watch a bunch more when the PyCon 2013 talks are online - the docs alone can't do justice to the possibilities the Notebook creates. We've only begun to take advantage of this fantastic environment.
• I mentioned my ambition to create an IPython-based SQL client to Fernando Perez from the IPython team, and he jumped to show me what I need to know. The day after coming home, I checked in a %%sql magic. It's not ready for prime time yet (or even PyPi) and it may warrant merging into a similar project, but it was a delight to play off the capability of the Notebook.
• Peter Wang and Travis Oliphant showed me - personally! - Wakari, an amazingly powerful online hosted Python environment. I can't wait to play.
• I'm not going to list all the people I loved seeing and catching up with, because it wouldn't mean much to most blog readers. But the fact is it would be a lonely year if I couldn't see my PyCon friends (and make some new ones).
• If GitHub had an AI, it would be looking at me funny and asking, "What's got into you lately?" PyCon, that's what. And I'm nowhere near done.

Thank you to the fantastic bunch of volunteers who make this such an amazing conference, and to all the participants who bring their ideas and their friendship.

HTSQL lightning talk slides

I posted the slideshow from my PyPGDay HTSQL lightning talk here. Thanks to everybody involved with PyPGDay, I loved it!

Forsooth, a dataset

Do you ever want a demo or sample dataset that doesn't bore you to death? How about one steaming with sex, murder, and mayhem?

I'll be giving a lightning talk on HTSQL at PyPGDay this Wednesday, and wanted to show it off with some data worthy of its awesomeness. How about Shakespeare? Yeah! Luckily, Open Source Shakespeare has published a database of all Will's works. Unluckily, they've only published it as flat text files and as a Microsoft Access database. ("Open Source" Shakespeare? In a closed-source database? And a horrible one at that? Yeah, I know.)

So I fixed that; opensourceshakespeare on GitHub is a port of the opensourceshakespeare.org data to the RDBMS the Bard himself would have used, PostgreSQL. Porting further to MySQL and SQLite is left as an exercise for the reader (for now; maybe I'll add those after PyCon.)

Enjoy! And if you're ever in Cincinnati, you have GOT to see the Cincinnati Shakespeare Company. They're like... they're like actually eating the food, when all you've done before is look at the recipes.

Dayton Python Workshop, April 5-6

Announcing the first Dayton Python Workshop for women and their friends, April 5-6, 2013!

tinyurl.com/day-py-workshop

The Workshop is a free, friendly, hands-on, beginners' introduction to computer programming. In one short weekend (a Friday evening and a Saturday), participants get a real handle on programming for practical tasks, using the easy yet powerful Python language... while having a fun time with new friends!

The primary target audience is women of all ages and backgrounds, including those who have never programmed before. Men can participate as the guest of a female attendee - that's where the "women and their friends" part comes in. If you're a man who wants to take part, ask a woman you know to join you. Don't underestimate the power of a personal invitation - you'll never know which of your friends just needs a nudge to try it until you give her that nudge.

The workshop is the latest in a series based on the famous Boston Python Workshop; they've already introduced hundreds of beginners to programming in Boston, Columbus, Indianapolis, Portland, Chicago, and Kansas City. It's designed for true beginners to programming - absolutely no programming or computer experience is assumed. (Even experienced programmers have enjoyed the workshop, though - so long as they're new to Python!)

We'll enjoy food and a great venue at New Horizons in Fairborn.

Space is limited, so please sign up at tinyurl.com/day-py-workshop

New Horizons: newhorizons.com

Hooray for Columbus Python Workshop 1!

The first Columbus Python Workshop for women and their friends was a success!

We had about 18 students from all sorts of backgrounds, plus TAs Alyssa and Ryan from the Central Ohio Python User Group. They dived in with a will and learned an amazing amount in a short time. We all had fun and I loved having them there... there's something uniquely wonderful about teaching to faces that are smiling at you. By the end, we had several nice variants on the Color Wall.

Food sponsorship by LeadingEdje kept us all nicely fed and caffeinated, and Pillar provided The Forge, its new creative workspace in the Short North.

Over lunch on Saturday, COHPy members Ann Elliot and Thomas Winningham stopped in to show samples of some of the seriously fun things our Python programmers can look forward to as their skills increase. Ann showed some awesome astronomical charts she'd made with PyEphem, and Thomas used Python and an Arduino to make a string of multicolor LEDs do tricks. (With a little forethought, we could have married up the students' ColorWall scripts to Thomas's lights... maybe next time!)

For a first follow-up, some of us worked together at GirlDevelopIt Columbus's Hack Night:

... and we're expecting to see a bunch more at the next Central Ohio Python User Group meeting on Monday.

My experiment with running the exercises through IPython Doctester rather than CodingBat was... partially successful. Many students had to fall back on CodingBat thanks to trouble installing pyzmq on Windows. So I know some improvements to make for next time around.

To get involved with our next workshop in Columbus, sign up for the Central Ohio Python User Group. The next Ohio workshop, though, will be in Dayton... home sweet home at last!

If you came to the workshop, I would love for you to stay in touch, to ask questions or for any other reason. Thanks to all the students, the sponsors, and to the Boston Python Workshop for leading the way!

PyPgDay

I was thinking about staying home from PyCon in 2013.

I know, that's a horrible idea. But financing the trip has gotten hard recently, and I want to spend weeks and weeks in Montreal for PyCon 2014, so I was thinking maybe I'd save up my travel money for a spectacular trip to Quebec in a year.

But now look what they've done. PyPgDay. Postgres and Python. They've got my number. Resistance is futile.

The CFP is out; I'll be thinking about what I can propose. Maybe this will lead to a spasm of EDD (embarrassment-driven development) on sqlpython and cmd2.

So keep close watch over your kidneys, because I may resort to stealing them to make the trip, but I'll see you in Santa Clara.

ipython_doctester 0.2

The big improvement for 0.2 is that ipython_doctester now optionally reports test results to an instructor's webapp, http://ipython-docent.appspot.com. In a classroom setting, this helps a teacher see who needs more time or help.

To turn on reporting, at the beginning of the IPython Notebook session,

import ipython_doctester
ipython_doctester.workshop_name = 'demo1'
ipython_doctester.student_name = 'Catherine'
from ipython_doctester import test

The instructor should run through the notebook first, executing all cells just to register each of the function names in order. Then she can check on the class's progress at http://ipython-docent.appspot.com/workshop_name.

You can also optionally set ipython_doctester.verbose = True to get the full table of results for every exercise, even the successes.

I plan to use this for the Columbus Python Workshop on Jan 18-19. Anyone you know from central Ohio needs to hear about this class!

ipython_doctester

That IPython project I was kicking around has started to mature, and I've released it into PyPI:

ipython_doctester

I'm much happier with using it as a decorator, it stays out of the student's way that way:

GiveCamp

I really enjoyed Southwest Ohio GiveCamp this year. There are lots of community events where I learn stuff and have fun, but it's especially nice when we can also look back on something good done for a good cause.

I was on a small team that redid URS Dayton's website into a WordPress site. (The new version isn't online yet, so don't go clicking today and saying, "Ew, you're proud of that?") I'll say this for WordPress: despite having no genuinely experienced WordPress users on the team, we made a very presentable site in just an evening and a day. There was even enough time left last night for me to jump into another project for an hour - and being able to contribute a little bit within an hour of jumping into an unknown project says something about just how easy GitHub makes collaboration.

It's interesting: nothing stops any of us from volunteering for any of those nonprofits anytime, but something about gathering for an intense push generates enthusiasm and momentum in a way that scattered efforts rarely do. I guess that's why conference sprints exist, too. This being Sunday, it occurs to me that most churches have pitiable websites; I wonder if churchgoing geeks could do a gathering to fix some of those up. Call it a "Sprint to Emmaus" or something.

Anyway, the broadening is nice and all, but someday I'd like to be on a Pythonista team at GiveCamp. I think it would need to be preceded by an effort to convince a requesting nonprofit to ask for a Python-based project. Nonprofits always come to SWOGC requesting a specific technology, almost always something from the PHP or Microsoft universes, simply because that's what they know about. Hmm, I wonder if it would be unethical to say, "For your project, we've used a special cutting-edge version of WordPress called 'Django'..."

A great weekend. Thanks to Andrew Douglas, my team leader, and to the GiveCamp organizers!

Columbus Python Workshop, Jan 18-19

Happy Ada Lovelace Day!

Announcing the first

Columbus Python Workshop

for women and their friends

Jan. 18-19, 2013

The Columbus Python Workshop for women and their friends is a free hands-on introduction to computer programming that's fun, accessible, and practical even to those who've never programmed at all before. We empower women of all ages and backgrounds to learn programming in a beginner-friendly environment.

Thanks to Pillar Technologies for hosting the workshop in their brand-new office in Columbus' Short North!

The workshop is the latest in a series based on the famous Boston Python Workshop; they've already introduced hundreds of beginners to programming in Boston, Indianapolis, Portland, Chicago, and Kansas City. Now it's Ohio's turn, so spread the word!

Get more details and sign up now: https://openhatch.org/wiki/Columbus_Python_Workshop

Traditionally, Ada Lovelace Day is celebrated by highlighting the achievements of present and past women in science and technology. Today, instead, I'm saluting the contributors of the future!

IPython Notebook tutorial with doctest feedback

I'm increasingly amazed at IPython Notebook, and want to use it for an interactive tutorial. I'd like the notebook to be full of exercises for the student to fill out, and to get feedback from the notebook itself - a lot like CodingBat but in IPython Notebook format.

Here's the code: ipython_doctester

Next step - maybe - would be to finish with a scorecard cell summarizing the student's overall progress.

And then... automatically push data on a student's progress up to a webserver on the instructor's machine? Which presents a dashboard showing her where everybody is and what everybody is struggling with...

Like every open-source author, I'm eager for feedback!

This dip into IPython Notebook reminds me of people who suggested years ago that I should integrate SQLPython into IPy. Indeed, writing a SQL-handling extension for IPy sounds really attractive... I wonder how many insomniac nights it would take...

doctesting just one function

There's no such function as doctest.testfunc. Or doctest.testclass, or, for that matter, doctest.testobj. However, you can run the doctests on a single function or class with doctest.run_docstring_examples, like so:

doctest.run_docstring_examples(myfunction, globals=globals())

"That's OK", I thought, "My function doesn't call any globals, so I'll just pass in an empty dictionary." Wrong. run_doctstring_examples actually needs your globals to find the object you're testing; you'll get

    NameError: name 'myfunction' is not defined

The docs say the function is "rarely useful", but I'm finding it very useful for an IPython Notebook-based interactive tutorial I'm hoping to work up shortly!

Indianapolis Python Workshop for women and their friends - the results

This April 13-14, Mel Chua, Jessica McKellar, and I ran the first annual Indianapolis Python Workshop for women and their friends... and I loved it!

We borrowed our curriculum (not to mention Jessica) from the Boston workshop, and the time they've put into fine-tuning it really shows. The participants varied from absolute first-time programmers to computer science graduate students, but the materials did a great job of not leaving anybody behind.

I chatted with a few of the participants about their backgrounds and motivations for coming, and I was fascinated by the diversity among them. One woman was re-entering the programming workforce after a decade away. One was an artist and designer of 3D games who wanted to learn to script her favorite gaming engine. One was the business manager for an IT firm who wanted a deeper understanding of the nuts and bolts of her own company. And, of course, several were students, from high school through graduate school.

We had roughly 20 participants, plus assistants from IndyPy, and the classroom dynamics were great. Usually, when running an IT event, I strain to convince the participants on to work together. Far too many programmers tend to struggle along in private silence, each at their own machine, which ruins the point of having a group learning event in the first place. Not our workshop students, though! They did a wonderful job of grouping up and helping each other out. I loved the sound of students teaching each other and laughing with each other. We had lots to teach and we worked them hard, but they met the challenge with enthusiasm and teamwork. I'm really looking forward to seeing them again at IndyPy's project night on May 8.

We have many thanks to give! Thanks to the Python Software Foundation and FewerHassles for their financial support; to DirectEmployers for their logistical help; to Indiana LinuxFest for providing a locale; and to the Boston Python Workshop for sending Jessica and all her experience. Mel and I are looking forward to keeping the ball rolling with more workshops in Indianapolis and elsewhere in the Midwest. Please contact <mpw-staff at lists.openhatch.org> if you'd like to help make it happen.

Indianapolis Python Workshop for women and their friends

Mel Chua, I, the team from the Boston Python Workshop, and IndyPy are all working together to bring about the Indianapolis Python Workshop for women and their friends in just two weeks!

There are several great things about this style of workshop:

- It's designed for beginners. No, really. I've taught what I thought of as "introductions" before, but like most non-beginners, I'm way too prone to zoom ahead and forget what it's really like for beginners. The BPW staff has worked hard to overcome this, and put together an amazing curriculum that really covers what beginners need in detail.

- The participants will be mostly (but not exclusively) women. There is an awkwardness about being the "only girl in the room" which, when heaped on top of garden-variety beginners' nervousness, can make people decide to just stay home... or make them nervous as heck if they do come. (There's nothing like feeling like the reputation of all women everywhere rest on your n00bish shoulders...) The quickest way to short-circuit that problem is to make sure that there are plenty of other girls in the room!

- It's very hands-on, concentrating on getting people using and enjoying code in a short time.

- It's designed to help participants plug immediately into using and enjoying their local developers' community, moving from the workshop into cooperative Project Nights with their local user group. In fact, this workshop takes place at Indiana LinuxFest, so their first exposure to the user community is right outside the door.

Please tell the Midwesterners you know. And if your company is excited about this, too, talk to me about sponsorship opportunities!

PyCon 2012

Best. PyCon. Ever.

And since PyCon is like my Christmas (in the 5-year-old sense: "I get new toys to play with!"), that's extremely high praise.

This isn't a proper retrospective - don't know if I'll have time to write one - but I just wanted to give thanks and glory to Jesse Noller, the entire volunteer staff (from whom I was notably absent), the proud ranks of the sponsors, and the awesome participants (the word is definitely "participant", not "attendee"). If you were wondering if PyCon's magic could scale, be reassured: the rainbows only proliferate.

PyOhio CFP should be out within a couple weeks, too.

SENDING TO EVERYONE. PASS IT ON.

I think I need to create a script that

1. Scans my gmail inbox for any mail whose subject line uses ALL CAPS and which urges me to forward it to everyone I know;
2. Searches snopes.com for the message body;
3. Replies with a link to the top Snopes hit and its text, plus a preachy little sermon by me on the importance of Truth

... all automatically.

What sort of tools can I use to make such a script?

If you don't know, please forward my question on to everyone you know. It's very important. It probably involves President Obama and/or dead babies.

slides from Saturday class

Hi, Python students from Saturday! Hope you enjoyed the class, and thanks for your patience as we struggled with wireless issues.

Here's the updated class materials I promised - if you can, install Visual Python on your machine and try out some of the suggestions given in the comments at the bottom of move.py and gravity.py.

I'm very fortunate to be able to go to PyCon this year, and I plan to come to the Dayton Dynamic Languages meeting the following Wednesday (March 14) bubbling over with stuff I learn there - I'd love to see you there! Bring whatever Python project you've been working on and we'll help push you over any bumps you've hit, or strategize on ideas for new projects.