SacredPy seeking collaborators

I'm looking for collaborators who want to build web programming experience on an interesting project...

During my job search, I was contacted by Kai Schraml, a seminary graduate who wants to scratch an itch. Seminarians have a serious need to discuss, debate, and seek consensus on the translations of difficult texts, like sacred scriptures. But the software tools currently available for the purpose are closed-source and expensive. That just seems wrong - not just because seminary students are broke, but because of the nature of the texts themselves. After all, Jesus released his teachings under a very strong open-source license!*

So we're starting to work on an alternative, provisionally called "SacredPy". (It could be applied to any difficult texts, of course, so if Beowulf is sacred to you, have at it.) I'm quite employed now, but I'm dabbling at it a bit for the sheer interest and open-sourcey glory of it all. It's possible income could eventually come from this project - Kai could tell you more about the prospects - but certainly not soon, so this is no substitute for proper employment. But it might be great resume builder for a new Python programmer. It looks like we'll most likely build something atop Askbot, a Django-based project, so if you'd like to move into the thriving "experienced Djano developer" segment of the economy...

Let me know at moc.liamg@nilved.enirehtac and we'll talk!

* - Matthew 10:8 - δωρεὰν ἐλάβετε, δωρεὰν δότε ("Freely you have received, freely give")

Presentation links

I have just had a VERY. Busy. Week. (In a good way!) I've promised the world many talk materials, so:

• At Ohio LinuxFest, IPython for non-Pythonistas
• At APCUG Regional Conference, Python
• At Postgres Open, IPython: your new SQL client

As for Postgres Open, I absolutely loved it! So happy I finally got to go. I am proud to say that I was the very first to buy my admission for 2014! Hope to blog more about that later...

I'm not available

I'm happy to say that I'll shortly be starting a new position as a PostgreSQL DBA and Python developer for Zoro Tools!

We software types seem to have hardware envy sometimes. We have "builds" and "engines" and "forges" and "factory functions". But as it turns out, the "Tools" in "Zoro Tools" isn't a metaphor for cleverly arranged bytes. That's right - they're talking about the physical objects in your garage! Imagine! Lucky for me the interviewers didn't ask to review my junior high shop project.

So disregard my earlier post about being available. Thanks for all your well-wishes!

Depending on how you reckon it, my job search arguably only took forty minutes, though it took a while for gears to grind and finalize everything. Years of building relationships at PyCon made this the best job search ever; the only unpleasant part was having to choose from among the opportunities to work with my favorite technologies and people. I'm very glad I made the investment in PyCon over the years... and if you're thinking "that's easy for you to say, I can't afford it", don't forget PyCon's financial aid program.

And speaking of conferences, I'll be at Postgres Open next month (my first one!) - hope to see some of you there!

IPython at Ohio LinuxFest 2013

Are you signed up yet for Ohio LinuxFest on Sep. 13-15? I'll be there to present

IPython for non-Pythonistas

Break out of your (bash) shell! IPython and the IPython Notebook have swept over the Python programming community, but they're not just for Python programmers - they make for high-powered shell replacements even with little to no Python knowledge. They'll also let you document your work and collaborate with others like never before. Find out how these beautiful tools can improve your daily Linux work!

At PyOhio, I argued that all Python programmers need IPython. At OLF, I'll make the case that non-Pythonistas need IPython, too. Perhaps my next talk will be "Even Your Cat Needs IPython".

Also at OLF, look for PyOhio's booth for info on next year's PyOhio, other Python events around the region, and general Python love!

Python Workshop for Women Indy #2 and CMH #2 coming up!

The Midwest Python Workshop for women and their friends is back! We've got new workshops scheduled, ready to take new batches of students:

Indianapolis Python Workshop, Sep. 27-28, 2013; sponsored by Six Feet Up and hosted at Launch Fishers

Columbus Python Workshop, Oct. 18-19, 2013; sponsored by LeadingEdje and hosted at The Forge by Pillar

The Workshop is a free, friendly, hands-on introduction to computer programming using Python. Women of all ages and backgrounds are the primary target (but you can bring a male participant as your guest).

Please spread the word!

I'm available

I'm available for hire! If you need a database expert with lots of programming skill, or a Python programmer with deep database experience, please check out:

Resume / portfolio

But: you must be telecommute-friendly, or in the Dayton area. I'm sorry, but I'm not available to relocate.

IPython %helloworld extension

At Monday's after-PyOhio sprint, I changed ipython-sql from an IPython Plugin to an Extension; this makes it compatible with IPython 1.0. Fortunately, this was really easy; mostly I just deleted Plugin code I didn't understand anyway.

But I do feel like "Writing Extensions" docs are lacking a "Hello World" example. Here's mine.

from IPython.core.magic import Magics, magics_class, line_magic, cell_magic

@magics_class
class HelloWorldMagics(Magics):
    """A simple Hello, <name> magic.
   
    """

    @line_magic  # or ``@line_magic("hi")`` to make ``%hi`` the name of the magic
    @cell_magic  
    def helloworld(self, line='', cell=None):
        """Virtually empty magic for demonstration purposes.

        Example::
        
          In [1]: %load_ext helloworld

          In [2]: %helloworld Catherine
          Out[2]: u'Hello, Catherine'
        

        """
        return "Hello, %s\n%s" % (line, cell or "")
       
def load_ipython_extension(ip):
    ip.register_magics(HelloWorldMagics)

PyOhio Stone Soup

Loved PyOhio once again! Thanks so much to everybody who came, participated, and made it happen! I get such a rush of joy from seeing the Ohio Union fill up with happy Pythonistas.

PyOhio has been a classic case of the Stone Soup story. When we started planning the first one, we really didn't have the resources to pull off a conference; we were just a handful of PyCon 2008 attendees who wanted to bring something like PyCon home. But as we put it together, people appeared, pitched in, and we had a modest, amateurish - but fun! - little conference in the Columbus Public Library. PyOhio 2008 drew participants and volunteers who helped make PyOhio 2009 bigger and better; 2009 drew in more involvement for 2010; and so forth, year after year.

July 26-27, 2014. See you in Columbus!

The IPython Notebook Revolution

Among the many great talks coming to PyOhio at the end of this month:

The IPython Notebook Revolution Catherine Devlin If you think of IPython as simply an enhanced version of the live Python prompt, you are in need of re-education. No matter what you do with Python, applying the IPython Notebook imaginatively will revolutionize the way you do it.

I'd like to focus on aspects of IPython outside the traditional number-crunching, plot-making realm, simply because those have been covered so well already - videos by the actual IPython team already have. I'd like to fill up a talk with edgy, imaginative, experimental uses of IPython that aren't well-known yet, or that suggest new ways IPython (and especially the Notebook) may be used in the future. I have a bunch of ideas along those lines...

... but I'd like your input! I don't want to miss anything awesome just because I wasn't aware, and there's a lot being done in the IPython world - more than I've been able to keep track of. Erik Welch has already thoughtfully given me a bunch of links and suggestions from SciPy. Let's crowdsource my talk even further!

Some of the goodies I already plan to include:

• notebook-based presentations
• ipython_blocks: probably my Holy Grail of imaginative uses)
• d3js in IPython: (OK, this still fits the data graphing theme, but it's also ultra-snazzy)
• ipython_sql: (everybody's got to toot her own horn sometimes)
• ipfl (web-style forms in a Notebook - very preliminary but an interesting idea)
• xkcd and hand-drawn mode
• Wakari

How would you shake up people's notions of "what IPython is for"?

Easy HTML output in IPython Notebook

If any object has a _repr_html_ method, the IPython Notebook will use it to render HTML output. It's really easy to make a simple class that permits general dynamic HTML-rich output with Markdown. Markdown is a superset of HTML, so HTML in your output string will work, too.

import markdown
class MD(str):
    def _repr_html_(self):
        return markdown.markdown(self)

Four little lines, and you can do this!

ipython-sql bind variables

Thanks to Mike Wilson, ipython-sql now supports bind variables!

In [12]: name = 'Countess'

In [13]: %sql select description from character where charname = :name
Out[13]: [(u'mother to Bertram',)]

Review of Learning IPython for Interactive Computing and Data Visualization

valuable but traditional

May 25, 2013 by Catherine Devlinproduct

Learning IPython for Interactive Computing and Data Visualization

4 stars (of 5)

Packt Publishing recently asked if I could review their new title, Learning IPython for Interactive Computing and Data Visualization. (I got the e-book free for doing the review, but they don't put any conditions on what I say about it.) I don't often do reviews like that, but I couldn't pass one this up because I'm so excited about the IPython Notebook.

It's a mini title, but it does contain a lot of information I was very pleased to see. First and foremost, this is the first book to focus on the IPython Notebook. That's huge. Also:

• The installation section is thorough and goes well beyond the obvious, discussing options like using prepackaged all-in-one Python distributions like Anaconda.
• Some of the improvements IPython can make to a programming workflow are nicely introduced, like the ease of debugging, source code inspection, and profiling with the appropriate magics.
• The section on writing new IPython extensions is extremely valuable - it contains more complete examples than the official documentation does and would have saved me lots of time and excess code if I'd had it when I was writing ipython-sql.
• There are introductions to all the classic uses that scientists doing numerical simulations value IPython for: convenience in array handling, Pandas integration, plotting, parallel computing, image processing, Cython for faster CPU-bound operations, etc. The book makes no claim to go deeply into any of these, but it gives introductory examples that at least give an idea of how the problems are approached and why IPython excels at them.

So what don't I like? Well, I wish for more. It's not fair to ask for more bulk in a small book that was brought to market swiftly, but I can wish for a more forward-looking, imaginative treatment. The IPython Notebook is ready to go far beyond IPython's traditional core usership in the SciPy community, but this book doesn't really make that pitch. It only touches lightly on how easily and beautifully IPython can replace shell scripting. It doesn't get much into the unexplored possibilities that IPython Notebook's rich display capabilities open up. (I'm thinking of IPython Blocks as a great example of things we can do with IPython Notebook that we never imagined at first glance). This book is a good introduction to IPython's uses as traditionally understood, but it's not the manifesto for the upcoming IPython Notebook Revolution.

The power of hybrid documentation/programs for learning and individual and group productivity is one more of IPython Notebook's emerging possibilities that this book only mentions in passing, and passes up a great chance to demonstrate. The sample code is downloadable as IPython Notebook .ipynb files, but the bare code is alone in the cells, with no use of Markdown cells to annotate or clarify. Perhaps this is just because Packt was afraid that more complete Notebook files would be pirated, but it's a shame.

Overall, this is a short book that achieves its modest goal: a technical introduction to IPython in its traditional uses. You should get it, because IPython Notebook is too important to sit around waiting for the ultimate book - you should be using the Notebook today. But save space on your bookshelf for future books, because there's much more to be said on the topic, some of which hasn't even been imagined yet.

This hReview brought to you by the hReview Creator.

cannot import name MAXREPEAT

When I upgraded from Xubuntu 12.10 to 13.04 today, all my existing Python virtualenvs broke! Fortunately, they're just virtualenvs and easy to replace (that's kind of the point). But don't panic if you start seeing these.

$ ipython
Traceback (most recent call last):
  File "/home/catherine/ve/e2/bin/ipython", line 5, in 
    from pkg_resources import load_entry_point
  File "build/bdist.linux-i686/egg/pkg_resources.py", line 16, in 
  File "/home/catherine/ve/e2/lib/python2.7/re.py", line 105, in 
    import sre_compile
  File "/home/catherine/ve/e2/lib/python2.7/sre_compile.py", line 14, in 
    import sre_parse
  File "/home/catherine/ve/e2/lib/python2.7/sre_parse.py", line 17, in 
    from sre_constants import *
  File "/home/catherine/ve/e2/lib/python2.7/sre_constants.py", line 18, in 
    from _sre import MAXREPEAT
ImportError: cannot import name MAXREPEAT

Apparently Python 2.7.4 introduces _sre.MAXREPEAT. Here it is in my (new) system Python, 2.7.4:

Python 2.7.4 (default, Apr 19 2013, 18:28:01) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import _sre
>>> _sre.MAXREPEAT
4294967295L 

... but the virtualenvs I created before the upgrade still use Python 2.7.3

Python 2.7.3 (default, Sep 26 2012, 21:51:14) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import _sre
>>> _sre.MAXREPEAT
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'module' object has no attribute 'MAXREPEAT'

If I were a deeper hacker I'd try to figure out why code running within my old virtualenvs is trying to access a 2.7.4-only attribute, or what's the most efficient way to recover my old virtualenvs. But I'll settle for recognizing the problem and spinning up new virtualenvs instead. That fixes the problem.

I know... I could have avoided this problem by using Python 3! I've got code that depends on fabric, though, which still isn't available for 3.

Speak at PyOhio

Have you responded yet to PyOhio's Call For Proposals (due date: June 1)? You should. Here's why.

Why you should speak at PyOhio

We need you. We have a great group of people who contribute talks, but we don't ever want that group to become a stagnant pool - we want to always be drawing in new contributors with a variety of backgrounds, perspectives, and interests. Unless your clone sibling just submitted a talk, we need your contribution!

Meet people. You'll always meet people at PyOhio, of course, but speaking makes you a beacon to the other people who share your interests.

Practice. We all know public speaking is an important skill, but it's hard to muster the courage to get real-life practice in it. PyOhio is perfect because of the friendliness of the Python community - everybody there wants you to succeed. Hone your skills with us, among friends... you can go speak at snotty places later!

Push yourself. That skill you've been wanting to learn, or that package you know you need to polish up for release? Committing to speak on it is a surefire way to break the curse of "haven't quite gotten around to it"! Sometimes we call this "embarrassment-driven development".

But I'm not an expert

Perfect. Because the great curse of expert teachers is that they can forget what it was like to be a beginner. So dive into something you want to learn, take careful notes as you go about what confused you and how you resolved it, and you'll blaze a trail that you can guide other beginners along. Your non-expert perspective will make you a great teacher!

Team up

You can draw on that friendly community to help you present, too! Share a presentation with somebody more or less experienced to make an expert-beginner duet, or have a friend cover an aspect of your topic that they know better. Get a friend to review your talk as you develop it. Shop your ideas around your local Python usergroup and see what suggestions they have.

Many ways to contribute

PyOhio is not all about talks, of course (for me, the talks are kind of the excuse we use to get together and do the other stuff.) Also consider proposing something like

• a tutorial
• an Open Space
• a Sprint
• a Lightning Talk (actually, you propose these on-the-spot, but you can get it ready in advance)

Thank you, and spread the word!

ipython-sql for multi-database comparisons

For my newest ipython-sql trick, I needed to compare some queries run across different databases. How hard would it be to get side-by-side results into tidy IPython Notebook output?

Not hard at all, it turns out, if you're willing to violate basic principles of human decency.

That's an itty-bitty image, so here's the crazy part:

class SQL_Comparitor(object):
    def __init__(self, *connection_strings):
        self.connection_strings = connection_strings
    def run(self, qry):
        dframes = []
        for connection_string in self.connection_strings:
            result = %sql $connection_string $qry

Did you catch that? I used %sql magic and IPython variable substitution inside an instance method. It feels so wrong! But it works! Provided you're running within IPython, of course; normal Python will not under any circumstances run an unholy perlish abomination like this. I'm just really amazed that we can use IPython tricks inside class definitions, but it's real.

Since the result is a Pandas DataFrame, it's easy to apply transformations. For instance, say you only want the rows where the values are different:

diff = results[results['svr1/db1_Value'] != results['svr2/db2_Value']]
HTML(diff.to_html())

I'm not sure how to distribute this class, since it's small and it's not actually valid Python, just valid IPython. For now I've made a gist (and its nbviewer version).

Linux Installfest, Dayton, Sat April 13

Dayton folks - are you coming to the Dayton Linux User Group's Installfest tomorrow?

The main point of our Installfests is to get everyone together for mutual help setting up and configuring computers and programs. But we like to throw in some educational talks, too, and I volunteered to give two:

Corrupting the Youth: a survey of programming environments for kids

Several open-source projects have been created to help introduce kids to computer programming by creating programming environments more intuitive and fun than simply writing code at a text editor. We'll take a quick look at several of them, some suitable for teens and others for kids barely into elementary school.

Introduction to Object-Oriented Programming Concepts

Object-oriented is the most common programming paradigm in use today, but some people who've only programmed in procedural languages find object-oriented terms and concepts mysterious and intimidating. We'll take the mystery out by explaining the motivations and fundamental techniques of OO programming with some easy-to-understand examples.

Hope to see some of you there!

%sql to Pandas

After getting %sql magic for IPython working, my next big goal was to figure out how to get those results into Pandas.

Er, OK, not such a big goal. Even with zero Pandas experience, it took about five minutes of skimming the first page of documentation to figure out:

In [1]: %load_ext sql

In [2]: data = %sql postgresql://will:longliveliz@localhost/shakes select * from work

In [3]: import pandas as pd

In [4]: s = pd.DataFrame.from_records(data, columns=data.keys)

This is not the only way to move data from an RDBMS to Pandas (there's pandas.io.sql, for example), and I don't know enough about Pandas to know if it's the best way. But I bet it's the easiest way.

released: %sql magic for IPython

Inspired and informed by discussions with the IPython developers at PyCon 2013, I've released ipython-sql, a %sql magic for IPython.

With this, I really think the IPython Notebook will become the most amazing database tool ever. In fact, virtually every computing problem will become a lot more workable when manipulated via the IPython Notebook - you can remember, inspect, and annotate all the steps to investigate whatever your issue is. All hail the Notebook! The FSF really chose well in choosing Fernando Perez for their 2012 award.

The Canadian menace

Incidentally, some people have been asking, "Wait a minute - PyCon-US in Canada? How does that work? Wouldn't it be more correct to call it PyCon-NA for North America?"

It might, if this were a case of nations cooperating to share PyCon. However, that is not the case. You will notice that the Canadians haven't cancelled their own PyCon. Rather, they have seized PyCon-US by ruthless volunteerism and are even now dragging it off to their stronghold on the St. Lawrence, to hoard it along with the PyCon they already have. That's right, they want ALL THE PYCONS.

Pythonistas of the world, be warned! When friendly faces from the North arrive to lend a hand, watch them carefully! Or you'll soon be flying to Vancouver for PyAr and Winnipeg for EuroPython.

Just to clarify: I am giddy over having PyCon in Montreal. I'm so excited that they'll probably need to name a Montreal Syndrome to go along with Jerusalem Syndrome and Paris Syndrome.

post-PyCon post

You might be sick of me saying after each PyCon, "That was the best PyCon ever!", but it's not my fault if it's true.

I hardly know where to start summing up the highlights...

• PyPGDay was a great addition! I've had virtually no exposure to the PostgreSQL community before, so this was very valuable to me. Evan Klitzke from Uber gave a talk on migrating to PG from MySQL that is going to save me a ton of time, and Jeff Davis from Aster showed off the huge usefulness of range types.
• Naomi had to talk me into the Education Summit, but I'm glad she did - I got a lot of great ideas and inspiration that will help in teaching future workshops.
• One of these ideas was the use of Matt Davis' fantastic ipythonblocks, which will let us do graphical exercises right within the IPython Notebook - an amazing, intuitive, seamless learning experience.
• Speaking of the IPython Notebook, this PyCon was really its tour de force. Seemed like everyone was using it to do and show just amazing things. If you haven't seen some talks on it yet, go watch a bunch of videos immediately and then watch a bunch more when the PyCon 2013 talks are online - the docs alone can't do justice to the possibilities the Notebook creates. We've only begun to take advantage of this fantastic environment.
• I mentioned my ambition to create an IPython-based SQL client to Fernando Perez from the IPython team, and he jumped to show me what I need to know. The day after coming home, I checked in a %%sql magic. It's not ready for prime time yet (or even PyPi) and it may warrant merging into a similar project, but it was a delight to play off the capability of the Notebook.
• Peter Wang and Travis Oliphant showed me - personally! - Wakari, an amazingly powerful online hosted Python environment. I can't wait to play.
• I'm not going to list all the people I loved seeing and catching up with, because it wouldn't mean much to most blog readers. But the fact is it would be a lonely year if I couldn't see my PyCon friends (and make some new ones).
• If GitHub had an AI, it would be looking at me funny and asking, "What's got into you lately?" PyCon, that's what. And I'm nowhere near done.

Thank you to the fantastic bunch of volunteers who make this such an amazing conference, and to all the participants who bring their ideas and their friendship.

HTSQL lightning talk slides

I posted the slideshow from my PyPGDay HTSQL lightning talk here. Thanks to everybody involved with PyPGDay, I loved it!

Forsooth, a dataset

Do you ever want a demo or sample dataset that doesn't bore you to death? How about one steaming with sex, murder, and mayhem?

I'll be giving a lightning talk on HTSQL at PyPGDay this Wednesday, and wanted to show it off with some data worthy of its awesomeness. How about Shakespeare? Yeah! Luckily, Open Source Shakespeare has published a database of all Will's works. Unluckily, they've only published it as flat text files and as a Microsoft Access database. ("Open Source" Shakespeare? In a closed-source database? And a horrible one at that? Yeah, I know.)

So I fixed that; opensourceshakespeare on GitHub is a port of the opensourceshakespeare.org data to the RDBMS the Bard himself would have used, PostgreSQL. Porting further to MySQL and SQLite is left as an exercise for the reader (for now; maybe I'll add those after PyCon.)

Enjoy! And if you're ever in Cincinnati, you have GOT to see the Cincinnati Shakespeare Company. They're like... they're like actually eating the food, when all you've done before is look at the recipes.

Dayton Python Workshop, April 5-6

Announcing the first Dayton Python Workshop for women and their friends, April 5-6, 2013!

tinyurl.com/day-py-workshop

The Workshop is a free, friendly, hands-on, beginners' introduction to computer programming. In one short weekend (a Friday evening and a Saturday), participants get a real handle on programming for practical tasks, using the easy yet powerful Python language... while having a fun time with new friends!

The primary target audience is women of all ages and backgrounds, including those who have never programmed before. Men can participate as the guest of a female attendee - that's where the "women and their friends" part comes in. If you're a man who wants to take part, ask a woman you know to join you. Don't underestimate the power of a personal invitation - you'll never know which of your friends just needs a nudge to try it until you give her that nudge.

The workshop is the latest in a series based on the famous Boston Python Workshop; they've already introduced hundreds of beginners to programming in Boston, Columbus, Indianapolis, Portland, Chicago, and Kansas City. It's designed for true beginners to programming - absolutely no programming or computer experience is assumed. (Even experienced programmers have enjoyed the workshop, though - so long as they're new to Python!)

We'll enjoy food and a great venue at New Horizons in Fairborn.

Space is limited, so please sign up at tinyurl.com/day-py-workshop

New Horizons: newhorizons.com

Hooray for Columbus Python Workshop 1!

The first Columbus Python Workshop for women and their friends was a success!

We had about 18 students from all sorts of backgrounds, plus TAs Alyssa and Ryan from the Central Ohio Python User Group. They dived in with a will and learned an amazing amount in a short time. We all had fun and I loved having them there... there's something uniquely wonderful about teaching to faces that are smiling at you. By the end, we had several nice variants on the Color Wall.

Food sponsorship by LeadingEdje kept us all nicely fed and caffeinated, and Pillar provided The Forge, its new creative workspace in the Short North.

Over lunch on Saturday, COHPy members Ann Elliot and Thomas Winningham stopped in to show samples of some of the seriously fun things our Python programmers can look forward to as their skills increase. Ann showed some awesome astronomical charts she'd made with PyEphem, and Thomas used Python and an Arduino to make a string of multicolor LEDs do tricks. (With a little forethought, we could have married up the students' ColorWall scripts to Thomas's lights... maybe next time!)

For a first follow-up, some of us worked together at GirlDevelopIt Columbus's Hack Night:

... and we're expecting to see a bunch more at the next Central Ohio Python User Group meeting on Monday.

My experiment with running the exercises through IPython Doctester rather than CodingBat was... partially successful. Many students had to fall back on CodingBat thanks to trouble installing pyzmq on Windows. So I know some improvements to make for next time around.

To get involved with our next workshop in Columbus, sign up for the Central Ohio Python User Group. The next Ohio workshop, though, will be in Dayton... home sweet home at last!

If you came to the workshop, I would love for you to stay in touch, to ask questions or for any other reason. Thanks to all the students, the sponsors, and to the Boston Python Workshop for leading the way!