tag:blogger.com,1999:blog-118022922024-03-13T11:41:01.985-07:00Catherine: pyOraGeekDatabases taste better with Python.Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.comBlogger359125tag:blogger.com,1999:blog-11802292.post-7226750995753631012015-11-13T09:42:00.001-08:002015-11-13T10:05:38.077-08:00Wanted: RDBMS superpower summary for app developers
<p>At last night's <a href="http://www.meetup.com/WWCode-Cincinnati/">WWCode Cincinnati</a> panel, I recommended that developers talk to their DBA about what advanced capabilities their RDBMS can offer, so that they don't end up reimplementing functionality in the app that are already available (better and more efficiently) in the database itself. Devs can waste a lot of effort by thinking of databases as dumb, inert data boxes.</p>
<p>I was asked an excellent question: "Where can a dev quickly familiarize herself with what those capabilities are?" My answer was, "Um."</p>
<p><em>Do not</em> say they should read the docs. That is a "let them eat cake" answer. The <a href="http://www.postgresql.org/docs/">PostgreSQL docs</a> are over 2900 pages. That's not what they need.</p>
<p>Suggestions, folks? Python developers have built great summary sites, like the <a href="http://docs.python-guide.org/en/latest/">Hitchhiker's Guide to Python</a>. What are the equivalents in the database world? Do they exist? Do we need to write them?</p>
Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com0tag:blogger.com,1999:blog-11802292.post-13999121202420040912015-07-10T15:44:00.000-07:002015-07-10T16:08:15.891-07:00Code Studio rocks; diversity does, too<p>If you want to quickly get some kids introduced to computer programming concepts,
you could do a lot worse than using <a href="https://studio.code.org/">Code Studio</a>
from code.org. That's what I did the last couple weeks - took two hours to lightly shepherd the
<a href="http://www.ywcadayton.org">Dayton YWCA</a> day camp through a programming intro.</p>
<p>It's really well-organized and easy to understand - frankly, it pretty much drives itself.
It's based on block-dragging for turtle graphics and/or simple 2D games, all
easy and appealing stuff. (They even got their turtle graphics branded as the sisters from
Frozen ice-skating!) I didn't need to do much more than stand there
and demonstrate that programmers actually exist in the flesh, and occasionally nudge a student
over a bump. Though, by pair programming, they did most of the nudging themselves.</p>
<p>Here's most of my awesome class. Sorry I'm as bad at photography as at CSS.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiacR0YAyf96AetHr2GOvBlEdKxc4CqrRiK1_x34lpcQpzmKpyfryB6Uepx47Mlx_0kI1KNUaG-jFa_N_0rxSmXBN1uhMbNX4VHajyspjogdAi7L0i_pS8qFOh9PlhsAj9jPPMPHg/s1600/ywca.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiacR0YAyf96AetHr2GOvBlEdKxc4CqrRiK1_x34lpcQpzmKpyfryB6Uepx47Mlx_0kI1KNUaG-jFa_N_0rxSmXBN1uhMbNX4VHajyspjogdAi7L0i_pS8qFOh9PlhsAj9jPPMPHg/s200/ywca.jpg" /></a></div>
<p>Hey - we got demographics, huh? Right - if you announce that you're teaching a coding class
through your usual geeky circles, they spread the word among their circles and recruit you a class that
looks pretty much like the industry already looks. And if you seek a venue through your geeky circles,
the usual suspects will step up to host. In badly segregated Dayton, that means "as far from the
colored parts of town as possible." That's less than inviting to the people who don't live there.</p>
<p>But if you partner with groups that already have connections in diverse communities -
like the YWCA, which makes anti-racism one of its keystones - getting some fresh faces can be
pretty easy! And there <i>are</i> venues available outside the bleached-white exurbs
you're used to - you just need to think to look.</p>
<p>Another benefit of Code Studio is that it's entirely web-based, so you don't need to restrict
your demographics to "kids whose parents can afford to get them laptops". The public library's
computer classroom did the job with flying colors.</p>
<p>Seriously, this was about the easiest outreach I've ever done. I'm working on the follow-up,
but I think I'll be able to find further lazy options. Quite likely it will leverage
<a href="http://www.codecademy.com/">CodeAcademy</a>. So, what's your excuse for not doing it in your city?</p>
<p>Now, in other news: You are running out of time to <a href="http://www.pyohio.org/">register for PyOhio</a>,
a fantastic, friendly, free, all-levels Python conference, and my pride and joy. The schedule is amazing this year,
and for better or for worse, I'm keynoting. So please come and add to my terror.</p>
Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com1tag:blogger.com,1999:blog-11802292.post-14081828409541293642014-11-14T07:13:00.000-08:002014-11-14T08:06:52.578-08:00rdbms-subsetter<p>I've never had a tool I really liked that would extract a chunk of a large production database for testing purposes while respecting the database's foreign keys. This past week I finally got to write one: <a href="https://github.com/18F/rdbms-subsetter">rdbms-subsetter</a>.</p>
<code><pre>rdbms-subsetter postgresql://user:passwd@host/source_db postgresql://user:passwd@host/excerpted_db 0.001</pre></code>
<p>Getting it to respect referential integrity "upward" - guaranteeing every needed parent record would be included for each child row - took less than a day. Trying to get it to also guarantee referential integrity "downward" - including all child records for each parent record - was a Quixotic idea that had me tilting at windmills for days. It's important, because parent records without child records are often useless or illogical. Yet trying to pull them all in led to an endlessly propagating process - percolation, in chemical engineering terms - that threatened to make every test database a complete (but extremely slow) clone of production. After all, if every row in parent table P1 demands rows in child tables C1, C2, and C3, and those child rows demand new rows in parent tables P2 and P3, which demand more rows in C1, C2, and C3, which demand more rows in their parent tables... I felt like I was trying to cut a little sweater out of a big sweater without snipping any yarns.</p>
<p>So I can't <em>guarantee</em> child records - instead, the final process <em>prioritizes</em> creating records that will fill out the empty child slots in existing parent records. But there will almost inevitably be some child slots left open when the program is done.</p>
<p>I've been using it against one multi-GB, highly interconnected production data warehouse, so it's had some testing, but <a href="https://github.com/18F/rdbms-subsetter/issues">your bug reports are welcome</a>.</p>
<p>Like virtually everything else I do, this project depends utterly on <a href="http://sqlalchemy.org">SQLAlchemy</a>.</p>
<p>I developed this for use at <a href="https://18f.gsa.gov">18F</a>, and my choice of a workplace where <em>everything</em> <a href="https://github.com/18F/open-source-policy/blob/master/policy.md">defaults to open</a> was wonderfully validated when I asked about the procedure for releasing my 18F work to PyPI. The procedure is - and I quote -</p>
<blockquote>Just go for it.</blockquote>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com1tag:blogger.com,1999:blog-11802292.post-79686381524933779682014-08-26T04:03:00.000-07:002014-08-26T04:03:38.790-07:00%sql: To Pandas and Back<p>A Pandas DataFrame has a nice <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_sql.html"><var>to_sql(table_name, sqlalchemy_engine)</var> method</a> that saves itself to a database.</p>
<p>The only trouble is that coming up with the SQLAlchemy Engine object is a little bit of a pain, and if you're using the <a href="https://pypi.python.org/pypi/ipython-sql">IPython %sql magic</a>, your %sql session already has an SQLAlchemy engine anyway. So I created a bogus <code>PERSIST</code> pseudo-SQL command that simply calls <code>to_sql</code> with the open database connection:</p>
<code><pre>%sql PERSIST mydataframe</pre></code>
<p>The result is that your data can make a very convenient round-trip from your database, to Pandas and whatever transformations you want to apply there, and back to your database:<p>
<code>
<pre>
In [1]: %load_ext sql
In [2]: %sql postgresql://@localhost/
Out[2]: u'Connected: @'
In [3]: ohio = %sql select * from cities_of_ohio;
246 rows affected.
In [4]: df = ohio.DataFrame()
In [5]: montgomery = df[df['county']=='Montgomery County']
In [6]: %sql PERSIST montgomery
Out[6]: u'Persisted montgomery'
In [7]: %sql SELECT * FROM montgomery
11 rows affected.
Out[7]:
[(27L, u'Brookville', u'5,884', u'Montgomery County'),
(54L, u'Dayton', u'141,527', u'Montgomery County'),
(66L, u'Englewood', u'13,465', u'Montgomery County'),
(81L, u'Germantown', u'6,215', u'Montgomery County'),
(130L, u'Miamisburg', u'20,181', u'Montgomery County'),
(136L, u'Moraine', u'6,307', u'Montgomery County'),
(157L, u'Oakwood', u'9,202', u'Montgomery County'),
(180L, u'Riverside', u'25,201', u'Montgomery County'),
(210L, u'Trotwood', u'24,431', u'Montgomery County'),
(220L, u'Vandalia', u'15,246', u'Montgomery County'),
(230L, u'West Carrollton', u'13,143', u'Montgomery County')]
</pre>
</code>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com3tag:blogger.com,1999:blog-11802292.post-18482804166603979492014-07-28T14:30:00.000-07:002014-07-28T14:30:43.765-07:00auto-generate SQLAlchemy models<p><a href="pyohio.org">PyOhio</a> gave <a href="http://nbviewer.ipython.org/gist/catherinedevlin/ade09abd15b2364ea73f">my lightning talk</a> on <a href="https://pypi.python.org/pypi/ddlgenerator">ddlgenerator</a> a warm reception, and Brandon Lorenz got me thinking, and PyOhio sprints filled my with py-drenaline, and now ddlgenerator can inspect your data and spit out SQLAlchemy model definitions for you:</p>
<code><pre>
$ cat merovingians.yaml
-
name: Clovis I
reign:
from: 486
to: 511
-
name: Childebert I
reign:
from: 511
to: 558
$ ddlgenerator --inserts sqlalchemy merovingians.yaml
from sqlalchemy import create_engine, Column, Integer, Table, Unicode
engine = create_engine(r'sqlite:///:memory:')
metadata = MetaData(bind=engine)
merovingians = Table('merovingians', metadata,
Column('name', Unicode(length=12), nullable=False),
Column('reign_from', Integer(), nullable=False),
Column('reign_to', Integer(), nullable=False),
schema=None)
metadata.create_all()
conn = engine.connect()
inserter = merovingians.insert()
conn.execute(inserter, **{'name': 'Clovis I', 'reign_from': 486, 'reign_to': 511})
conn.execute(inserter, **{'name': 'Childebert I', 'reign_from': 511, 'reign_to': 558})
conn.connection.commit()
</pre></code>
<p>Brandon's working on a pull request to provide similar functionality for Django models!</p>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com0tag:blogger.com,1999:blog-11802292.post-51900818309592413902014-07-01T09:37:00.000-07:002014-07-01T09:37:59.461-07:0018F<p>Yesterday was my first day at <a href="">18F</a>!</p>
<p>What is 18F? We're a small, little-known government organization that works outside the usual channels to accomplish special projects. It involves black outfits and a lot of martial arts.</p>
<p>Kidding! Sort of. 18F is a new agency within the GSA that does citizen-focused work for other parts of the U.S. Government, working small, quick projects to make information more accessible. We're using all the tricks: small teams, agile development, rapid iteration, open-source software, test-first, continuous integration. We do our work <a href="https://github.com/18f/">in the open</a>.</p>
<p>Sure, this is old hat to you, faithful blog readers. But bringing it into government IT work is what makes it exciting. We're hoping that the techniques we use will ripple out beyond the immediate projects we work on, popularizing them throughout government IT and helping efficiency and responsiveness throughout. This is a chance to put all the techniques I've learned from you to work for all of us. Who wouldn't love to get paid to work for the common good? </p>
<p>Obviously, this is still my <em>personal</em> blog, so nothing I say about 18F counts as official information. Just take it as my usual enthusiastic babbling.</p>
Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com1tag:blogger.com,1999:blog-11802292.post-30555107633853768132014-05-23T13:23:00.000-07:002014-05-23T14:09:03.815-07:00ddlgenerator<p>I've had it on github for a while, but I finally released <a href="https://pypi.python.org/pypi/ddlgenerator/0.1.1">ddlgenerator</a> to PyPI.</p>
<p>I've been frustrated for years that there was no good open-source way to set up RDBMS tables from flat data files. Sure, you could import the data - after setting up the DDL by hand. ddlgenerator handles that; in fact, you can go from zero, setting up and populating a table in a single line. Nothing up my sleeve:</p>
<pre>
$ psql -c "SELECT * FROM knights"
ERROR: relation "knights" does not exist
LINE 1: SELECT * FROM knights
^
$ ddlgenerator --inserts postgresql knights.yaml | psql
CREATE TABLE
INSERT 0 1
INSERT 0 1
INSERT 0 1
INSERT 0 1
$ psql -c "SELECT * FROM knights"
name | dob | kg | brave
------------+---------------------+---------+-------
Lancelot | 0471-01-09 00:00:00 | 82.0000 | t
Gawain | | 69.2000 | t
Robin | 0471-01-09 00:00:00 | | f
Reepacheep | | 0.0691 | t
</pre>
<p>This is a fairly complex tool so I'm sure you'll be using <a href="https://github.com/catherinedevlin/ddl-generator/issues">the bug tracker</a>. But I hope you'll enjoy it nonetheless!</p>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com0tag:blogger.com,1999:blog-11802292.post-2776208873232045952014-05-21T09:50:00.000-07:002014-05-21T09:50:23.481-07:00data_dispenser<p>I went down a refactoring rabbit hole on <a href="https://github.com/catherinedevlin/ddl-generator">ddl-generator</a> and ended up pulling out the portion that pulls in data from various file formats. Perhaps it will be useful to others.</p>
<pre>
>>> from data_dispenser.sources import Source
>>> for row in Source('animals.csv'):
... print(row)
...
OrderedDict([('name', 'Alfred'), ('species', 'wart hog'), ('kg', '22'), ('notes', 'loves turnips')])
OrderedDict([('name', 'Gertrude'), ('species', 'polar bear'), ('kg', '312.7'), ('notes', 'deep thinker')])
OrderedDict([('name', 'Emily'), ('species', 'salamander'), ('kg', '0.3'), ('notes', '')])
</pre>
<p>Basically, I wanted a consistent way to consume rows of data - no matter where those rows come from. Right now, JSON, CSV, YAML, etc. all require separate libraries, each with its own API. This abstracts all that out, for reading purposes; now each data source is just a Source.</p>
<p>I'd love bug reports, and sample files to test against. And feel free to contribute patches! For example, it wouldn't be hard to <a href="https://github.com/catherinedevlin/data-dispenser/issues/1">add MS Excel</a> as a data source.</p>
<p><a href="https://pypi.python.org/pypi/data_dispenser">https://pypi.python.org/pypi/data_dispenser</a></p>
Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com0tag:blogger.com,1999:blog-11802292.post-88110095121008162012014-05-06T19:29:00.000-07:002014-05-06T20:09:07.501-07:00G+ Public Hangout Fail<h1><em>tl;dr:</em>Do not use public Google+ Hangouts under any circumstances, because people suck.</h1>
<p>Before the PyCon 2014 CFP came due, <a href="http://www.pyladies.com/blog/pycon-2014-cfp-brainstorm/">PyLadies hosted several G+ hangouts for talk proposal brainstorming</a>. Potential speakers could talk over and flesh out their ideas with each other, producing better talk proposals. More importantly, it was a nice psychological stepping stone on the way to filling out that big, scary CFP form all alone. I thought they went great.</p>
<p>I wanted to emulate them for <a href="http://postgresopen.org/2014/callforpapers/">Postgres Open</a> and <a href="http://www.pyohio.org/call-for-proposals/">PyOhio</a>, which both have CFPs open now. The PyLadies hangouts had used EventBrite to preregister attendees, and I unfortunately did not consider this and the reasons why. Instead, I just scheduled hangouts, made them public, and sent out invitations with the hangout URLs, encouraging people to forward the invites onward. Why make participating any harder than it has to be?</p>
<p>The more worldly of you are already shaking your heads at my naiveté. It turns out that the world's exhibitionists have figured out how to automatically detect and join public hangouts. For several seconds I tried kicking out and banning them as they joined, but new ones kept arriving, faster than one per second. Then I hung up - which unfortunately did not terminate the hangout. It took me frantic minutes to find how to delete a hangout in progress. I dearly hope that no actual tech community members made it to the hangout during that time.</p>
<p>I had intended to create a place where new speakers, and women especially, would feel safe increasing their community participation. The absoluteness of my failure infuriates me.</p>
<p>Hey, Google: public G+ hangouts have been completely broken, not by technical failure, but by the degraded human condition. You need to remove them immediately. The option can only cause harm, as people accidentally expose themselves and others to sexual harrassment.</p>
<p>In the future, a "public" hangout URL should actually take you to a page where you request entrance from the organizer by text message (which should get the same spam filtration that an email would). But fix that later. Take the public hangouts away now.</p>
<p>Everybody else, if you had heard about the hangouts and were planning to participate, THANK YOU - but I've cancelled the rest of them. You should present anyway, though! I'd love to be contacted directly to talk over your ideas for proposals.</p>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com4tag:blogger.com,1999:blog-11802292.post-11699470313540233902014-01-29T21:47:00.000-08:002014-01-29T21:47:00.568-08:00TRUCEConf<p>Please consider participating in <a href="truceconf.com">TRUCEConf</a> (March 18-19 in Cincinnati)!</p>
<blockquote>
The goal is to help the tech community heal, through learning from others outside our industry and having an open dialogue and on how we can be better humans to each other in the world of tech.
</blockquote>
<p>You may remember fierce controversy around TRUCEConf when virtually nothing was known
about it but its name; without solid information, it was easy to read bad connotations
into the name. I would have been uneasy myself if I hadn't known the founder, Elizabeth Naramore. </p>
<p>But now there's plenty of information, including the
<a href="http://truceconf.com/schedule/">schedule</a>, that should replace those
concerns with enthusiasm. I think the format - a day of mind-opening speakers
from all over, followed by an unconference day - should be very productive!</p>
<p>I'm really looking forward to it and hope that many of you can come.
If you can't come in person, consider supporting the conference with a donation -
they're going without corporate sponsors
so your individual support means a ton. Thanks!</p>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com3tag:blogger.com,1999:blog-11802292.post-11247739154906065232013-10-03T07:21:00.000-07:002013-10-03T07:21:10.000-07:00SacredPy seeking collaborators<p>I'm looking for collaborators who want to build web programming experience on an interesting project...</p>
<p>During my job search, I was contacted by <a href="https://plus.google.com/102884090920800802466">Kai Schraml</a>, a seminary graduate who wants to scratch an itch. Seminarians have a serious need to discuss, debate, and seek consensus on the translations of difficult texts, like sacred scriptures. But the software tools currently available for the purpose are closed-source and expensive. That just seems wrong - not just because seminary students are broke, but because of the nature of the texts themselves. After all, Jesus released his teachings under a very strong open-source license!* </p>
<p>So we're starting to work on an alternative, provisionally called "SacredPy". (It could be applied to any difficult texts, of course, so if <i>Beowulf</i> is sacred to you, have at it.) I'm quite employed now, but I'm dabbling at it a bit for the sheer interest and open-sourcey glory of it all. It's possible income could eventually come from this project - Kai could tell you more about the prospects - but certainly not soon, so this is no substitute for proper employment. But it might be great resume builder for a new Python programmer. It looks like we'll most likely build something atop <a href="askbot.org">Askbot</a>, a Django-based project, so if you'd like to move into the thriving "experienced Djano developer" segment of the economy...</p>
<p>Let me know at
<span style="unicode-bidi:bidi-override; direction: rtl;">moc.liamg@nilved.enirehtac</span> and we'll talk!</p>
<p>* - <a href="http://www.biblegateway.com/passage/?search=Matthew+10">Matthew 10:8</a> - δωρεὰν ἐλάβετε, δωρεὰν δότε ("Freely you have received, freely give")</p>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com1tag:blogger.com,1999:blog-11802292.post-75837472380873756552013-09-19T16:59:00.001-07:002013-09-19T16:59:13.754-07:00Presentation links<p>I have just had a VERY. Busy. Week. (In a good way!) I've promised the world many talk materials, so:
<ul>
<li>At <a href="http://ohiolinux.org">Ohio LinuxFest</a>, <a href="http://spkrbar.com/talk/46">IPython for non-Pythonistas</a></li>
<li>At <a href="http://rleevanceapcug.wix.com/apcug-2013-regional">APCUG Regional Conference</a>, <a href="http://tinyurl.com/apcug-2013-python-intro">Python</a></li>
<li>At <a href="http://postgresopen.org">Postgres Open</a>, <a href="http://tinyurl.com/ipgopen2013-ipy-sql">IPython: your new SQL client</a></li>
</ul>
<p>As for Postgres Open, I absolutely loved it! So happy I finally got to go. I am proud to say that I was the very first to buy my admission for 2014! Hope to blog more about that later...</p>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com3tag:blogger.com,1999:blog-11802292.post-2177564378626257602013-08-29T11:23:00.002-07:002013-08-29T11:23:52.044-07:00I'm not available<p>I'm happy to say that I'll shortly be starting a new position as a PostgreSQL DBA and Python developer for Zoro Tools!</p>
<img src="http://www.zorotools.com/img/zoro_logo_color.gif" />
<p>We software types seem to have hardware envy sometimes. We have "builds" and "engines" and "forges" and "factory functions". But as it turns out, the "Tools" in "Zoro Tools" isn't a metaphor for cleverly arranged bytes. That's right - they're talking about the <i>physical objects in your garage!</i> Imagine! Lucky for me the interviewers didn't ask to review my junior high shop project.</p>
<p>So disregard my <a href="http://catherinedevlin.blogspot.com/2013/08/im-available.html">earlier post about being available</a>. Thanks for all your well-wishes!</p>
<p>Depending on how you reckon it, my job search arguably only took forty minutes, though it took a while for gears to grind and finalize everything. Years of building relationships at PyCon made this the best job search ever; the only unpleasant part was having to choose from among the opportunities to work with my favorite technologies and people. I'm very glad I made the investment in PyCon over the years... and if you're thinking "that's easy for you to say, I can't afford it", don't forget PyCon's <a href="http://us.pycon.org/2014/assistance/">financial aid program</a>.</p>
<p>And speaking of conferences, I'll be at <a href="http://postgresopen.org/2013/home/">Postgres Open</a> next month (my first one!) - hope to see some of you there!</p>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com5tag:blogger.com,1999:blog-11802292.post-80488430223479621632013-08-26T02:56:00.000-07:002013-08-26T02:56:13.180-07:00IPython at Ohio LinuxFest 2013<p>Are you signed up yet for <a href="ohiolinux.org">Ohio LinuxFest</a> on Sep. 13-15? I'll be there to present</p>
<h2>IPython for non-Pythonistas</h2>
<blockquote>Break out of your (bash) shell! IPython and the IPython Notebook have swept over the Python programming community, but they're not just for Python programmers - they make for high-powered shell replacements even with little to no Python knowledge. They'll also let you document your work and collaborate with others like never before. Find out how these beautiful tools can improve your daily Linux work! </blockquote>
<p>At PyOhio, <a href="http://pyvideo.org/video/2286/the-ipython-notebook-revolution">I argued</a> that all Python programmers need IPython. At OLF, I'll make the case that non-Pythonistas need IPython, too. Perhaps my next talk will be "Even Your Cat Needs IPython".</p>
<p>Also at OLF, look for <a href="pyohio.org">PyOhio</a>'s booth for info on next year's PyOhio, other Python events around the region, and general Python love!</p>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com0tag:blogger.com,1999:blog-11802292.post-18411422441920389672013-08-19T12:59:00.000-07:002013-08-19T12:59:10.855-07:00Python Workshop for Women Indy #2 and CMH #2 coming up!<p>The Midwest Python Workshop for women and their friends is back! We've got new workshops scheduled, ready to take new batches of students:</p>
<p><a href="tinyurl.com/indy-py-workshop">Indianapolis Python Workshop</a>, Sep. 27-28, 2013; sponsored by <a href="sixfeetup.com">Six Feet Up</a> and hosted at <a href="launchfishers.com">Launch Fishers</a></p>
<p><a href="tinyurl.com/cmh-py-workshop">Columbus Python Workshop</a>, Oct. 18-19, 2013; sponsored by <a href="leadingedje.com">LeadingEdje</a> and hosted at <a href="">The Forge by Pillar</a></p>
<p>The Workshop is a free, friendly, hands-on introduction to computer programming using Python. Women of all ages and backgrounds are the primary target (but you can bring a male participant as your guest).</p>
<p>Please spread the word!</p>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com0tag:blogger.com,1999:blog-11802292.post-58407900775144140022013-08-11T19:11:00.000-07:002013-08-11T19:11:33.875-07:00I'm available<p>I'm available for hire! If you need a database expert with lots of programming skill, or a Python programmer with deep database experience, please check out:</p>
<p><a href="http://catherinedevlin.pythoneers.com">Resume / portfolio</a></p>
<p><em>But:</em> you must be telecommute-friendly, or in the Dayton area. I'm sorry, but I'm not available to relocate.</p>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com0tag:blogger.com,1999:blog-11802292.post-85130334906018740242013-07-30T02:45:00.000-07:002013-07-30T02:47:26.179-07:00IPython %helloworld extension<p>At Monday's after-PyOhio sprint, I changed <a href="https://pypi.python.org/pypi/ipython-sql">ipython-sql</a> from an IPython Plugin to an Extension; this makes it compatible with IPython 1.0. Fortunately, this was really easy; mostly I just deleted Plugin code I didn't understand anyway.</p>
<p>But I do feel like <a href="http://ipython.org/ipython-doc/dev/config/extensions/index.html#writing-extensions">"Writing Extensions" docs</a> are lacking a "Hello World" example. Here's mine.</p>
<code><pre>
from IPython.core.magic import Magics, magics_class, line_magic, cell_magic
@magics_class
class HelloWorldMagics(Magics):
"""A simple Hello, <name> magic.
"""
@line_magic # or ``@line_magic("hi")`` to make ``%hi`` the name of the magic
@cell_magic
def helloworld(self, line='', cell=None):
"""Virtually empty magic for demonstration purposes.
Example::
In [1]: %load_ext helloworld
In [2]: %helloworld Catherine
Out[2]: u'Hello, Catherine'
"""
return "Hello, %s\n%s" % (line, cell or "")
def load_ipython_extension(ip):
ip.register_magics(HelloWorldMagics)
</pre><code>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com0tag:blogger.com,1999:blog-11802292.post-91960969006776509282013-07-30T02:12:00.000-07:002013-07-30T02:12:30.799-07:00PyOhio Stone Soup<p>Loved PyOhio once again! Thanks so much to everybody who came, participated, and made it happen! I get such a rush of joy from seeing the Ohio Union fill up with happy Pythonistas.</p>
<p>PyOhio has been a classic case of the Stone Soup story. When we started planning the first one, we really didn't have the resources to pull off a conference; we were just a handful of PyCon 2008 attendees who wanted to bring something like PyCon home. But as we put it together, people appeared, pitched in, and we had a modest, amateurish - but fun! - little conference in the Columbus Public Library. PyOhio 2008 drew participants and volunteers who helped make PyOhio 2009 bigger and better; 2009 drew in more involvement for 2010; and so forth, year after year.</p>
<p>July 26-27, 2014. See you in Columbus!</p>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com0tag:blogger.com,1999:blog-11802292.post-37845979656660324582013-07-19T07:19:00.000-07:002013-07-19T07:19:02.648-07:00The IPython Notebook RevolutionAmong the <a href="http://pyohio.org/schedule/talks/list/">many great talks</a> coming to PyOhio at the end of this month:
<blockquote>
The IPython Notebook Revolution
Catherine Devlin
If you think of IPython as simply an enhanced version of the live Python prompt, you are in need of re-education. No matter what you do with Python, applying the IPython Notebook imaginatively will revolutionize the way you do it.
</blockquote>
<p>I'd like to focus on aspects of IPython outside the traditional number-crunching, plot-making realm, simply because those have been covered so well already - <a href="http://pyvideo.org/search?models=videos.video&q=ipython">videos by the actual IPython team</a> already have. I'd like to fill up a talk with edgy, imaginative, experimental uses of IPython that aren't well-known yet, or that suggest new ways IPython (and especially the Notebook) may be used in the future. I have a bunch of ideas along those lines...</p>
<p>... but I'd like your input! I don't want to miss anything awesome just because I wasn't aware, and there's a <i>lot</i> being done in the IPython world - more than I've been able to keep track of. Erik Welch has already thoughtfully given me a bunch of links and suggestions from SciPy. Let's crowdsource my talk even further!</p>
<p>Some of the goodies I already plan to include:
<ul>
<li>notebook-based presentations</li>
<li>ipython_blocks: probably my Holy Grail of imaginative uses)</li>
<li>d3js in IPython: (OK, this still fits the data graphing theme, but it's also ultra-snazzy)</li>
<li>ipython_sql: (everybody's got to toot her own horn sometimes)</li>
<li>ipfl (web-style forms in a Notebook - very preliminary but an interesting idea)</li>
<li>xkcd and hand-drawn mode</li>
<li>Wakari</li>
</ul>
</p>
<p>How would <i>you</i> shake up people's notions of "what IPython is for"?</p>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com7tag:blogger.com,1999:blog-11802292.post-34752327294839850482013-06-28T10:54:00.000-07:002013-06-28T10:54:17.150-07:00Easy HTML output in IPython NotebookIf any object has a <var>_repr_html_</var> method, the IPython Notebook will use it to render HTML output. It's really easy to make a simple class that permits general dynamic HTML-rich output with Markdown. Markdown is a superset of HTML, so HTML in your output string will work, too.
<code><pre>
import markdown
class MD(str):
def _repr_html_(self):
return markdown.markdown(self)
</pre></code>
Four little lines, and you can do this!
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjh_SKkcYMLdkr2SZq8v1GZjv9BwApmtdYpY2H0aF-Ql9Hky5uHByG7ULT9M1affxZsjAkeJ6LywBDlwS9QzC67QABVxvS6QAa_cBj5oWKWhjUK0inRmVyA2qc0kyQOoHZsrZIW2w/s543/mddemo.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjh_SKkcYMLdkr2SZq8v1GZjv9BwApmtdYpY2H0aF-Ql9Hky5uHByG7ULT9M1affxZsjAkeJ6LywBDlwS9QzC67QABVxvS6QAa_cBj5oWKWhjUK0inRmVyA2qc0kyQOoHZsrZIW2w/s543/mddemo.png" /></a></div>
Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com1tag:blogger.com,1999:blog-11802292.post-41801029394767222932013-05-30T14:55:00.001-07:002013-05-30T14:57:39.954-07:00ipython-sql bind variables<p>Thanks to <a href="https://plus.google.com/103754023380662812553/posts">Mike Wilson</a>, <a href="https://pypi.python.org/pypi/ipython-sql">ipython-sql</a> now supports bind variables!</p>
<code><pre>
In [12]: name = 'Countess'
In [13]: %sql select description from character where charname = :name
Out[13]: [(u'mother to Bertram',)]
</pre></code>
</p>
Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com0tag:blogger.com,1999:blog-11802292.post-18334105327957664842013-05-25T19:27:00.000-07:002013-05-25T19:27:42.112-07:00Review of Learning IPython for Interactive Computing and Data Visualization<div id="hreview-valuable-but-traditional" class="hreview"><h2 class="summary">valuable but traditional</h2><abbr title="2013-05-25T21:56-04:00" class="dtreviewed">May 25, 2013</abbr> by <span class="reviewer vcard"><span class="fn">Catherine Devlin</span></span><span style="display: none;" class="type">product</span><img class="photo" src="http://dgdsbygo8mp3h.cloudfront.net/sites/default/files/imagecache/productview_larger/9932OS.jpg" alt="photo of 'Learning IPython for Interactive Computing and Data Visualization'"><div class="item"><a href="www.packtpub.com/learning-ipython-for-interactive-computing-and-data-visualization/book" class="fn url">Learning IPython for Interactive Computing and Data Visualization</a></div>
<span><span class="rating">4</span> stars (of 5)</span>
<div class="description">
<p>
<a href-"http://www.packtpub.com/">Packt Publishing</a> recently asked if I could review their new title, <a
href="http://www.packtpub.com/learning-ipython-for-interactive-computing-and-data-visualization/book">Learning IPython for Interactive Computing and Data Visualization</a>.
(I got the e-book free for doing the review, but they don't put any conditions on what I say about it.)
I don't often do reviews like that, but I couldn't pass one this up because I'm so excited about the IPython Notebook.</p>
<p>It's a mini title, but it does contain a lot of information I was very pleased to see.
First and foremost, this is the first book to focus on the IPython Notebook. That's huge. Also:</p>
<ul>
<li>The installation section is thorough and goes well beyond the obvious, discussing options like
using prepackaged all-in-one Python distributions like <a href="https://store.continuum.io/">
Anaconda</a>.</li>
<li>Some of the improvements IPython can make to a programming workflow are nicely introduced,
like the ease of debugging, source code inspection, and profiling with the appropriate magics.</li>
<li>The section on writing new IPython extensions is extremely valuable - it contains more complete
examples than the official documentation does and would have saved me lots of time and excess code if I'd had it when I was writing
<a href="https://pypi.python.org/pypi/ipython-sql">ipython-sql</a>. </li>
<li>There are introductions to all the classic uses that scientists doing numerical simulations value
IPython for: convenience in array handling, Pandas integration, plotting, parallel computing, image
processing, Cython for faster CPU-bound operations, etc. The book makes no claim to go deeply into any of these, but it gives introductory examples that
at least give an idea of how the problems are approached and why IPython excels at them.</li>
</ul>
<p>So what don't I like? Well, I wish for more. It's not fair to ask for more bulk in a small book
that was brought to market swiftly, but I can wish for a more forward-looking, imaginative treatment.
The IPython Notebook is ready to go far
beyond IPython's traditional core usership in the SciPy community, but this book doesn't really make
that pitch. It only touches lightly on how easily and beautifully IPython can replace shell scripting.
It doesn't get much into the unexplored possibilities that
IPython Notebook's rich display capabilities open up. (I'm thinking of
<a href="https://pypi.python.org/pypi/ipythonblocks/">IPython Blocks</a> as a great example
of things we can do with IPython Notebook that we never imagined at first glance). This book
is a good introduction to IPython's
uses as traditionally understood, but it's not the manifesto for the upcoming IPython Notebook Revolution.</p>
<p>
The power of hybrid documentation/programs for learning and individual
and group productivity is one more of IPython Notebook's emerging possibilities that this book only
mentions in passing, and passes up a great chance to demonstrate.
The sample code is downloadable as IPython Notebook <var>.ipynb</var> files, but the bare code is alone
in the cells, with no use of Markdown cells to
annotate or clarify. Perhaps this is just because Packt was afraid that more complete Notebook files would be pirated, but it's a shame.
</p>
<p>Overall, this is a short book that achieves its modest goal: a technical introduction to IPython
in its traditional uses. You should get it, because IPython Notebook is too important to sit around
waiting for the ultimate book - you should be using the Notebook <em>today</em>.
But save space on your bookshelf for future books, because there's much more to be said on the topic, some of which hasn't even been imagined yet.</p>
</div><p style="font-size: small;">This <a href="http://microformats.org/wiki/hreview">hReview</a> brought to you by the <a href="http://microformats.org/code/hreview/creator">hReview Creator</a>.</p></div>
Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com2tag:blogger.com,1999:blog-11802292.post-54179267858832578442013-05-08T13:42:00.000-07:002013-05-08T13:42:39.761-07:00cannot import name MAXREPEAT<p>When I upgraded from Xubuntu 12.10 to 13.04 today, all my existing Python virtualenvs broke! Fortunately, they're just virtualenvs and easy to replace (that's kind of the point). But don't panic if you start seeing these.</p>
<pre>
$ ipython
Traceback (most recent call last):
File "/home/catherine/ve/e2/bin/ipython", line 5, in <module>
from pkg_resources import load_entry_point
File "build/bdist.linux-i686/egg/pkg_resources.py", line 16, in <module>
File "/home/catherine/ve/e2/lib/python2.7/re.py", line 105, in <module>
import sre_compile
File "/home/catherine/ve/e2/lib/python2.7/sre_compile.py", line 14, in <module>
import sre_parse
File "/home/catherine/ve/e2/lib/python2.7/sre_parse.py", line 17, in <module>
from sre_constants import *
File "/home/catherine/ve/e2/lib/python2.7/sre_constants.py", line 18, in <module>
from _sre import MAXREPEAT
ImportError: cannot import name MAXREPEAT
</pre>
<p>Apparently Python 2.7.4 introduces _sre.MAXREPEAT. Here it is in my (new)
system Python, 2.7.4:</p>
<pre>
Python 2.7.4 (default, Apr 19 2013, 18:28:01)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import _sre
>>> _sre.MAXREPEAT
4294967295L
</pre>
<p>... but the virtualenvs I created before the upgrade still use Python 2.7.3</p>
<pre>
Python 2.7.3 (default, Sep 26 2012, 21:51:14)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import _sre
>>> _sre.MAXREPEAT
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'MAXREPEAT'
</pre>
<p>If I were a deeper hacker I'd try to figure out why code running within my old virtualenvs is trying to access a 2.7.4-only attribute,
or what's the most efficient way to recover my old virtualenvs. But I'll settle for recognizing the problem and spinning up new
virtualenvs instead. That fixes the problem.</p>
<p>I know... I could have avoided this problem by using Python 3! I've got code that depends on fabric, though, which still isn't available for 3.</p>
Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com3tag:blogger.com,1999:blog-11802292.post-1522389955247325032013-05-04T08:51:00.001-07:002013-05-04T08:53:43.904-07:00Speak at PyOhio<p>Have you responded yet to <a href="http://pyohio.org/call-for-proposals/">PyOhio's Call For Proposals</a> (due date: June 1)? You should. Here's why.</p>
<h2>Why you should speak at PyOhio</h2>
<li><i>We need you.</i> We have a great group of people who contribute talks, but we don't ever want that group to become a stagnant pool - we want to always be drawing in new contributors with a variety of backgrounds, perspectives, and interests. Unless your clone sibling just submitted a talk, we need your contribution!</li>
<li><i>Meet people.</i> You'll always meet people at PyOhio, of course, but speaking makes you a beacon to the other people who share your interests. </li>
<li><i>Practice.</i> We all know public speaking is an important skill, but it's hard to muster the courage to get real-life practice in it. PyOhio is perfect because of the friendliness of the Python community - everybody there <i>wants</i> you to succeed. Hone your skills with us, among friends... you can go speak at snotty places later!</li>
<li><i>Push yourself.</i> That skill you've been wanting to learn, or that package you know you need to polish up for release? Committing to speak on it is a surefire way to break the curse of "haven't quite gotten around to it"! Sometimes we call this "embarrassment-driven development".</li>
<h2>But I'm not an expert</h2>
<p><i>Perfect.</i> Because the great curse of expert teachers is that they can forget what it was like to be a beginner. So dive into something you want to learn, take careful notes as you go about what confused you and how you resolved it, and you'll blaze a trail that you can guide other beginners along. Your non-expert perspective will make you a great teacher!</p>
<h2>Team up</h2>
<p>You can draw on that friendly community to help you present, too! Share a presentation with somebody more or less experienced to make an expert-beginner duet, or have a friend cover an aspect of your topic that they know better. Get a friend to review your talk as you develop it. Shop your ideas around your local Python usergroup and see what suggestions they have.</p>
<h2>Many ways to contribute</h2>
<p>PyOhio is not all about talks, of course (for me, the talks are kind of the excuse we use to get together and do the other stuff.) Also consider proposing something like</p>
<ul>
<li>a tutorial</li>
<li>an Open Space</li>
<li>a Sprint</li>
<li>a Lightning Talk (actually, you propose these on-the-spot, but you can get it ready in advance)</li>
</ul>
<p>Thank you, and spread the word!</p>Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com0tag:blogger.com,1999:blog-11802292.post-4114534644039069332013-05-03T16:27:00.000-07:002013-05-03T16:27:26.694-07:00ipython-sql for multi-database comparisons<p>For my newest <a href="https://pypi.python.org/pypi/ipython-sql">ipython-sql</a> trick, I needed to compare some queries run across different databases. How hard would it be to get side-by-side results into tidy IPython Notebook output?</p>
<p>Not hard at all, it turns out, if you're willing to violate basic principles of human decency.</p>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEit-gXaAwTmojSLDuWeUxA_6OmwSAHNCbeAUoqPeJSVM5Pq4aLt_5CYDhH-QR8QgEWZMyOubE23hTc4xu0sOjeGEUHS3LuIuwdFNFRvKnZwDwXUrdNQxixpjWbbE7RW_IglX4TUnw/s1600/SQL_Comparitor.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEit-gXaAwTmojSLDuWeUxA_6OmwSAHNCbeAUoqPeJSVM5Pq4aLt_5CYDhH-QR8QgEWZMyOubE23hTc4xu0sOjeGEUHS3LuIuwdFNFRvKnZwDwXUrdNQxixpjWbbE7RW_IglX4TUnw/s320/SQL_Comparitor.png" /></a>
<p>That's an itty-bitty image, so here's the crazy part:</p>
<code><pre>
class SQL_Comparitor(object):
def __init__(self, *connection_strings):
self.connection_strings = connection_strings
def run(self, qry):
dframes = []
for connection_string in self.connection_strings:
result = %sql $connection_string $qry
</pre></code>
<p>Did you catch that? I used %sql magic and IPython variable substitution <i>inside an instance method</i>. It feels so wrong! But it works! Provided you're running within IPython, of course; normal Python will not under any circumstances run an unholy perlish abomination like this. I'm just really amazed that we can use IPython tricks inside class definitions, but it's real.</p>
<p>Since the result is a Pandas DataFrame, it's easy to apply transformations. For instance, say you only want the rows where the values are different:</p>
<code><pre>
diff = results[results['svr1/db1_Value'] != results['svr2/db2_Value']]
HTML(diff.to_html())
</pre></code>
<p>I'm not sure how to distribute this class, since it's small and it's not actually valid Python, just valid IPython. For now I've made a <a href="https://gist.github.com/catherinedevlin/5514949">gist</a> (and its <a href="http://nbviewer.ipython.org/5514949">nbviewer version</a>).</p>
Anonymoushttp://www.blogger.com/profile/12229578427522022392noreply@blogger.com2