Thoughts on Python

Most people at Mozilla are remote so each quarter we sync up face to face as a group for an all-hands meeting. There are over 600 employees! We of course sync up in smaller groups more frequently but this is a chance to see what's going on across the entire Mozilla horizon.

So what's happening at Mozilla? We're on the cusp of a huge shift towards an open web platform. That is, something more than a web browser -- something you can run "native" apps on. There's a lot of work left to do, of course. Here is a random dump of interesting projects in the works...

Fudge, the python mock tool, goes 1.0! You can grab it with pip install -U fudge or directly from PyPI. This marks the end of a long incubation period where the community and I used Fudge in real world scenarios to see what worked and what didn't. I'm sure there are many more improvements to make but as of 1.0 I'm very satisfied with what we've accomplished. This is thanks to its small but vocal community of users, to all contributors and to everyone who pointed out flaws...

As web developers we are faced with this problem: how do we scale up our code to handle high traffic? A lot of time and engineering goes into this problem -- time to simulate the traffic we expect and add servers to our cluster, cache heavy database access, etc, in anticipation of the load. Time is precious. This time could be spent optimizing the usefulness of our web product and creating interesting content. No one really congratulates you when a website works, they expect it to work.

When Google App Engine was released their pitch was...

Whenever I'd hear about someone from the Python community getting hired by Mozilla I'd get really excited because I knew they'd continue to share and collaborate in the open source world that I was a part of. So here I am about a month into joining Mozilla myself to work with the WebDev team. Everything Mozilla does is right out in the open: ideas are posted on blogs, code is committed to public repositories--free to use, free to fork, etc. They take a firm stance that everything you do on the web should be free and open even to the point where the new Firefox 4 audio API (which is amazing) doesn't even support the patented, closed MP3 format despite its ubiquity.

This transparent approach to technology is really powerful...

The Mac dev community has outgrown Python 2.5 it seems but you might still need it if, say, you work on a Google App Engine project. I recently set up a fresh new Snow Leopard machine and really didn't want to eff it up with Macports. I like Macports but it leaks dynamic libraries here and there and this screwed me on my last machine. Instead I've had good luck with Homebrew (so far) but I couldn't find a Python 2.5 recipe anywhere. Snow Leopard ships with a version of Python 2.5 but I ran into some trouble with it as soon as I tried using virtualenv. Apple usually messes up Python in some way so I figured I'd just compile one. Since I couldn't find anything succinct while searching online here are my notes...

Most deployment systems depend on PyPI, the Python Package Index, for fetching and installing dependencies. Although performance and reliability has greatly improved, there are still days when you may find PyPI down and thus are unable to deploy through the normal fashion.

Finally, there is now a concerted effort to create official mirrors of PyPI that everyone can use (see PEP 381). The mirrors are pretty simple--they're not full blown package indexes--and you can already start using them for experimentation. Also, thanks to Richard Jones, Martin Löwis, Guido, and others at EuroPython, my half-baked idea of using Google App Engine as a PyPI mirror has been resurrected! I don't know if it's fully implemented yet but you can try it out at pypi.appspot.com. App Engine seems like a logical place for a mirror due to its scalability. However, I am skeptical of how well it will perform. App Engine still has a long way to go with regards to stability.

To get involved with the mirroring project you can follow the discussions on the Python Catalog SIG mailing list. The next step will be modifying clients to discover and fallback on available mirrors.

Proposals for talks at PyCon 2010 are due October 1st. That's in two days! PyCon is really a little more like an unconference and is an excellent place to share your ideas or showcase your work to the Python developer community. If you're having fun with something Python related then chances are it will make a great talk. Also, don't be afraid to submit a proposal that you feel is unfinished. As long as you have a basic idea for a talk and can provide a decent outline then now is the time to sneak it in under the deadline.

I have given some talks at PyCon that I thought went pretty well over the past couple years: A talk about the Fixture module, Unicode In Python Completely Demystified (these slides still get a lot of traffic), and Strategies For Testing Ajax. All these talks were really fun. I considered proposing another talk but I've come to the realization that I have way too many spare-time projects on the go and I need every single minute of the next 12+ months to jam. I will of course be attending PyCon and hopefully can cook up a lightning talk on Fudge or one of my other pet projects in time.

The fixture Python module is a utility for managing data needed for automated tests. Its new version, 1.3, adds support for the Django backend. This was a champion effort by Ben Ford who wanted an alternative to Django's own JSON / YAML based data loading mechanism. Thanks, Ben! Here is the complete changelog. As usual, you can run easy_install fixture or pip install fixture to get it. Or you can download it from PyPI.

Zed Shaw recently wrote a clear and concise defense for why he used the GPL on Lamson. I've seen a few mentions on twitter that alarmed me because people seemed to think now is the time to release all software as GPL. Here's what you need to ask yourself before you license your code as GPL. First, do you have a business plan that involves selling your software? Most people do not, most business plans have to do with actually using custom software. If that's you then your software and your business become better as more developers work on your software. I.E. when your software is applied to more real world situations, more bugs are fixed and more patches for features are received. GPL does not help you gain users because it shuts out most commercial enterprises ...

Nose 0.11 has just been released. Woo! This has been a long time in the making and got a nice boost from many devs sprinting during PyCon. Here are its nifty new features:

• Vastly more readable documentation with a ton of new documentation added.
• Parallelize your test runs with the --processes switch (note: except on windows, support for which is in the works)
• Output test results in the popular Xunit xml style using --with-xunit; this was designed for Hudson or other CI tools that display or collect stats on test runs.
• Hate dealing with the logging module? So do I! :) Nose now captures logging messages logged during failing tests.
• Re-run only the last batch of failing tests with the --failed switch until you get them to pass. This one was inspired by TestNG.
• Collect tests in all modules without running them.
• Better support for IronPython. (Note that Nose has supported Jython for several versions now.)

Also, for forward thinking types, there is branch to support Python 3k but it's not ready for production use and is not recommended for daily use.

For a Pylons site I have been working on I wanted to provide Ajax functionality for the users but also allow the content to be crawled by search engines. Let me point out that not all of the Ajax content needs to degrade to static HTML, only the content that a user might search for on a search engine. Some people might decide that none of their site needs to be crawled by search engines, say, if it was Gmail or something similar.

Since the site is using Pylons it also uses Mako for templating and on the JavaScript side it uses the Dojo toolkit. The easiest way to render content dynamically (via altering the DOM in JavaScript) and render static HTML seemed to be by sharing my templates between Pylons and Dojo. While probably not the most elegant solution, here's how I did it ...

At PyCon 2009 the fact that Python needs to solve the "packaging problem" came up a few times. This is not a new discussion. However, the problem is still not completely solved so here I'll point out the details of the problem, the unsolved parts, the solved parts, and how the solved parts could be solved better ...

I really like PyCon. It's been said many times but is worth repeating: the hallway track is what makes PyCon such a fun unconference. With that said, here are some happenings:

• I'll be talking Friday after lunch about some fun I've been having trying to test Ajax web applications: Strategies For Testing Ajax. I'm pretty excited about it because there are still a lot of unsolved problems so I'm interested to hear about how other people are testing Ajax.
  • slides are here
  • and a recent video of my talk is here
• There will be a Testing In Python BoF (birds of a feather). Not really sure what we'll do but a lot of people seem interested in it. There have been murmurs of a mock library shootout. Hmm ... I better bring my gun :)
• I'll be on a panel Sunday to discuss Functional Testing Tools in Python. I'll be offering the Nose perspective.
• Ian Bicking will be talking about Topics of Interest which sounds mysterious. As it happened I got a hot lead that there may be ... shall we say ... refreshments to aid in conversation (or instigate heckling?). Don't miss this!
• One of my colleagues Kevin Boers is giving an ambitious talk called Building a Simple Configuration-driven Web Testing Framework With Twill. It's pretty neat.
• Another one of my colleagues, Terry Peppers, is giving a very entertaining talk called A Configuration Comparison in Python
• Too many great talks to mention! I'm psyched about the Windmill talk, Jesse Noller's talk on Multiprocessing, and pretty much everything tagged with testing.

See you there.

Oh! I almost forgot. If you are coming in from out of town, check the wiki for cool restaurants and bars in Chicago. I love this town.

Just a quick note that there is a new version of Fudge, a mock and stub library for Python. This fixes a lot of bugs in the old release and adds some nice new features:

• You can now declare method expectations more expressively using argument inspectors.
• Declaring an expected method order is now possible with fudge.Fake.remember_order().
• Added fudge.Fake.raises() for simulating exceptions.
• Declaring and modifying call chains is easier and more readable now.
• Lots of bugs fixed, all listed in the Changelog.

Thanks for all the feedback thus far. Special thanks to June Kim for testing this release early and providing feedback on the new interfaces.

I keep getting asked why I created yet another Python mock framework. I really didn't want to and explain my motivation here. I am a huge fan of PyPI and would be lost without all the hard work from the open source community but there is always room for more packages. It provides more options to developers and oftentimes rewriting software can be largely rewarding at a small cost. For example, since I wrote Fudge from the ground up I was able to focus on small things like ensuring that all object representations are sane and that exception messages are as informative as possible. Little things like that can be hard to retrofit into an existing library if they were not written right the first time.

I just released 0.9.1 of the Fudge module which is a tool for working with fake objects while testing Python code. Some call these mocks, stubs, or actors, but I just call them all fakes because that way you don't have to change the names in code if you update your tests. You can get Fudge from PyPI or by running easy_install -U fudge. This release contains some nice new features and several contributions by Cristian Esquivias. It has more documentation and some bug fixes but note that some functions have been deprecated.

See the changelog for all new features and details on the deprecations. Big thanks to Cristian for his contributions. Also, thanks goes to Marius Gedminas whose comments on my original Fudge announcement led to better names for some commonly used functions.

There is also an experimental, partially-implemented JavaScript port if you feel adventurous.