Thoughts on Projects

Fudge, the python mock tool, goes 1.0! You can grab it with pip install -U fudge or directly from PyPI. This marks the end of a long incubation period where the community and I used Fudge in real world scenarios to see what worked and what didn't. I'm sure there are many more improvements to make but as of 1.0 I'm very satisfied with what we've accomplished. This is thanks to its small but vocal community of users, to all contributors and to everyone who pointed out flaws...

I just released 0.9.1 of the Fudge module which is a tool for working with fake objects while testing Python code. Some call these mocks, stubs, or actors, but I just call them all fakes because that way you don't have to change the names in code if you update your tests. You can get Fudge from PyPI or by running easy_install -U fudge. This release contains some nice new features and several contributions by Cristian Esquivias. It has more documentation and some bug fixes but note that some functions have been deprecated.

See the changelog for all new features and details on the deprecations. Big thanks to Cristian for his contributions. Also, thanks goes to Marius Gedminas whose comments on my original Fudge announcement led to better names for some commonly used functions.

There is also an experimental, partially-implemented JavaScript port if you feel adventurous.

Y'aharrr me seabound mateys! Thar be'a fine gully of'a wind shakin' ye jigger today as m'announce a new release of Fixture, a python module fer loadin' and referencing test data. O'er yonder ye find a cap'n's Changelog fer ye royal subjects.

Riches abound! Booty abaft! Me could'a n'er dunnit not be'a the help o'a few fine pirates amidst ye Python vagabonds. Me best salute go t'a Tomas Holas, Alex Marandon, and bslesinsky fer'a ya bug reports and patches. Been'a some quiet waters thus far but much treasure huntin' lies o'er th'horizon.

Plunder Fixture 1.1.1!

(And it be'a fine day fer pirates, aye)

I'm very pleased to announce a website I just launched:

http://aintjustsoul.net/

I'm calling it "a portable record player for the Internet" because I'm a record collector myself and listening to as much music as possible has been the best way to expand my collection. Yep, I'm one of those guys who goes to record conventions and brings a portable record player.

There are a few places to find used vinyl online — since 161,783 records are for sale on eBay right now, I'd say the Internet is a good venue. The problem is that you can't always listen to a record before you buy it and it's impossible to find great music by name alone when there is so much to choose from. I'm hoping to change all that with this site.

As you can see, it's still in very experimental stages and leaves a lot to be desired. eBay is the only marketplace supported so far but I plan to integrate Music Stack next. If anyone knows of another marketplace, please let me know. Also, if you have a friend who buys records then please send on the link (I'm interested to hear feedback from record buyers).

It took me about 6 months to get the data in order (doing all this in my spare time) so it is very exciting to be able to release an actual site now. The data pipeline is home-grown ETL and uses the eBay API + the super speedy and tolerant lxml.html (contrib by Ian Bicking), the excellent SQLAlchemy, and other tools. As I take the data pipeline to the next level I really hope to start using the very cool SnapLogic ETL framework for Python.

On the UI side this has been my first chance to really dive into Dojo. I like Dojo because it takes semantic HTML seriously, namespacing seriously, unit testing seriously, and object oriented JS seriously. It's a bit monolithic and I still haven't figured out how to build a custom minified version but so far I like it. Also, I really wish the Dojo API reference was more comprehensive. On the backend UI side, I'm using Pylons since I like how well it integrates with 3rd party Python libs. I don't like some of the magic it inherited from Rails but that is pretty easy to work around. Oh, I almost forgot to mention SoundManager 2, the ultimate javascript/flash mp3 player.

Today was Chicago's Google App Engine Hack-A-Thon and I managed to get some good work done. Well, I had planned to make packages more ephemeral on the PyPi mirror since it quickly hit the 500MB limit as is. But instead I decided to add Datastore support to the fixture module so that loading sample data is easier when testing an App Engine site. It was very easy to do so I spent most of the time writing some documentation with a complete example for how to go about testing an App Engine site with fixture, WebTest, nose, and NoseGAE.

Here it is: Using Fixture To Test A Google App Engine Site.

Big Thanks to Marzia Niccolai and Mano Marks for trekking out to the Windy City and to all the folks at the Chicago Google office who helped make it happen. As for the hackers, I think we were a shy bunch; there wasn't much show and tell afterwards. I know Ian Bicking got damn close to making a Datastore version of enough builtin file I/O methods for Moin Moin to run on the App Engine. Maybe you'll hear about that soon. Someone did demo a cool iPhone app (literally passed his phone around). It analyzes geo location to provide users with a local message board. I didn't catch the name or a URL — can you drop a comment if you have the info?. UPDATE: The name is Puppyo. Oh yeah, and Harper Reed was hacking on http://excla.im/, an App Engine site that is complimented by a jabber bot allowing you to simply send an IM to post to twitter.

I've just pushed 1.0 — the I Heart Data release — of fixture, a Python module for loading and referencing test data. It is used heavily at my work in two test suites: one for the functional tests of an ETL framework and another for a Pylons + Elixir (SQLAlchemy) + Ext JS web application.

easy_install -U fixture

or download from PyPi

Highlights of This Release

• SQLAlchemy 0.4 now finally works as a nice fixture backend (0.5 beta support is shaky, but mostly there).
• Even more documentation! The new docs have been completely reorganized thanks to the venerable Sphinx. Of special note is a new tutorial that shows how to use fixture to test a Pylons + SQLAlchemy application.

Many thanks to those who submitted issues and patches, especially Manuel Aristarán, sokann, Brian Lee Hawthorne, and Jeffrey Cousens.

What's next? The fixture command that generates DataSet classes from a real database needs some attention. It doesn't work with SQLAlchemy 0.5 yet and could work better with SQLAlchemy based data models in general. Aside from that, fixture seems to have stabilized for my apps at work so I'll be waiting to hear from the community about what other areas to improve on.

Like many of you, I've had my jaw on the floor since the release of Google App Engine. Although there are skeptics out there, a careful read of their terms will show you that it's for real — Google has released GOOGLE to the world and it's not for scary marketing purposes. In fact, I've been growing tired of paranoid Google haters; I'm hoping this will shut them up for a while.

Why is App Engine such a breakthrough? The concept of a hosted web application is nothing new but it has never been done this well. Mundane server maintenance? Gone. Infinite scalability? Check. 100% uptime? Let's face it, if Google went offline you'd probably be down in a nuclear bunker playing Parcheesi.

So ... how should we leverage this tool for the greater good of the community? I can't count the ways without getting dizzy. How about let's start with a mirror of PyPi, the Python Package Index?

PyPi on the App Engine

I barely spent two days on it, but here it is: http://pypi.appspot.com/. Test it out, play with it, try to break it.

As Python grows, especially due to App Engine, PyPi needs to scale too. Zope has put together a PyPi mirror but that's the only other one I know of (actually, I can't even find the link to it right now). Coincidentally, PyPi even went offline for a few min while I was writing this blog post.

Issues...

• urlfetch can't get big packages
  • fetching SQLAlchemy triggered the ResponseTooLarge error but its package was barely over 1MB. We might be able to create a download proxy to work around this.
• May not be fully compatible with easy_install yet
• Can't run scripts on App Engine
  • Not too big a deal, you have to think of web pages as scripts and use a separate crontab to make timed requests. There are security issues with this, but for mirroring pypi it's not a problem
  • Note that I haven't actually mirrored all of PyPi, I'm still testing it out.
• It's not very pretty yet (see below)
• The prototype works, now it needs some tests :)
  • I'm curious to give WebTest a whirl on top of webapp.

You Can Help

I'm not dedicated to this project, I just thought it sounded like a good idea and would be a fun way to experiment with the App Engine. If anyone is interested in working on it just let me know --kumar.mcmillan@gmail.com. If there is enough interest I'll put it on Google Code. Possibly the most exciting feature of App Engine is the Datastore API (aka BigTable) and Ben Bangert agrees. It's a little hard for me to wrap my head around it but so far the Expando class—besides being the coolest name for a class—seems to work great for storing package data. If EGG-INO grows a new parameter, it just gets tacked on to the row dynamically.

This has also been a great way to dig up bugs, some of which have already been fixed.

A while ago I started using Titus Brown's wsgi_intercept module to test some XML services I had been working on. It was a great way to set up a stub response so that the tests never hit our real services. We even created a nifty little decorator that would look for an environment variable, $USE_REAL_SERVICE, then send the stub response when False or hit a "staged" version of the service when True. This allowed us to transparently perform a slower but more comprehensive test at will (currently running once a day from buildbot).

Although I was able to whip this up pretty quick, I had trouble importing from wsgi_intercept since all its pieces were in disparate modules. No biggie, so I bundled them all together and submitted a patch back to Titus. Me and my big mouth! One thing led to another and I've since agreed to take over maintenance of the project. But seriously, I'm happy to announce that wsgi_intercept has a new home and is available for download from the cheeseshop.

This first release tries to remain true to the interface that twill can work with, although a few import paths probably need changing and I'm not sure how easy it is to use a global version of wsgi_intercept with out-of-the-box twill.

I'm not thrilled about the current interface to wsgi_intercept so the next release will probably deprecate a lot of methods in lieu of a cleaner interface ;) If anyone has ideas for improvement please let me know, preferably as a feature request ticket. Stay tuned. Oh, it also needs some love for python 2.5 so the first person to submit me an issue for all the broken pieces is automatically "awesome."

In a mad sea of too-busy-to-blink I've managed to write documentation on the fixture module for python. If this interests anyone could he/she please let me know how the docs read? Too much? Too little? Hard to navigate? Examples too complicated? Thanks.

For reference, here is the main fixture project page