Farm Development

Thoughts on Testing

back to all thoughts

Fixture Goes 1.0 (Testing With Data In Python)

posted in Projects, Python, Testing by kumar on Friday Jul 18th, 2008 at 10:51a.m.

I've just pushed 1.0 — the I Heart Data release — of fixture, a Python module for loading and referencing test data. It is used heavily at my work in two test suites: one for the functional tests of an ETL framework and another for a Pylons + Elixir (SQLAlchemy) + Ext JS web application. 

easy_install -U fixture

or download from PyPi

Highlights of This Release

Many thanks to those who submitted issues and patches, especially Manuel Aristarán, sokann, Brian Lee Hawthorne, and Jeffrey Cousens.

What's next? The fixture command that generates DataSet classes from a real database needs some attention. It doesn't work with SQLAlchemy 0.5 yet and could work better with SQLAlchemy based data models in general. Aside from that, fixture seems to have stabilized for my apps at work so I'll be waiting to hear from the community about what other areas to improve on.

Testing Google App Engine sites

posted in Google App Engine, Python, Testing by kumar on Thursday Apr 17th, 2008 at 3:07p.m.

The Google App Engine SDK sets up a fairly restrictive Python environment on your local machine to simulate their runtime restrictions. Roughly this consists of no access to sockets and thus barely working httplib, urllib, etc (replaced by urlfetch.fetch()), inability to use Python C modules, and no access to the file system. The SDK lets you run your app very easily using the dev_appserver.py script but I thought, how the heck would I test my app without the dev server?

It turns out this was ridiculously easy. You just run about 10 lines of code at the start of your test suite. Of course, it might change with an SDK upgrade but here's what worked in my tests/__init__.py in today's SDK (besides sys.path munging): 

import os
from google.appengine.tools import dev_appserver
from google.appengine.tools.dev_appserver_main import *
option_dict = DEFAULT_ARGS.copy()
option_dict[ARG_CLEAR_DATASTORE] = True

def setup():
    ## path to app:
    root_path = os.path.join(os.path.dirname(__file__), '..')
    logging.basicConfig(
        level=option_dict[ARG_LOG_LEVEL],
        format='%(levelname)-8s %(asctime)s %(filename)s] %(message)s')
    config, matcher = dev_appserver.LoadAppConfig(root_path, {})
    ## commented out stuff that checked for SDK updates
    dev_appserver.SetupStubs(config.application, **option_dict)

The setup() is called by nose at the beginning of all tests but if you weren't using nose you could put it at the module level or anywhere else to be called once.

The really cool thing is running tests like this will automatically add indexes for your queries (just like the SDK dev server will) so if you had good code coverage your app would be ready to go live.

Next, you can test your URLs with something like WebTest like so: 

from YOURAPP import application
from webtest import TestApp
def test_index():
    app = TestApp(application)
    response = app.get('/')

...where application is any WSGI app, like one defined in the Hello World tutorial: 

from google.appengine.ext import webapp
class MainPage(webapp.RequestHandler):
  def get(self):
    self.response.out.write('Hello, webapp World!')

application = webapp.WSGIApplication(
                [('/', MainPage)], debug=True)

I haven't tried this for Django, but I'm pretty sure it would work as advertised, making your application object this: 

application = django.core.handlers.wsgi.WSGIHandler()

And there you have it. The two modules used here aren't in the SDK but that's fine because you don't need to upload them to App Engine anyway. You can run easy_install nose WebTest and be ready to test. You were planning on testing your App Engine site, weren't you? :)

If you want to poke around in the test suite I made for pypi.appspot.com, the code is all in the Pypione project, specifically, the tests directory.

UPDATE

I have since realize the above method does not actually simulate all restrictions, it just inserts some stub modules. Specifically, it doesn't simulate restricting imports of a lot of builtin modules, nor does it remove file() and open().

Jason Pellerin is working on a nose plugin, noseGAE (mentioned below in comments), that aims to get all this simulation accurate. It is coming along nicely.

Building Flash/ActionScript sites entirely in code and using FireBug for debugging

posted in ActionScript, Flash, JavaScript, Python, Testing by kumar on Monday Jan 28th, 2008 at 1:09p.m.

Way back in the olden days I used to dabble in Flash for building dynamic websites. This was incredibly painful, as I recall, because I couldn't stand the Flash IDE. It was clunky and hard to navigate, there weren't enough key commands, and the code editor was a sad version of notepad at best. And why use Flash anyway? JavaScript will do most of what you need for a dynamic site nowadays and I believe things like Google AdSense read the DOM for content filtering, which would render a Flash site useless. For me, the answer was audio playback. I wanted to play some audio on a site and for this Flash seems like the only option.

Digging deep into my vault of web dabblery I remembered getting so fed up with the Flash IDE that I wrote an entire site in ActionScript alone. I was quite proud of this as it was an early foray into programming and made me realize how much I enjoyed writing software in code. By some stroke of luck, this ActionScript-only site written circa 2002 is still online! OK, the intro was made with some timeline animation (someone else did that) but everything else is pure code, I swear.

(UPDATE: Me an my big mouth: A few months after I wrote this article the site changed, it's not longer the 2002 version :( But it looks and behaves similar so they might have reused some of my code.)

So there I was, planning out my soon-to-blow-your-mind Flash audio player, yet without Creative Suite 3, the latest Flash IDE. In fact, I didn't want it — $699.00, ouch! Not to mention: the pain of installing another massive, bloated application on my hard drive. A little Googling around later, I discovered Motion-Twin ActionScript 2 Compiler, or mtasc for short, written in OCaml. This was exactly what I was hoping for, an open source command line tool for compiling ActionScript code into SWF (Flash) movies without the need for any clunky IDE. And it works very nicely.

Unfortanately, ActionScript seems to have no error handling whatsoever (or I haven't figured out how to handle errors yet) so doing stupid stuff like calling a method that doesn't exist will not stop program execution as it should. Great. So I quickly became familiar with hooks that mtasc provides to the trace() command. Calling trace() writes to the Flash IDE's debug log. But since you aren't in a Flash IDE when testing a freshly compiled swf, mtasc allows you to define your own implementation of trace. Cool! So naturally, I created my own trace that writes to the FireBug log. FireBug, of course, is the ultimate webapp-debugging tool.

Here it is first added to the example app (from the mtasc tutorial): 

class Tuto {

    static var app : Tuto;

    function Tuto() {
        // creates a 'tf' TextField size 800x600 at pos 0,0
        _root.createTextField("tf",0,0,0,800,600);
        // write some text into it
        _root.tf.text = "Hello world !";
        
        trace("debugging with trace rocks!");
    }

    // entry point
    static function main(mc) {
        // note that this seems to help mtasc find the trace method
        var f = new FireTrace();
        app = new Tuto();
    }
}

...saved to Tuto.as. And here is my trace implementation, saved to FireTrace.as:

class FireTrace {
    static function trace(msg, class_, file, line) {
        getURL('javascript:console.debug("[' + file + ':' + line + ' ' + class_ + '] ' + msg + '")');
    }
}

I compiled it like so: 

mtasc -swf Tuto.swf -trace FireTrace.trace -main -header 800:600:20 Tuto.as

And used a simple HTML page to run it:

<html>
<head>
<script type="text/javascript">

if (!window.console || !console.firebug)
{
    var names = ["log", "debug", "info", "warn", "error", "assert", "dir", "dirxml",
    "group", "groupEnd", "time", "timeEnd", "count", "trace", "profile", "profileEnd"];

    window.console = {};
    for (var i = 0; i < names.length; ++i)
        window.console[names[i]] = function() {}
}

</script>
</head>
<body>
<object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" name="TopPlayer" id="TopPlayer"
     codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=5,0,0,0"
     width="800" height="600">
 <param name="movie" value="Tuto.swf"> 
 <param name="quality" value="high">
 <param name="loop" value="false">
 <param name="play" value="true">
 <!--<param name="bgcolor" value="#ffffff">-->
 <param name="swliveconnect" value="true">

 <embed src="Tuto.swf" quality="high" width="800" height="600"
 	type="application/x-shockwave-flash" name="TopPlayer" id="TopPlayer"
 	pluginspage="http://www.macromedia.com/shockwave/download/index.cgi?p1_prod_version=shockwaveflash"
 	swliveconnect="true">
 </embed>
</object>
</body>
</html>

...and here is what it prints to my FireBug console: 

[Tuto.as:11 Tuto::Tuto] debugging with trace rocks!

Notice that I had to add this line of code to the Tuto class: 

var f = new FireTrace();

I don't know if this is a bug in mtasc but without it the custom trace seems like it never gets loaded. Also notice that I used the FireBug Lite JavaScript snippet for gracefully degrading console methods when FireBug isn't running.

After my romantic reunion with writing ActionScript, what has changed? The first thing I've noticed is that ActionScript 3 now looks exactly Java but I have no idea why. Luckily, it remains compatible with ActionScript 2 so you can leave out all the type declarations and everything still seems to work. Thus, it is pretty much a clone of JavaScript with some extra craziness you probably don't need. I'm still trying to figure out a better way to handle errors; putting debug statements everywhere is pretty lame. If anyone has a suggestion let me know. At the least, I would like to get an error in the log when I stupidly try to call a non-existant method.

Now that I have a prototype, the next step is to write automated tests, of course! This will be a lot of fun as I love watching tests pass. At first glance, the AsUnit (ActionScript Unit Test Runner) seems like it will be useful. I have no doubt that GUI testing Flash is no easier than it is in other languages, and thus not so easy to automate ... but one always needs unit tests.

UPDATE

Ah, looks like there is an even better way to call JavaScript by way of the ExternalInterface class: 

ExternalInterface.call("console.log", variable1, variable2, variableN);

I haven't tried it but this article suggests to limit your calls to String, Number, Array, Object, and Boolean datatypes.

Leapfrog Online is looking for some Django developers (Chicago area)

posted in Chicago, Django, JavaScript, Python, Testing by kumar on Thursday Dec 13th, 2007 at 4:45p.m.

Leapfrog Online is looking to hire several Python developers to work on a Django site. If you know Python but not Django, this is an excellent opportunity to learn. If you know Django but want to learn how to use Python in other contexts, you'll get to do that, too. You'll be working on a high traffic website that hooks into several web services to help customers find Broadband Internet connectivity based on geo location (just US and Canada at the moment). Surrounding that basic function are all kinds of front-end and back-end features, services and systems.

The Software Engineer position is outlined in detail here.

You can send your resume to kumar.mcmillan@gmail.com or send it through the site above. These are full-time positions but if you'd rather work with us as a contractor that may be possible.

What Do We Do?

Leapfrog Online does performance-based customer acquisition, which translates to "we don't make money unless our clients make money." Because of this our software has to work well and we need to collect lots of structured, sensical data so our analysts can build the right marketing strategies. In a more abstract sense, the interesting challenges we face are building high-availability websites, fault-tolerant web services, pushing and pulling at hundreds of gigs of data, and accounting for tight security all along the way. As for the atmosphere, we're still a small company but we're not a struggling startup.

We Care About Open Source

We use open source tools that are right for the job. Currently we use Python or Ruby for websites / web services (Django, Pylons, Paste, Ruby on Rails), Python for backend tools, PostgreSQL for databases, and Trac for our projects. We use rich web interface libraries like Ext JS and we even wrote a distributed content system in Erlang because it was a good fit.

We have contributed patches to most of the projects listed above and maintain our own projects like nose, a few nose plugins, fixture, wsgi_intercept, and sv. We give talks at conferences like Pycon (see #24, #85 and #127). Also, Jeff Cohen (one of our senior developers) runs a popular blog called Softies on Rails and teaches and writes books about Rails.

Scrum: You'll Like It

We started with Extreme Programming a few years ago and have moved towards Scrum and other Agile methods as our approach to software development. We are constantly refining our process, keeping what works, discarding what doesn't. The company is on board with Scrum all the way up to the principles and we are always working to improve how Scrum is integrated holistically (a training program is in the works).

We think you'll like Agile for development. We have several teams of no more than three developers who work in two week "sprints." The sprints are planned out by product owners, developers, and project managers with user stories estimated in "story points" so that the business gets what it needs in order of priority. A sprint is exactly what it sounds like -- you just work! At the end of each sprint the work is released and you attend a retrospective meeting to see what was good, bad, and ugly, and how much work you did. Nothing is perfect so, of course, there are emergencies and derailments here and there but for the most part Scrum keeps things moving at a productive pace. As a developer, I find this discipline empowering and highly motivational.

You Must Test It

We are nutty about automated testing (in case you didn't notice). All code must have automated regression tests so if you're not familiar with this way of writing software, you will learn! We have a fairly involved continuous integration process running in buildbot (though probably moving to Bamboo soon) that performs several builds of each app, one with stable 3rd party libs, the others with trunk versions of 3rd party libs. As well as getting immediate feedback when a bad change is checked in, this also helps us pinpoint bugs in our dependencies before they are released. Our QA department is also different than most in that it consists of developers who are writing functional and/or integration tests in code and adding these to the build process. They are essentially software engineers like the rest of us.

Your Time Is Valuable

No one has a sleeping bad under their desk here; we work until 5 or 6 (weekdays only) to achieve a "sustainable pace." Most of us have been through the "death march" routine at other companies so we know it doesn't work long term. Scrum helps us maintain this ethic.

No Pigeonholes

While we are currently looking for Python/Django programmers, we are always interested in meeting people who think in Ruby/Rails, PHP 5 and other open source web technologies, too. We're especially interested if you're feeling ecumenical and want to learn about and work with, say, both Python and Ruby. You might only work in one language most of the time, but we think it is important for developers to stretch themselves and understand what tools are best for the job.

WSGI Intercept Has A New Home

posted in Projects, Python, Testing by kumar on Monday Dec 3rd, 2007 at 9:46a.m.

A while ago I started using Titus Brown's wsgi_intercept module to test some XML services I had been working on. It was a great way to set up a stub response so that the tests never hit our real services. We even created a nifty little decorator that would look for an environment variable, $USE_REAL_SERVICE, then send the stub response when False or hit a "staged" version of the service when True. This allowed us to transparently perform a slower but more comprehensive test at will (currently running once a day from buildbot).

Although I was able to whip this up pretty quick, I had trouble importing from wsgi_intercept since all its pieces were in disparate modules. No biggie, so I bundled them all together and submitted a patch back to Titus. Me and my big mouth! One thing led to another and I've since agreed to take over maintenance of the project. But seriously, I'm happy to announce that wsgi_intercept has a new home and is available for download from the cheeseshop.

This first release tries to remain true to the interface that twill can work with, although a few import paths probably need changing and I'm not sure how easy it is to use a global version of wsgi_intercept with out-of-the-box twill.

I'm not thrilled about the current interface to wsgi_intercept so the next release will probably deprecate a lot of methods in lieu of a cleaner interface ;) If anyone has ideas for improvement please let me know, preferably as a feature request ticket. Stay tuned. Oh, it also needs some love for python 2.5 so the first person to submit me an issue for all the broken pieces is automatically "awesome."

GTAC Highlights Part 1 - Selenium is Alive and Well, Model Based Testing Is Smart, And...

posted in GTAC, Java, Python, Ruby, Testing, The Future by kumar on Saturday Aug 25th, 2007 at 5:34p.m.

I just got back from the GTAC (Google Test Automation Conference) in New York and had a great time. It spanned 2 days and had a single track — this made it very laid back (no headaches trying to decide what talk to attend) and the timing was perfect. Especially since my traveling managed to dodge one of the worst summer storm systems to hit Chicago in at least a decade!.

I've put together some highlights using the notes I took at each talk. Please bear in mind that this is not a comprehensive report on the conference and may contain misinformation (feel free to comment with corrections). The Google folk did an impressive job of posting video of most talks online within hours. A youtube search for GTAC lists them all. Or ... you can watch them from a playlist

Allen Hutchison - First Principles

Patrick Copeland - Keynote

Simon Stewart - Web Driver for Java

Ryan Gerard and Ramya Venkataramu on Test Hygiene

Matt Heusser & Sean McMillan - Interaction Based Testing

Adam Porter & Atif Memon - Skoll DCQAS

Apple Chow & Santiago Etchebehere - Building an Automated Framework Around Selenium

Doug Sellers - CustomInk Domain Specific Language for automating an AJAX based application

Risto Kumplainin - Automated testing for F-Secure's Linux/UNIX Anti-Virus products

Jennifer Bevan & Jason Huggins - Extending Selenium With Grid Computing

That's all I have time for at the moment. Check back for Part 2 - coming soon!

context_tools, bridging the gap between test methods and test classes?

posted in Python, Testing by kumar on Tuesday Jun 26th, 2007 at 10:31a.m.

Collin Winter just released context_tools, which strikes me as a step towards bridging the gap between test methods in nose and test classes in unittest/py.test/nose/etc. Currently, it can be hard to refactor tests that start as simple test methods when later you decide to use a class for a more complex setUp() function. Specifically, context_tools lets you do: 

class Test(unittest.TestCase):

  setUp, tearDown = context_tools.test_with(foo_bar=my_manager())

  def test_foo(self):
    frobnicate(self.foo_bar)
    self.assertEqual(frob_count, 1)

(an example from the docs)

As far as wrapping a def with an argument to access what data was setup, I also had to do this for the fixture with_data() method.

It would be nice to replace that code with context_tools since it's much simpler! However, after a glance at the code I think it would break the ability to chain together already decorated nose functions. in other words: 

def my_custom_setup():
   whatever = 'fooz'

@nose.tools.with_setup(my_custom_setup)
@dbfixture.with_data(Foo, Bar)
def test_my_data_model(sample_data):
    # here we need my_custom_setup to run
    # but also with_data()
    assert Foo.selectfirst().id == sample_data.Foo.id

Then again, the code I am using internally to accomplish this (scroll to with_data()) is very ugly :(

Going to the GTAC (Google Test Automation Conference)

posted in Python, Testing by kumar on Thursday Jun 21st, 2007 at 11:19a.m.

UPDATE: Here were my highlights (part 1)

I just like saying GTAC "G-TACK," it rolls off the tongue nicely. It's funny when you hear techie things said out loud that you've only see on the screen. At work, I just interviewed a Software Engineer candidate who has used scotch to set up an automated recording/playback environment for testing an AJAX app. I said "yeah this and that, blah blah, WiSGIee, blah blah" and he said, "I've never heard of WiSGIee, what is that?"

"W-S-G-I"

"ohhhhh"

Then I delivered the whole WiSGIee ... scotch ... get it?! joke and that got an even bigger "ohhhhhhhh" :) ar ar, I KILL me. I stole the joke from Titus, of course.

I'm really psyched to be fortunate enough to get an invite to attend the GTAC conference (Google Test Automation Conference) in late August in NY. All the talks look amazing but these look especially interesting :

Since it's free and only 150 spaces for attendees they made us write an essay on why we should attend! sheesh, had me sweating.

I also like how the conference is scheduled on Thursday and Friday leaving the weekend free for roaming the city. Besides seeing some friends I will most definitely be dropping in on the best hole-in-the-wall record shop for dancehall and roots reggae on the planet, Deadly Dragon Sound

documentation for fixture module

posted in Projects, Python, Testing by kumar on Tuesday Apr 17th, 2007 at 1:31p.m.

In a mad sea of too-busy-to-blink I've managed to write documentation on the fixture module for python. If this interests anyone could he/she please let me know how the docs read? Too much? Too little? Hard to navigate? Examples too complicated? Thanks.

For reference, here is the main fixture project page

testing just got easier (a few nose plugins)

posted in Python, Testing by kumar on Thursday Mar 22nd, 2007 at 5:25p.m.

While procrastinating on writing documentation for fixture I managed to code up a few nose plugins. (Seriously though, the fixture docs are nearing a stage of completion, I swear it!)

If you're not familiar with nose and its nosetests command for running test files, then it's worth checking out. Titus Brown even wrote a comprehensive introduction and usage guide.

The coolest part of course is that you can write plugins very easily (installable via easy_install even). Secondly, nosetests is for programmers and ... programmers are motivated to create software to make their life easier! Thus, here are a few useful plugins that myself and others have released lately:

On a separate but related note, in writing these plugins, I came up with a fairly easy way to make functional tests for plugins themselves. It's a combination of two classes, PluginTester and NoseStream; these will probably be part of nose 0.10 but if you want a sneak peak, take a looksee at the nosetrim test suite. The nosetty test suite also makes use of it, but that one is a little more confusing to read because it automates interactaction with the subprocess.

You vs. The Real World: Writing Tests With Fixtures (Sunday at Pycon!)

posted in Python, Testing by kumar on Monday Feb 19th, 2007 at 1:42p.m.

I've been having some trouble trying to edit my proposal summary (maybe it's not possible anymore?) so I wanted to point out that the talk will cover the fixture module, not the testtools.fixtures module. This is actually a new module, a rewrite of the testtools one from scratch, that I started in November when my proposal was accepted.

I'll post the slides up tonight but the gist of the talk is how to use fixture to load and reference test data. Here is an updated summary:

One of the biggest challenges of testing is creating an environment akin to the real world that your code lives in. This talk will focus on general strategies for tackling the problem and then will move into specific examples using the fixture module for setting up and interacting with data stored in databases and other storage media.

The goal of the talk is to promote better code coverage in tests, more maintainable test suites, and techniques for easy and painless refactoring.

You vs. The Real World: Testing With Fixtures (Coming Soon)

posted in Python, Testing by kumar on Thursday Nov 30th, 2006 at 11:34a.m.

I'm very pleased to announce that my proposal for a talk at pycon 2007 was accepted (#83). I'm pretty new to pycon but, wow, what a tough time this year the reviewers had! 104 submissions, only 50-60 could be accepted; ouch.

The talk is titled You vs. The Real World: Testing With Fixtures and is a way to demonstrate some practical usage for the fixtures portion of the testtools module, as well as talk about why testing with real data is highly effective and fairly easy.

Actually, before submitting the proposal I began pulling out the fixtures logic into a new module for distribution, named simply fixture. This will be a sort-of 1.0 of testtools and will allow me to address the many problems I've run into by changing the interface some.

I hope to have the 1.0 release as the new module somewhat stable before my talk (grins) and with also a few new features. I.E. An official release of the command line fixture generator; A better interface for defining rows in a fixture, like cloning a super row and handling id sequences automatically; support for the with statement (this will work like the current @with_fixtures decorator); and better docs, examples, etc.

Recent Projects

  • fixture

    Python module for loading and referencing test data.

  • pypione

    A mirror of PyPi running on Google App Engine

  • Wikir

    converts reStructuredText documents to various Wiki formats.