Thoughts on Testing

I've been at Bluecrew for a month now and one thing I'm actively improving is their continuous integration (CI) strategy. Every team has values around their code and systems. The existential question I ask about each value is this: does CI enforce the value? If not, the value does not exist. You could think of this as an all or nothing CI strategy.

I first heard about test-driven development (TDD) around 2006-ish and I still hear about it often. TDD encourages you to write effective tests because you always see both states: the failing state and the passing state. This is great but it's easy to write the wrong code for a feature as you learn more about it. Duplicating that exploratory effort on tests is a waste of time. After many years of building software on small to medium sized teams, I prefer a leaner approach that accomplishes the same goal: something I call spike, test, break, fix.

I hear a lot about how static typing makes JavaScript safer but what about its influence on code architecture? As someone who has worked extensively with dynamically typed languages, I was surprised to realize that type safe applications can be written in entirely new ways. I'd like to illustrate a fast and effective unit testing strategy that is only possible once you have type safety. I'll be talking about React in TypeScript but this may apply to similar realms...

I wrote an article for Mozilla Hacks about testing React / Redux apps. TL;DR Setting up a test for a Redux connected component is simple: just dispatch the actions needed to enter a desired state. There's no need for mock objects or API calls. If you need to assert that the component dispatches an action in response to a UI event, you can spy on store.dispatch(). Shallow rendering lets you run fast, encapsulated unit tests. For example, breakage in an component (that might be used all across the app) would not break your entire test suite. Testing component interfaces rather than complete end to end coverage improves encapsulation and makes tests more maintainable. For example, if the component under test passes an onSubmit prop to another component, it's better to simulate the integration by calling otherComponent.prop('onSubmit')(). There's no need to fully mount the other component. I recommend using static typing with these testing strategies to ensure all components are integrated correctly.

Fudge, the python mock tool, goes 1.0! You can grab it with pip install -U fudge or directly from PyPI. This marks the end of a long incubation period where the community and I used Fudge in real world scenarios to see what worked and what didn't. I'm sure there are many more improvements to make but as of 1.0 I'm very satisfied with what we've accomplished. This is thanks to its small but vocal community of users, to all contributors and to everyone who pointed out flaws...

If you're building software that will be used by hundreds of millions of people at once it's pretty tricky to simulate that kind of load in a testing environment. And without realistic load tests, you can't be all that sure if your infrastructure will stand up to the pressure. MySpace tried to use 800 EC2 instances to simulate one million concurrent users on their new video features but before they could reach the limit of their own app they hit the physical limits of their Akamai datastore due to the geographic location of the EC2 nodes. D'oh! Instead of simulating load, why not just deploy the feature to see what happens without disrupting usability? Facebook calls this a dark launch of the feature...

The fixture Python module is a utility for managing data needed for automated tests. Its new version, 1.3, adds support for the Django backend. This was a champion effort by Ben Ford who wanted an alternative to Django's own JSON / YAML based data loading mechanism. Thanks...

I've been experimenting with a new tool that was released open source recently called JsTestDriver It's a tool that makes unit testing JavaScript simultaneously on many browsers from the command line very easy for the developer. Actually, there aren't many other tools like it that I know of. It also provides hooks for continuous integration (e.g. Xunit output) and is designed to help you run all unit tests when you click the save button in your code editor. Here are some features it provides that I thought were nice ...

Nose 0.11 has just been released. Woo! This has been a long time in the making and got a nice boost from many devs sprinting during PyCon. Here are its nifty new features: Vastly more readable documentation with a ton of new documentation added. Parallelize your test run...

I really like PyCon. It's been said many times but is worth repeating: the hallway track is what makes PyCon such a fun unconference. With that said, here are some happenings: I'll be talking Friday after lunch about some fun I've been having trying to test Ajax web appli...

Just a quick note that there is a new version of Fudge, a mock and stub library for Python. This fixes a lot of bugs in the old release and adds some nice new features: You can now declare method expectations more expressively using argument inspectors. Declaring an expected...

I just released 0.9.1 of the Fudge module which is a tool for working with fake objects while testing Python code. Some call these mocks, stubs, or actors, but I just call them all fakes because that way you don't have to change the names in code if you update your tests. Yo...

I'm excited to announce the release of Fudge, a Python module for replacing real objects with fakes (mocks, stubs, etc) while testing. Fudge started when a co-worker introduced me to Mocha, a mocking framework for Ruby and a simpler version of jMock (Java). Up to that point I had been building mocks "by hand" in Python with a post mortem approach; I'd set up some fakes then inspect their call stacks at the end of the test. I like the jMock approach better—you declare a fake object that expects methods and arguments, you replace the real object with the fake one, run your code, and you're done. If your code doesn't live up to the expectations then your test fails.

Jens W. Klein has just released a pretty cool doctest debugger tool called interlude. It's designed for a situation where you are writing doctests (perhaps in the comments of your code) and you think to yourself, hmm, what happens when I run this test? Instead of the back and forth run-test-edit cycle, well, why not just drop into a doctest session from your test suite, interact with the shell until you got it right, then copy / paste the session back into your comments? This little tool is genius. And surprisingly simple: 11 lines long (3 of those are for the shell startup message). Jens describes an installation process that involves invoking a custom doctest runner. This can introduce a bootstrapping problem, especially if you are using a doctest runner like Nose because it's hard to customize the doctest runner. Well, actually, this bootstrapping step isn't even necessary. Here's an example ...

I went to my first Chicago JavaScript meetup on Thursday (the 4th meetup they've had so far) and it looks like a promising group. The topics were pretty basic but the speakers seemed very deep into JavaScript so I'm looking forward to future meetings. By raise of hands we decided to split into two groups, an advanced and a beginner, which was an excellent idea. The turnout was incredible — about 60 (or more?) people showed up. The original venue that got canceled only held 40 so some of those people were actually on a waiting list ...

As the US economy spins into a death spiral I unfortunately now know some highly skilled web developers (Python, Ruby, JavaScript, etc) in the Chicago area who are looking for work. And I mean highly skilled people whose ninja like qualities will be sorely missed. Most hiri...

Last year at the Google Test Automation Conference (GTAC) 2007, the talk by Atif was one of my favorites. He had been working on a system of model based testing for desktop GUI applications (affectionately called GUITAR) and hinted that he will be applying the work to web applications next. Now, at GTAC 2008, here is a taste of what his dept. has been up to. The talk is by Oluwaseun Akinmade and Prof. Atif M. Memon. Both are at University of Maryland. The idea of automated model based testing hints at a future where software can be used to figure out how to test itself. That is, when software is modeled in a way that exposes inputs, outputs, event handlers, and end-points, then introspection can be done to find all possible interactions within an application and test them. Yes, it is one step away from artificial intelligence. This is fascinating to me but I think it still needs a lot of work. Atif is asking for as much feedback as possible from industry professionals to find out how this can best be used in the real world. Here are my notes from the talk ...

This was one of the talks at GTAC 2008 that I was most looking forward to before the conference. It was excellent, I was not let down. The talk was given by Markus Clermont and John Thomas who work at Google. Since the talk was right after lunch they decided to take a Q & A approach. It sort of went off in tangents at points but overall the format seemed to work. In my own work I've been struggling at maintaining a now bloated test suite for an AJAX website but their approach made something click in my head. I'm already working on a refactoring plan. Here is my abbreviated interpretation of the talk ...

Google Test Automation Conference (GTAC) is my all-time favorite conference. It's free. It's on a single track — this means you don't miss any talks and everyone experiences the same journey of thought. Also, since you have to apply for admittance with a short essay, everyone who attends is really passionate about testing. It's still sort of "underground" which keeps it small and very social. Last year, I made some kind of attempt to live blog summaries of the GTAC talks but I never made it past part 1. We'll see how far I get this year, stay tuned. The videos for 2008 aren't online yet but check youtube often because last year they were up in less than a day. The Future of Testing was the first talk of the GTAC 2008 conference on Thursday Oct 23rd given by James A. Whittaker, a very entertaining speaker who works for Microsoft. His talk was excellent and I highly recommend keeping a lookout for the video. Here are my notes...

Y'aharrr me seabound mateys! Thar be'a fine gully of'a wind shakin' ye jigger today as m'announce a new release of Fixture, a python module fer loadin' and referencing test data. O'er yonder ye find a cap'n's Changelog fer ye royal subjects. Riches abound! Booty abaft! ...

"The world's largest particle collider passed its first major tests by firing two beams of protons in opposite directions around a 17-mile underground ring ... After a series of trial runs, two white dots flashed on a computer screen at 10:26 a.m. indicating that the pro...

Today was Chicago's Google App Engine Hack-A-Thon and I managed to get some good work done. Well, I had planned to make packages more ephemeral on the PyPi mirror since it quickly hit the 500MB limit as is. But instead I decided to add Datastore support to the fixture m...

I've just pushed 1.0 — the I Heart Data release — of fixture, a Python module for loading and referencing test data. It is used heavily at my work in two test suites: one for the functional tests of an ETL framework and another for a Pylons + Elixir (SQLAlchemy) ...

The Google App Engine SDK sets up a fairly restrictive Python environment on your local machine to simulate their runtime restrictions. Roughly this consists of no access to sockets and thus barely working httplib, urllib, etc (replaced by urlfetch.fetch()), inability to...

Way back in the olden days I used to dabble in Flash for building dynamic websites. This was incredibly painful, as I recall, because I couldn't stand the Flash IDE. It was clunky and hard to navigate, there weren't enough key commands, and the code editor was a sad version...

Leapfrog Online is looking to hire several Python developers to work on a Django site. If you know Python but not Django, this is an excellent opportunity to learn. If you know Django but want to learn how to use Python in other contexts, you'll get to do that, too. You'll...

A while ago I started using Titus Brown's wsgi_intercept module to test some XML services I had been working on. It was a great way to set up a stub response so that the tests never hit our real services. We even created a nifty little decorator that would look for an envir...

I just got back from the GTAC (Google Test Automation Conference) in New York and had a great time. It spanned 2 days and had a single track — this made it very laid back (no headaches trying to decide what talk to attend) and the timing was perfect. Especially since my travel...

Collin Winter just released context_tools, which strikes me as a step towards bridging the gap between test methods in nose and test classes in unittest/py.test/nose/etc. Currently, it can be hard to refactor tests that start as simple test methods when later you decide to u...

UPDATE: Here were my highlights (part 1) I just like saying GTAC "G-TACK," it rolls off the tongue nicely. It's funny when you hear techie things said out loud that you've only see on the screen. At work, I just interviewed a Software Engineer candidate who has used scotc...

In a mad sea of too-busy-to-blink I've managed to write documentation on the fixture module for python. If this interests anyone could he/she please let me know how the docs read? Too much? Too little? Hard to navigate? Examples too complicated? Thanks. For reference,...

While procrastinating on writing documentation for fixture I managed to code up a few nose plugins. (Seriously though, the fixture docs are nearing a stage of completion, I swear it!) If you're not familiar with nose and its nosetests command for running test files, then i...

I've been having some trouble trying to edit my proposal summary (maybe it's not possible anymore?) so I wanted to point out that the talk will cover the fixture module, not the testtools.fixtures module. This is actually a new module, a rewrite of the testtools one from scra...

I'm very pleased to announce that my proposal for a talk at pycon 2007 was accepted (#83). I'm pretty new to pycon but, wow, what a tough time this year the reviewers had! 104 submissions, only 50-60 could be accepted; ouch. The talk is titled You vs. The Real Wor...