Google Test Automation Conference (GTAC) is my all-time favorite conference. It's free. It's on a single track — this means you don't miss any talks and everyone experiences the same journey of thought. Also, since you have to apply for admittance with a short essay, everyone who attends is really passionate about testing. It's still sort of "underground" which keeps it small and very social.
Last year, I made some kind of attempt to live blog summaries of the GTAC talks but I never made it past part 1. We'll see how far I get this year, stay tuned.
The videos for 2008 aren't online yet but check youtube often because last year they were up in less than a day.
The Future of Testing was the first talk of the GTAC 2008 conference on Thursday Oct 23rd given by James A. Whittaker, a very entertaining speaker who works for Microsoft. His talk was excellent and I highly recommend keeping a lookout for the video. Here are my notes.
- Magic of software
- Decoding DNA, discovering planets in the galaxy that may be earth like
- Technology (special keyboard) in Language of Autism
- Several software models predicted the current economic crisis we're in
- Software will save us. But what will save us from software?!
- Showed a bug in Microsoft map point where directions in Norway were slightly off.
- Showed a bug in a hotel wifi page with a query string containing fee=8.95. Hmmm, how about entering fee=0.0 ? Or fee=-100 !
- Ironically, during the Blue Hat Conference where Microsoft invites hackers to break their software, a user sent an email to the distribution list using Outlook but then recalled the message. This triggered a bug where several exchange servers spiraled for hours trying to retract the message from the 1275 users who were on the distribution list.
- Showed an actual Windows dialog message that simply read: An error occurred: Overflow
- A hint to hackers?!
- Then another dialog message: Stack overflow at line 0
- Marco! Polo!
- The Future
- Showed a video produced by Microsoft that showed a diabetic who was using some heart monitor while running; you see her doctor use a computer screen then send her a message, she enters her house and picks up an E-Ink tablet, starts a video conference with her doctor. There is another scene where a doctor remotely triggers a red circle to appear around a pill bottle on the table in a patient's home. The first woman enters the hospital, thumbs through a digital wallet (virtual credit cards), scans her thumb print.
- James said a bunch of engineers were shown this video. One turned to him and said: "this shit's never gonna work."
- Software testers: how are we going to make this work?
- Summarized his 8 points, The Future I See
- Test Sourcing:
- Insourcing, developers test their software, vendors provide tools
- Outsourcing, vendors provide testing as a service
- Crowdsourcing (pioneered by http://www.utest.com/ ?), just toss it into the crowd, vendors provide testers
- Said this is the future: vendors provide tests.
- There could be a library of common test cases.
- Some service could provide tests for all possible cross-site scripting exploits; just add a web app and run.
- Testers would become designers not implementors.
- Environments would have to be completely portable.
- Must be able to reproduce a bug anywhere
- Virtualization
- Test Sourcing:
- The Vista problem
- Has many tests, used in-house, used in beta program, used by early adopters, but as soon as it shipped, many new major bugs were discovered. Biggest problem was many environments were not tested (i.e. permutations of configuration).
- Visualization
- Told a story of his father while working at Ford. If a car was missing a bumper, everyone agreed it needed a bumper. Fixed. The difference in software is that you can't see it. What is missing? What does it look like?
- We have: interfaces, input, output, data flow, control flow, modules, files, bugs; let's visualize that.
- Showed a very cool heat map of Vista components, in squares relative to significance in the system. The darker squares had more lines of code
- When he started at Microsoft he used this to determine where to focus testing efforts. Components with more dense code need more tests.
- Showed Xbox test tool, a separate window that shows a tester where he/she is in a game (hierarchical tree like view) and what possible scenarios could be entered next.
- Game testers just want to fight monsters, need to help them actually test the game application
- Visualization that showed what parts of the game were most frequently tested by manual game testers (pin points on a map)
- Discovered that everyone liked the wishing well and no one wanted to go on the boats
- The boats were boring! Game designers discovered they needed to work more on the boats. They also temporarily disabled the wishing well during testing.
- Metrics
- There exists a countably infinite number of things to test
- Wants metrics for what's going on right now. I.E. See the javascript code that is currently being executed while hovering over a button. [Titus Brown actually hacked on an interesting experiment like this: a web interface to view runtime code coverage of the Sugar GUI framework for OLPC]
- Education problem
- When engineers are hired out of college they have to be re-trained on how to test code. This is not good. Testing must be taught in computer science programs.