Last year at the Google Test Automation Conference (GTAC) 2007, the talk by Atif was one of my favorites. He had been working on a system of model based testing for desktop GUI applications (affectionately called GUITAR) and hinted that he will be applying the work to web applications next. Now, at GTAC 2008, here is a taste of what his dept. has been up to. The talk is by Oluwaseun Akinmade and Prof. Atif M. Memon. Both are at University of Maryland.
The idea of automated model based testing hints at a future where software can be used to figure out how to test itself. That is, when software is modeled in a way that exposes inputs, outputs, event handlers, and end-points, then introspection can be done to find all possible interactions within an application and test them. Yes, it is one step away from artificial intelligence. This is fascinating to me but I think it still needs a lot of work. Atif is asking for as much feedback as possible from industry professionals to find out how this can best be used in the real world.
Here are my notes from the talk:
- All this applies to Event Driven Software (EDS)
- You have event handlers, dispatchers are registered
- App goes into a collective state where handlers are waiting for events
- Shows a diagram of state changes : S0 -> S1 (triggered by new event) -> S3 (triggered by another event)
- This is a nightmare to test
- You can unit test each event handler
- But you need to test each handler in multiple states to achieve coverage of interaction
- State sucks!
- Atif and team's work is in generating these sequences of events using models
- Test cases grows exponentially
- How can one sample this large space of event interactions?
- Shows demo of the Computer Management app on Windows
- Right clicks and enters the numeric address of a non-existant computer (inputs really long number until the box won't let him type anymore), gets a sensible error message, but when he right clicks and views properties of this new computer, the program crashes
- Want to be able to generate these events in a testing environment
- What do web applications look like?
- Presentation tier, middle tier (server response), data tier, reads from / writes to db
- Need test cases at each level: at db level, at server response
- Need to test through the UI (push button, enter text, etc, through a web browser)
- Shows another example on http://united.com/.
- He logs in with mileage number and password, can see welcome screen. Immediately logs out and this breaks the server (shows 500 error). Says that if he were to perform any other action before logging out then it works fine, so this is a state problem that obviously was not tested.
- Series of events can be very complicated, hard to test all combinations
- Today, we have manual testers
- Shows clip art animation of secretaries doing manual data entry. You can see their head tilt after entering data, Atif jokes that this is how they verify results. Verification is mostly visual.
- Can only maintain very few test cases
- Manual testing is still valuable
- Typically can only test common sequences (people want to do things that they like to do, the boat is boring)
- Can sometimes get a bird brain, someone who is good at breaking stuff. Computers are not always so good at this.
- Test cases are not re-runnable; must start over when app changes
- Today, we can automate browser tests with WebDriver and Selenium
- Tests can be replayed
- Can run on multiple machines
- Can capture / replay (Selenium IDE), store in db or test file, replayed by bot
- Can spread tests across servers, platforms
- Can achieve state with data setup / teardown
- How do you sample the space of all possible event interactions? This is not well understood yet.
- Regression testing (ensuring stable features do not break)
- Atif talks about his work on GUITAR (automated model based testing for GUI apps)
- Can capture / replay with a model in the middle. I.E. using Java Swing (?) can introspect object's accepted input and event handlers
- Shows state diagram: red nodes indicating states and the lines between showing how a user goes from one state to the next.
- Can create a profile of how a user uses an application
- Applies a probabilistic state machine
- Shows example of ZoomIn event in a graphics program. Can analyze state before and after triggering the event.
- Can make events generic and essentially build a "test oracle"
- Can generate test cases (code) that can be executed
- Sneak preview of the next step: applying this to web applications
- Shows event flow graph
- Program will identify all possible events (HTML links, forms, JavaScript events)
- Sometimes a user must perform one event before another so program will need to identify dependencies (chains of events)
- There are common sequences of events that generate faults so one idea is to be able to apply these to any application for better quality control
- Shows event interaction graph; talks about finding the smallest, most efficient paths through algorithms
- Each detected path can be made into a test case
- An incoming arrow indicates the "entry point" events (i.e. user opens a file)
- Shows an event flow graph for the search / replace GUI window in Word Pad -- a huge mess of events, way too complex
- Talks more about GUITAR
- It's a "GUI ripper"
- Depth-first traversal starting at entry point events
- Can stitch event graphs together
- Autodetected relationship are not always accurate so they can be edited later
- Can run tests in parallel
- Has found between 3 and 5 bugs each per sampled SourceForge projects: CrosswordSage, FreeMind, GanttProject, JMSDN
- Working on a similar web ripper
- Wants feedback from those in the industry
- Mentions Xun Yuan (phd student)'s work with GUITAR -- she is now at Google
- Identifies events that interact with each other by observation
- Executes e2 alone, examines state, executes e1+e2, sees if state is different. If so, then these two events probably interacted with each other
- In GUITAR, analyzed feedback between events and found more bugs in each SourceForge app (2 or 3 more bugs per app)
- Oluwaseun Akinmade (Shea) speaks about work done on web application testing
- Shows a simple login page to a website
- Shows state machine for this: you can enter username + password, just username, just password, press submit. You either get one of several errors or you enter the protected site
- Using WebDriver, shows code that enters combinations of events and asserts page title, etc
- Shows a spreadsheet for all possible events and combinations (matrix)
- What are the entry points? Hyperlinks, buttons
- Uses Microsoft translator that converts spreadsheet to AsmL (abstract statement machine language)
- Uses GraphViz (to make model?)
- AsmL code -> apply event seq generator -> gets event sequence and states -> WebDriver commands -> test cases
- The depth first traversal algorithm chooses a path, detects when it has arrived at state with no outgoing transitions
- Demos webdriver running through app, generating test cases
- Demos tool that scans web page to find states. It may need to be edited (same as table before, but in XML format)
- Atif shows Google Maps and asks audience: this is so complicated, we don't know where to begin creating a state machine map; how do we test this?
- Questions from audience ...
- Will it be user pluggable?
- In answer to something, Atif explains he has over 1200 machines to run the observational programs and resulting test cases as they are very cpu intensive
- We need to make web applications more test-friendly, possibly by embedding meta data or just following a common set of best practices