The Rational Tester reads reddit.com pretty much on a daily basis. The other day, one of the front page items on was “Why Does Vista Suck? Blame Automated Testing“.
The item points back to a Joel Spolsky presentation given at Yale where Joel stated that:
“one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug” by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP”
I’m not actually running Vista, so I can’t really offer an opinion on whether or not it’s inconsistent and polished. However, even if it is, I have a hard time blaming automation.
Test automation is only one part of a comprehensive quality management program. Test automation is meant to find regressions – things that used to work, but no longer do. Test automation is not designed to find usability bugs. That’s what usability testing is for. The process of regression testing is very time consuming, and grows exponentially with the size of the application. Test automation is meant to reduce the time taken on regression testing, ideally freeing up resources for other types of testing, such as usability testing.
Amazingly, if you read the comments on the blog entry or reddit, you’ll note that many of them are in line with the Rational Tester’s thinking on this. That was probably the most surprising thing about the article!