How to Test, by Peek
We here at Peek, at least I think, are good at testing and getting bugs killed… or so I think compared to other places I have been at.
Why do I say that?
1. Everybody tests – even Amol (CEO), even the marketing guys, everybody.
2. We do crazy & weird stress tests to drive stability. Download 1000 emails with < 1% failure rate, open up 100 images in a row, reply all to massive emails 100 times, etc.
3. We use some outsourced expert testing/QA vendors (5-Mobile), coupled with our passionate Peek staff, to ensure we test effectively.
4. We have good (and getting better) test automation & harnesses to provide sheer test-case coverage.
Another example, we realized we needed to improve how we did performance testing. Well several months ago we stepped up, changed how we did performance testing and brought in some outside expertise on embedded performance (www.olivercoady.com). Now our office is littered with flip video cameras and stop watches.
The attitude that many other companies take to testing almost angers me. It is often considered the domain for interns, or developers in another country, or people transfered from the dev team who couldn’t cut it.
If you need quality software then testing is basically the biggest lever you can pull as a manager. In fact, if you are an executive and you want your project to be on-time, stable, with the features working to spec then you need to exert significant energy, resources & budget into testing. Early on you should start banging on your product or project manager and various middle managers for test reports. And you need to get actual working product from them and test it yourself… early! Ask for reports including clear quality statistics/metrics:
1. Mean Time Between Failure – how often does it go between crashes
2. Other failure rates – e.g. in our case we track lost emails, lost sends, lost sockets
3. Performance of key tasks
4. Coverage of functionality
5. Percent pass on your crazy stress cases (e.g. only 100 bytes of free memory available, spam your Peek repeatedly)










