Thursday, December 04, 2008 10:34 AM
by
ascarb
Web Testing Frameworks Compared
There are four opensource frameworks that I'm going to compare here
Selenium, YUI Test, jsUnit, and WatiN. There are many others available to
use, but these four are the ones that fit in best with the developer workflow
here at Atalasoft. There are only two things that I think every web testing
framework should always do,
1. Not inject
anything into the AUT's (Application Under Test) DOM.
2. Have something like a
WaitForLoaded() method.
For the four frameworks, I'm not going to go into how to use them, or how to
set up your environment to use them, or even a recommendation of which one I think you should use, just a presentation of my perceived
pro's, and cons of each framework.
JsUnit:
|
Pros:
|
Cons:
|
|
Similar api to well known “x”Units we’re all familiar with.
|
Documentation is only sufficient.
|
|
Has good test result logging.
|
Doesn’t integrate easily with nUnit.
|
|
Doesn’t inject itself into the application under test’s DOM.
|
Doesn’t integrate easily with CruiseControl for test result
reporting.
|
|
Is just written in javascript.
|
Doesn’t have a WaitUntilLoaded() like method.
|
|
Is actively being developed.
|
Not easily distributed across many clients.
|
|
Runs in any browser that supports java script.
|
No test recorder.
|
|
Is fast (most of the time).
|
|
WatiN:
|
Every test is just written in C#.
|
Doesn’t have great support for Firefox.
|
|
Is runable in nUnit.
|
No support for Opera, or Safari.
|
|
Well, documentaed API.
|
Is slow.
|
|
Supports all of the latest .Net features (linq, lambda expressions,
etc…).
|
Is not easily distributable to many clients for parallel testing.
|
|
Has great IE support.
|
|
|
Doesn’t inject it self into the application under test’s DOM.
|
|
|
Has a WaitForComplete() method.
|
|
|
Actively being developed.
|
|
|
Reporting test results can be easily integrated through CruiseControl
via the well known nUnit results.
|
|
|
Has a easy to use test recorder.
|
|
YUI Test:
|
Awesome documentation.
|
Injects itself into the AUT’s DOM
|
|
Works anywhere YUI works.
|
Doesn’t have a WaitUntilLoaded like method.
|
|
Comes with YUI.
|
Comes with YUI.
|
|
Is fast.
|
Is not easily distributed to clients for parallel testing.
|
|
Is just a javascript API.
|
Doesn’t easily report test outcomes through CruiseControl.
|
|
API ultimately controlled by Yahoo, and not community.
|
API
ultimately controlled by Yahoo, and not community.
|
|
|
No
test recorder.
|
Selenium:
|
Run by the community, but is now heavily supported by google.
|
Documentation is scattered.
|
|
Expansive browser support.
|
Can be complicated to set up
|
|
Comes in a stand alone, remote control, and grid flavor.
|
Development efforts have stalled in the past.
|
|
Has an easy to use recorder for Firefox only.
|
Can be slow.
|
|
Doesn’t inject itself into the AUT’s DOM.
|
|
|
Has a WaitUntilLoaded() method
|
|
|
Can be fast.
|
|
|
Has a .net compiled C# version ready for nUnit. For that
matter, it supports all “x”Units currently available.
|
|
|
Has an easy to use html based scripting language.
|
|
|
Is distributable, and can run tests in parallel.
|
|