Selenium's uses & flaws
Selenium is a tool for looking at an installation and trying to
break it and filing bug reports, integration testing. But automated integration
testing, as we've now experienced, frequently breaks; it's brittle, not robust.
Selenium is an automated tests framework suited to integration testing: you
start Selenium and it fires up a browser, performs scripted actions as a user
would, and checks the actual output against the desired output. But that means
that we have to define the desired output in a way Selenium can
programmatically test, and so the tests break when we change things that actual
users wouldn't care about.
For example, a test might define success as "this DIV
should have the value 1". If we change the skin to output a SPAN element
instead of a DIV element, then the test will fail. And even if a failed test is
a legitimate signal of a problem, we have to start from scratch investigating
what to fix and how.
Installation
SeleniumFramework has the following major dependencies:
Selenium
SeleniumFramework curently uses Selenium Remote
Control (Selenium RC) (and also should use Selenium Grid in the
future.)
§
Download Selenium RC from http://seleniumhq.org/download
§
Installation instructions for Selenium RC can be found at http://seleniumhq.org/docs/05_selenium_rc.html#installation
PHPunit
Install PHPUnit via PHP's command line PEAR client
# if needed, add
the PEAR channels for PHPUnit and its dependencies
pear channel-discover pear.phpunit.de
pear channel-discover components.ez.no
pear channel-discover pear.symfony-project.com
# Install PHPUnit
pear install phpunit/PHPUnit
Testing_Selenium
Testing_Selenium is a PHP client for the Selenium Remote Control
test tool.
§
Install Testing_Selenium via PHP's command
line PEAR client
# you can try "pear
install Testing_Selenium", but it may give you an error if you don't
specify the stable version
pear install Testing_Selenium-0.4.3
PHPUnit and Testing_Selenium must be accessible within your PHP
path. If your PEAR installation is properly configured, then this should
already be the case.
Configuration
Create
a selenium_settings.ini file
The selenium_settings.ini file configures various
settings for SeleniumFramework, such as the browsers that Selenium will attempt
to use in testing and the base URL of the to be tested.
selenium_settings.ini must be created in the root or your
Media installation and should use one of the provided template files
from tests/selenium. The template to use depends on the Selenium project
(and, possibly, version of PHP) that you are running.
§
If you are running Selenium RC with PHP 5.2,
copy tests/selenium/selenium_settings_grid.ini.php52.sample into selenium_settings.ini.
Once you have created selenium_settings.ini, you will need
to customize the settings it contains. At a minimum, you will need to set the
URL configuration value to match the base URL of the Media instance you
are testing.
Update
your LocalSettings.php file
Once the preceding steps are complete, you will need to update
your LocalSettings.php file to enable SeleniumFramework.
To enable SeleniumFramework, you will need to add the following
code to LocalSettings.php:
$wgEnableSelenium = true;
$wgSeleniumConfigFile = $IP . '/selenium_settings.ini';
§
Hint: Depending on the folder where the tests are run, the
selenium config file locator should point from the root.
Loading test
configurations and suites
Once SeleniumFramework has been enabled, you'll need to tell it
what tests to run.
To do this, you'll need to add a bit more code
to LocalSettings.php.
The following examples show how to load various test suites.
# Basic example
require_once(
"tests/selenium/suites/SimpleSeleniumConfig.php" );
$wgSeleniumTestConfigs['SimpleSeleniumTestSuite'] =
'SimpleSeleniumConfig::getSettings';
require_once("extensions/Editor/tests/selenium/EditorSeleniumConfig.php");
$wgSeleniumTestConfigs['EditorTestSuite'] =
'EditorSeleniumConfig::getSettings';
Multiple Media instances
The configuration above assumes a single for everything. It is possible
to have two separate (one driving the tests, and one being tested). More
information can be found in Selenium Configuration.
Writing tests
Adding a test suite
To add a Selenium test, you must first create a test suite class
that extends class SeleniumTestSuite. Look
at SimpleSeleniumTestSuite.php for an example (which is in
thetests/selenium/suites directory of the source.
Add test cases to the suite by implementing
the addTests() function. If you have per-suite setup and teardown
code, override the setUp() and tearDown() functions.
class MyExtensionTestSuite extends SeleniumTestSuite
{
# Define only if you have
per-suite setup code
public function setUp(){
# Add setUp code here
parent::setUp();
}
public function addTests() {
$testFiles = array(
'extensions/MyExtension/tests/selenium/SimpleSeleniumTestCase.php',
'extensions/MyExtension/selenium/FooBarTestCase.php'
);
parent::addTestFiles( $testFiles );
}
# Define only if you have
per-suite teardown code
public function teardown(){
# Add setUp code here
parent::tearDown();
}
}
Adding individual tests
In each test file listed in addTests(), create a class that
extends SeleniumTestCase. Name the class after the file that contains the
class, omitting the extension. For example: A file
named SimpleSeleniumTestCase.php should contain
class SimpleSeleniumTestCase.
For each test that you wish to run, add a public function to the
class. Each function name should start with test. For
example: testWidget().
Test functions should contain no more than one call to any of
the various assert functions
(like assertEmpty() or assertEquals()). The reason for this is
that the logger only reports whether or not a test passed, not which assertions
in the test passed or failed. If there are multiple assertions in a test
function and the test fails, it will be impossible to know which assertion
caused the test to fail.
If you have per-test setup and teardown code, override
the setUp() and tearDown() functions.
setUp() and tearDown() are run for every test – not just at the beginning and end of all test runs.
class SimpleSeleniumTestCase extends SeleniumTestCase
{
public function testBasic( ) {
$this->open( $this->getUrl() . '/index.php?title=Selenium&action=edit'
);
$this->type( "wpTextbox1", "This is a basic
test" );
$this->click( "wpPreview" );
$this->waitForPageToLoad( 10000 );
// check result
$source = $this->getText( "//div[@id='Preview']/p"
);
$correct = strstr( $source, "This is a basic test" );
$this->assertEquals( $correct, true );
}
public function testGlobalVariable1( ) {
$this->open( $this->getUrl() .
'/index.php?&action=purge' );
$bodyClass = $this->getAttribute( "//body/@class"
);
$this-> assertContains('skin-vector', $bodyClass, 'Vector
skin not set');
}
}
Examples
For a very basic sample test suite and test case look in
tests/selenium/SimpleSeleniumTestSuite.php and
tests/selenium/SimpleSeleniumTestCase.php
§
You will need the following in the selenium_settings.ini file to
run these tests
[SeleniumTests]
testSuite[SimpleSeleniumTestSuite] =
"tests/selenium/SimpleSeleniumTestSuite.php"
§
A working example can also be found at the PagedTiffHandler
extension.
Test configuration
Multiple test suites that need different configurations are run
against a single test.
Recording tests with
Selenium IDE
You can record tests with Selenium IDE and still use them with
the framework. In order to do so, you have to follow these steps:
§
Record the test with Selenium IDE
§
Export the test to PHP
§
The IDE creates a method within the test class which has the
name of the test. Copy the inner code of this method (not the function head)
into the runTest method of SimpleSeleniumTestCase.
§
Of course, rename SimpleSeleniumTestCase to a more descriptive
name.
Running tests without
Selenium Framework
In some cases, Selenium Framework will not be available on the
under test. This is especially the case when testing the installer. In this
case, you need to tell the test not to use the dynamic reconfiguration and not
to try to log into the under test. This is done in the test suite like this:
class MyExtensionTestSuite extends SeleniumTestSuite
{
public function setUp(){
# Add setUp code here
parent::setUp();
$this->setTriggerClientTestResources( false );
$this->setLoginBeforeTests( false );
}
}
Running Tests
To run tests, run "php tests/RunSeleniumTests.php"
from the root directory.
§
Old way (now, tests are in tests/...): To run tests, run
"php maintenance/tests/RunSeleniumTests.php" from the root directory.
Requirements
The test should have a default configuration. This is the state
that test suites should assume for a test before setUp() is called.
§
Each test suite needs to tell the test what configuration
it needs. This may be
§
files to be included for an extension
§
global configuration variables
§
Could also be used to switch out the db and images directory.
(Do we need more fine grained control, like per test instead of
per test suite?)
§
At the end of a test suite, needs to go back to its original
default configuration.
Missing
Concurrent test suites point to the same
§
Whether this will work to change the db and images directory for
the test .
§
If any extensions do custom configuration that is not an include
or a global config variable.
Implementation details
Test suite:
§
The test suite's setUp() makes a request to the test with a
request parameter whose value is the test suite name.
§
After this the test can assume that the the test is
configured correctly for this test suite.
§
The test suite's tearDown() sends another request parameter
which will tell the test to do back to using the default configuration.
Test:
The following happens in WebStart.php only if a global variable $wgEnableSelenium is set.
The following happens in WebStart.php only if a global variable $wgEnableSelenium is set.
§
If request param setupTestSuite is set
§
set a cookie containing the test suite name
§
cookie is set to expire in 10 mins (Should this be configurable)
§
If request param clearTestSuite is set
§
clear the cookie containing the test suite name
§
If we find our testSuiteName cookie
§
Look for a global variable $wgSeleniumTestConfigs[testSuiteName].
If found, this is the test suite's function that is responsible for providing
the following.
the test specific include files
the test specific globals (key value pairs)
§
$wgSeleniumTestConfigs[testSuiteName] needs to be set in LocalSettings.php.
The reason this is not a done via a hook is because this configuration needs to
happen before Setup.php.
§
In the case that it finds neither of the request params
mentioned above nor the cookie, the Setup will proceed as usual with only the
default configuration.
Test configuration
example:
As an example see extensions/tests/selenium
As an example see extensions/tests/selenium
§
LocalSettings.php should contain
EditorSeleniumConfig::getSettings returns an array of
include files and configuration variables needed by the extension.
§
If a cookie has been set, indicating this is part of the
TestSuite test run, we use the configuration we get
fromSeleniumConfig::getSettings
Architecture
These files are part of the framework:
§
tests/RunSeleniumTests.php includes
test suites specified in extension directories.
§
tests/Selenium.php provides
access to selenium and implements common tasks.
§
tests/selenium/SeleniumLoader.php loads
all the neccessary classes for the framework.
§
tests/selenium/SeleniumTestSuite.php starts
and stops selenium tests.
§
tests/selenium/SeleniumTestCase.php provides
some additional assertions.
§
tests/selenium/SeleniumTestListener.php is
the interface to result logging.
§
tests/selenium/SeleniumConsoleLogger.php and tests/selenium/SeleniumHTMLLogger.php produce
the actual output.
The tests should be located here:
§
Tests for core should be placed in
the tests/selenium folder.
§
Extensions should place their tests in extensions/EXTENSION/tests/selenium
Testing with a clean database and file state
testrunner
start selenium which in turn starts a browser to talk to the
under test,send request for new test with unique test id and tests that will be
fired ,create cookie with test id, create temporal resources, according to
tests list, create test tracker with timestamp, return success code, start
testsuites via selenium,send a lot of individual,requests according to the test
testrunner is identified by test id, reconfigure database and
resources,according to test id, Do something with memcached ,execute
request,update timestamp in test tracker,send a teardown request,execute
teardown, delete, all resources associated with test id,delete test
tracker,return success code,stop selenium,Test style guide.
This section focuses on general test style issues specific to
Selenium Framework. For general issues of Selenium test style, see
the test design considerations section of the Selenium
documentation.
Leave no trace
A test should leave at best no traces.
§
If this is not possible, the test should leave in a state which
allows the test to be re-run.
Avoid brittle paths
Many developers will use the Selenium IDE to draft
tests. This approach allows developers to rapidly rough out tests, but may
generate locators that fail when used on other platforms or that are brittle
and break when exposed to trivial edge cases.
When possible, avoid absolute paths. Instead, start with an
element ID and work relative to the ID.
File naming conventions
The names for the test and configuration files should follow
these conventions:
§
All files should have a .php extension.
§
File names for configuration files should end with Config.
For example: SimpleSeleniumConfig.php
§
File names for test suite files should end with TestSuite.
For example: SimpleSeleniumTestSuite.php
§
File names for test files should end with TestCase. For example: SimpleSeleniumTestCase.php
Keep log output in mind
Do not expect someone to read your test code. If a test fails,
people will most likely read a log file. It is therefore very helpful to pay
attention to descriptive log lines.
Do not log with echo
Often, the output of tests is processed further, e.g. by a
continuous integration server. In this case, everything that is logged
via echo will be lost. More particular, the framework is equipped to
produce xml output compatible to phpunit, and echo messages are ignored in this
file.
Test cases vs. suites
In order to get a differenciated logging, it is possible to use
several test methods per test case, each of which produces an individual line
in the log. Unfortunately, the setUp method is called for each of
this methods, which makes perfect sense in phpunit but is not a behaviour we
want with selenium. So instead, we use a
method testSetUp() and testTearDown(), which are called only
once per test case. Here is an outline of such an architecture:
class GeneralTestCase extends SeleniumTestCase
{ ...
public function testSetUp() {
}
public function testFactboxExists() {
}
depends
testFactboxExists
public function testFactboxContainsRightStrength()
{
}
}
Here, you can even build in dependencies (see comment in the
code above), i.e. if a test fails, succeeding tests will not be executed if
they are dependent.
Number of assertions
Ideally, there is one assertion per test. However, this is often
not manageable in reality. But it is possible to add a log line to an assertion
that will be used in logs and xml output:
$this->assertTrue(
$this->isElementPresent("body"), "Element body is not
present" );
Notes and further improvements
Add common test tasks
§
Trigger user preferences
§
Edit a page (initial support already implemented)
§
Delete a page
§
Upload images (already in PagedTiffHandler tests)
§
Show edit and history mode
§
Revert pages
§
Refresh pages
§
Cross Browser testing
§
Stress / Load partial testing
§
Please add to this
list...
Add common assertions
Text within an article
§
Text in Headlines
§
Text in Table of Contents
§
Error messages
§
External link
§
Internal link
§
Please add to this list
§
Training Framework
Improvements
Make test result logging more talkative
§
Check prerequisites of configuration (initial support in
PagedTiffHandler tests)
§
Find a way to execute selenes recorded via Selenium IDE
§
Refactor naming conventions
§
Ways to configure for specific test suites or extensions.
§
A way to bring the database to a clean initial state before each
test
§
write collaborative Selenium tests
Further ideas
A test (or a single test suite) should be able to reconfigure
the according to its needs. Here, some serious security issues arise. This
might be a possible way:
§
Add a hook 'LocalSettingsEnd' at the end of LocalSettings.php
§
Add some URL parameter which indicates which test is being run
§
Within the extension, the hook code now changes the
configuration
If the hook is not triggered in LocalSettings, no
reconfiguration can take place.
Troubleshooting
Error about
Testing_Selenium (0.4.3 is installed from pear along with PHPUnit) not being
found
Replace /PEAR/ by the actual path to the repository
General Tips
Ignore port A proxy time you out port 4444.
§
Uncomment any tests in RunSeleniumTests.php to run different
test suites?
Backporting
Currently, the framework is only available in svn trunk and will
not be available in a packaged version before MW 1.17. In order to make it run
with older versions, you need to do this:
§
get a fresh svn checkout
§
copy tests/RunSeleniumTests.php and tests/selenium to your tests
folder
§
copy maintenance/Maintenance.php and
maintenance/doMaintenance.php to your maintenance folder
§
copy includes/SeleniumWebSettings.php to your includes folder
§
edit SeleniumWebSettings.php and add these lines after
LocalSettings.php is included (surrounding lines are given for context, in
MW1.17 this starts at l. 113):
# Include site settings.
$IP may be changed (hopefully before the AutoLoader is invoked)
require_once( MW_CONFIG_FILE );
}
if ( $wgEnableSelenium ) {
require_once( "$IP/includes/SeleniumWebSettings.php"
);
}
wfProfileOut( 'WebStart.php-conf' );
Experiences: Recent changes in handling of framework
PEAR PHPUnit and PEAR Testing_Selenium: Check whether
properly installed, esp. PHPUnit/Framework.php
§
Installation of selenium-rc is pretty easy:
§
Download selenium rc
§
start "java -jar selenium-server.jar"
No comments:
Post a Comment