./test-run [ OPTIONS ] [ TESTCASE ]
test-run.pl (aka test-run, dtr, mtr) is used to execute tests from the Drizzle test suite. These tests are included with Drizzle distributions and provide a way for users to verify that the system will operate according to expectations.
The tests use a diff-based paradigm, meaning that the test runner executes a test and then compares the results received with pre-recorded expected results. In the event of a test failure, the program will provide output highlighting the differences found between expected and actual results; this can be useful for troubleshooting and in bug reports.
While most users are concerned with ensuring general functionality, the program also allows a user to quickly spin up a server for ad-hoc testing and to run the test-suite against an already running Drizzle server.
There are several different ways to run tests using test-run.pl.
It should be noted that unless --force is used, the program will stop execution upon encountering the first failing test. --force is recommended if you are running several tests - it will allow you to view all successes and failures in one run.
If one only wants to run a few, specific tests, they may do so this way:
./test-run [OPTIONS] test1 [test2 ... testN]
Many of the tests supplied with Drizzle are organized into suites.
The tests within drizzle/tests/t are considered the ‘main’ suite. Other suites are located in either drizzle/tests/suite or within the various directories in drizzle/plugin. Tests for a specific plugin should live in the plugin’s directory - drizzle/plugin/example_plugin/tests
To run the tests in a specific suite:
./test-run [OPTIONS] --suite=SUITENAME
To run a specific set of tests within a suite:
./test-run [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]
Calling tests using <suitename>.<testname> currently does not work. One must specify the test suite via the --suite option.
Currently, the quickest way to execute all tests in all suites is to use ‘make test’ from the drizzle root.
Otherwise, one should simply name all suites:
./test-run [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN
The output of the test runner is quite simple. Every test should pass. In the event of a test failure, please take the time to file a bug here: https://bugs.launchpad.net/drizzle
Example output:
<snip>
main.snowman [ pass ] 9
main.statement_boundaries [ pass ] 17
main.status [ pass ] 12
main.strict [ pass ] 50
main.subselect [ pass ] 6778
main.subselect2 [ pass ] 51
main.subselect3 [ fail ]
drizzletest: At line 621: query 'select a, (select max(b) from t1) into outfile
<snip>
--------------------------------------------------------------------------------
Stopping All Servers
Failed 10/231 tests, 95.67% were successful.
The log files in var/log may give you some hint
of what went wrong.
If you want to report this error, go to:
http://bugs.launchpad.net/drizzle
The servers were restarted 16 times
Spent 64.364 of 115 seconds executing testcases
drizzle-test-run in default mode: *** Failing the test(s): main.exp1
main.func_str main.loaddata main.null main.outfile main.subselect3
main.warnings jp.like_utf8 jp.select_utf8 jp.where_utf8
test-run.pl allows a user to get a Drizzle server up and running quickly. This can be useful for fast ad-hoc testing.
To do so call:
./test-run --start-and-exit [*OPTIONS*]
This will start a Drizzle server that you can connect to and query
Using --start-dirty prevents test-run.pl from attempting to initialize (clean) the datadir. This can be useful if you want to use an already-populated datadir for testing.
test-run.pl uses a simple diff-based mechanism for testing. It will execute the statements contained in a test and compare the results to pre-recorded expected results. In the event of a test failure, you will be presented with a diff:
main.exp1 [ fail ]
--- drizzle/tests/r/exp1.result 2010-11-02 02:10:25.107013998 +0300
+++ drizzle/tests/r/exp1.reject 2010-11-02 02:10:32.017013999 +0300
@@ -5,4 +5,5 @@
a
1
2
+3
DROP TABLE t1;
A test case consists of a .test and a .result file. The .test file includes the various statements to be executed for a test. The .result file lists the expected results for a given test file. These files live in tests/t and tests/r, respectively. This structure is the same for all test suites.
The test-run.pl tool has several available options:
./test-run [ OPTIONS ] [ TESTCASE ]
Use the compressed protocol between client and server
Run the benchmark suite
Run the benchmarks with –small-tests –small-tables
The directory where the benchmark suite is stored (default: ../../mysql-bench)
The directory where temporary files are stored (default: ./var/tmp).
The directory where files generated from the test run is stored (default: ./var). Specifying a ramdisk or tmpfs will speed up tests.
Run testsuite in “memory” using tmpfs or ramdisk Attempts to find a suitable location using a builtin list of standard locations for tmpfs (/dev/shm) The option can also be set using environment variable DTR_MEM =[DIR]
Continue to run the suite after failure
Run test cases which name are prefixed with PREFIX or fulfills REGEX
Skip test cases which name are prefixed with PREFIX or fulfills REGEX
Run test cases starting from test prefixed with PREFIX suite[s]=NAME1,..,NAMEN Collect tests in suites from the comma separated list of suite names. The default is: “main,jp”
Skip the replication test cases. combination=”ARG1 .. ARG2” Specify a set of “drizzled” arguments for one combination.
Skip any combination options and combinations files
How many times to repeat each test (default: 1)
Specify the port number used by the first master
Specify the port number used by the first slave
Specify unique collection of ports. Can also be set by setting the environment variable DTR_BUILD_THREAD.
(Re)genereate the result file for TESTNAME
Check testcases for sideeffects
Log line number and elapsed time to <testname>.progress
Use running server for tests
User for connection to extern server
Start drizzletest client in ddd
Start drizzletest in the selected debugger
Start drizzletest client in gdb
Start drizzled in ddd
Dump trace output for all servers and client programs
Start drizzled in the selected debugger
Start the drizzled(s) in gdb
Let user manually start drizzled in debugger, before running test(s)
Let user manually start drizzled in gdb, before running test(s)
Let user manually start drizzled in ddd, before running test(s)
Specify the master “drizzled” to use
Specify the slave “drizzled” to use
Create strace output for drizzletest client
Limit the number of core files saved (to avoid filling up disks for heavily crashing server). Defaults to 5, set to 0 for no limit.
Todo
See online documentation on how to use it.
Run the drizzletest and drizzled executables using valgrind with default options
Synonym for --valgrind
Run “drizzleslap” with valgrind.
Run the drizzletest and drizzle_client_test executable with valgrind
Run the “drizzled” executable with valgrind
Deprecated, use --valgrind-option
Option to give valgrind, replaces default option(s), can be specified more then once
Path to the valgrind executable
Instruct valgrind to use callgrind
Instruct valgrind to use massif
Write STR to the output
Don’t show test case execution time
Debug this script itself
More verbose output
Only initialize and start the servers, using the startup settings for the specified test case (if any)
Only start the servers (without initialization) for the specified test case (if any)
Don’t try to clean up from earlier runs
Reorder tests to get fewer server restarts
Get this help text
Max test case run time (default 15)
Max test suite run time (default 180)
Pass –log-warnings to drizzled
Passed to drizzletest, will be used as fixed sleep time