test-run.pl - Drizzle testing tool

Synopsis

./test-run [ OPTIONS ] [ TESTCASE ]

Description

test-run.pl (aka test-run, dtr, mtr) is used to execute tests from the Drizzle test suite. These tests are included with Drizzle distributions and provide a way for users to verify that the system will operate according to expectations.

The tests use a diff-based paradigm, meaning that the test runner executes a test and then compares the results received with pre-recorded expected results. In the event of a test failure, the program will provide output highlighting the differences found between expected and actual results; this can be useful for troubleshooting and in bug reports.

While most users are concerned with ensuring general functionality, the program also allows a user to quickly spin up a server for ad-hoc testing and to run the test-suite against an already running Drizzle server.

Running tests

There are several different ways to run tests using test-run.pl.

It should be noted that unless --force is used, the program will stop execution upon encountering the first failing test. --force is recommended if you are running several tests - it will allow you to view all successes and failures in one run.

Running individual tests

If one only wants to run a few, specific tests, they may do so this way:

./test-run [OPTIONS] test1 [test2 ... testN]

Running all tests within a suite

Many of the tests supplied with Drizzle are organized into suites.

The tests within drizzle/tests/t are considered the ‘main’ suite. Other suites are located in either drizzle/tests/suite or within the various directories in drizzle/plugin. Tests for a specific plugin should live in the plugin’s directory - drizzle/plugin/example_plugin/tests

To run the tests in a specific suite:

./test-run [OPTIONS] --suite=SUITENAME

Running specific tests within a suite

To run a specific set of tests within a suite:

./test-run [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]

Calling tests using <suitename>.<testname> currently does not work. One must specify the test suite via the --suite option.

Running all available tests

Currently, the quickest way to execute all tests in all suites is to use ‘make test’ from the drizzle root.

Otherwise, one should simply name all suites:

./test-run [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN

Interpreting test results

The output of the test runner is quite simple. Every test should pass. In the event of a test failure, please take the time to file a bug here: https://bugs.launchpad.net/drizzle

During a run, the program will provide the user with:
  • test name (suite + name)
  • test status (pass/fail/skipped)
  • time spent executing each test
At the end of a run, the program will provide the user with a listing of:
  • how many tests were run
  • how many tests failed
  • percentage of passing tests
  • a listing of failing tests
  • total time spent executing the tests

Example output:

<snip>
main.snowman                                                 [ pass ]       9
main.statement_boundaries                                    [ pass ]      17
main.status                                                  [ pass ]      12
main.strict                                                  [ pass ]      50
main.subselect                                               [ pass ]    6778
main.subselect2                                              [ pass ]      51
main.subselect3                                              [ fail ]
drizzletest: At line 621: query 'select a, (select max(b) from t1) into outfile
<snip>
--------------------------------------------------------------------------------
Stopping All Servers
Failed 10/231 tests, 95.67% were successful.

The log files in var/log may give you some hint
of what went wrong.
If you want to report this error, go to:
    http://bugs.launchpad.net/drizzle
The servers were restarted 16 times
Spent 64.364 of 115 seconds executing testcases

drizzle-test-run in default mode: *** Failing the test(s): main.exp1
main.func_str main.loaddata main.null main.outfile main.subselect3
main.warnings jp.like_utf8 jp.select_utf8 jp.where_utf8

Additional uses

Starting a server for manual testing

test-run.pl allows a user to get a Drizzle server up and running quickly. This can be useful for fast ad-hoc testing.

To do so call:

./test-run --start-and-exit [*OPTIONS*]

This will start a Drizzle server that you can connect to and query

Starting a server against a pre-populated DATADIR

Using --start-dirty prevents test-run.pl from attempting to initialize (clean) the datadir. This can be useful if you want to use an already-populated datadir for testing.

Program architecture

test-run.pl uses a simple diff-based mechanism for testing. It will execute the statements contained in a test and compare the results to pre-recorded expected results. In the event of a test failure, you will be presented with a diff:

main.exp1                                                    [ fail ]
--- drizzle/tests/r/exp1.result     2010-11-02 02:10:25.107013998 +0300
+++ drizzle/tests/r/exp1.reject     2010-11-02 02:10:32.017013999 +0300
@@ -5,4 +5,5 @@
a
1
2
+3
DROP TABLE t1;

A test case consists of a .test and a .result file. The .test file includes the various statements to be executed for a test. The .result file lists the expected results for a given test file. These files live in tests/t and tests/r, respectively. This structure is the same for all test suites.

test-run.pl options

The test-run.pl tool has several available options:

./test-run [ OPTIONS ] [ TESTCASE ]

Options to control what engine/variation to run

--compress

Use the compressed protocol between client and server

--bench

Run the benchmark suite

--small-bench

Run the benchmarks with –small-tests –small-tables

Options to control directories to use

--benchdir=DIR

The directory where the benchmark suite is stored (default: ../../mysql-bench)

--tmpdir=DIR

The directory where temporary files are stored (default: ./var/tmp).

--vardir=DIR

The directory where files generated from the test run is stored (default: ./var). Specifying a ramdisk or tmpfs will speed up tests.

--mem

Run testsuite in “memory” using tmpfs or ramdisk Attempts to find a suitable location using a builtin list of standard locations for tmpfs (/dev/shm) The option can also be set using environment variable DTR_MEM =[DIR]

Options to control what test suites or cases to run

--force

Continue to run the suite after failure

--do-test=PREFIX or REGEX

Run test cases which name are prefixed with PREFIX or fulfills REGEX

--skip-test=PREFIX or REGEX

Skip test cases which name are prefixed with PREFIX or fulfills REGEX

--start-from=PREFIX

Run test cases starting from test prefixed with PREFIX suite[s]=NAME1,..,NAMEN Collect tests in suites from the comma separated list of suite names. The default is: “main,jp”

--skip-rpl

Skip the replication test cases. combination=”ARG1 .. ARG2” Specify a set of “drizzled” arguments for one combination.

--skip-combination

Skip any combination options and combinations files

--repeat-test=n

How many times to repeat each test (default: 1)

Options that specify ports

--master_port=PORT

Specify the port number used by the first master

--slave_port=PORT

Specify the port number used by the first slave

--dtr-build-thread=#

Specify unique collection of ports. Can also be set by setting the environment variable DTR_BUILD_THREAD.

Options for test case authoring

--record TESTNAME

(Re)genereate the result file for TESTNAME

--check-testcases

Check testcases for sideeffects

--mark-progress

Log line number and elapsed time to <testname>.progress

Options that pass on options

--drizzled=ARGS

Specify additional arguments to “drizzled”

Options to run test on running server

--extern

Use running server for tests

--user=USER

User for connection to extern server

Options for debugging the product

--client-ddd

Start drizzletest client in ddd

--client-debugger=NAME

Start drizzletest in the selected debugger

--client-gdb

Start drizzletest client in gdb

--ddd

Start drizzled in ddd

--debug

Dump trace output for all servers and client programs

--debugger=NAME

Start drizzled in the selected debugger

--gdb

Start the drizzled(s) in gdb

--manual-debug

Let user manually start drizzled in debugger, before running test(s)

--manual-gdb

Let user manually start drizzled in gdb, before running test(s)

--manual-ddd

Let user manually start drizzled in ddd, before running test(s)

--master-binary=PATH

Specify the master “drizzled” to use

--slave-binary=PATH

Specify the slave “drizzled” to use

--strace-client

Create strace output for drizzletest client

--max-save-core

Limit the number of core files saved (to avoid filling up disks for heavily crashing server). Defaults to 5, set to 0 for no limit.

Options for coverage, profiling etc

Todo

--gcov
--gprof

See online documentation on how to use it.

--valgrind

Run the drizzletest and drizzled executables using valgrind with default options

--valgrind-all

Synonym for --valgrind

--valgrind-drizzleslap

Run “drizzleslap” with valgrind.

--valgrind-drizzletest

Run the drizzletest and drizzle_client_test executable with valgrind

--valgrind-drizzled

Run the “drizzled” executable with valgrind

--valgrind-options=ARGS

Deprecated, use --valgrind-option

--valgrind-option=ARGS

Option to give valgrind, replaces default option(s), can be specified more then once

--valgrind-path=[EXE]

Path to the valgrind executable

--callgrind

Instruct valgrind to use callgrind

--massif

Instruct valgrind to use massif

Misc options

--comment=STR

Write STR to the output

--notimer

Don’t show test case execution time

--script-debug

Debug this script itself

--verbose

More verbose output

--start-and-exit

Only initialize and start the servers, using the startup settings for the specified test case (if any)

--start-dirty

Only start the servers (without initialization) for the specified test case (if any)

--fast

Don’t try to clean up from earlier runs

--reorder

Reorder tests to get fewer server restarts

--help

Get this help text

--testcase-timeout=MINUTES

Max test case run time (default 15)

--suite-timeout=MINUTES

Max test suite run time (default 180)

--warnings | log-warnings

Pass –log-warnings to drizzled

--sleep=SECONDS

Passed to drizzletest, will be used as fixed sleep time