Package test :: Module regrtest
[hide private]
[frames] | no frames]

Module regrtest

Regression test.

This will find all modules whose name is "test_*" in the test
directory, and run them.  Various command line options provide
additional facilities.

Command line options:

-v: verbose    -- run tests in verbose mode with output to stdout
-w: verbose2   -- re-run failed tests in verbose mode
-q: quiet      -- don't print anything except if a test fails
-g: generate   -- write the output file for a test instead of comparing it
-x: exclude    -- arguments are tests to *exclude*
-s: single     -- run only a single test (see below)
-r: random     -- randomize test execution order
-f: fromfile   -- read names of tests to run from a file (see below)
-l: findleaks  -- if GC is available detect tests that leak memory
-u: use        -- specify which special resource intensive tests to run
-h: help       -- print this text and exit
-t: threshold  -- call gc.set_threshold(N)
-T: coverage   -- turn on code coverage using the trace module
-D: coverdir   -- Directory where coverage files are put
-N: nocoverdir -- Put coverage files alongside modules
-L: runleaks   -- run the leaks(1) command just before exit
-R: huntrleaks -- search for reference leaks (needs debug build, v. slow)
-M: memlimit   -- run very large memory-consuming tests

If non-option arguments are present, they are names for tests to run,
unless -x is given, in which case they are names for tests not to run.
If no test names are given, all tests are run.

-v is incompatible with -g and does not compare test output files.

-T turns on code coverage tracing with the trace module.

-D specifies the directory where coverage files are put.

-N Put coverage files alongside modules.

-s means to run only a single test and exit.  This is useful when
doing memory analysis on the Python interpreter (which tend to consume
too many resources to run the full regression test non-stop).  The
file /tmp/pynexttest is read to find the next test to run.  If this
file is missing, the first test_*.py file in testdir or on the command
line is used.  (actually tempfile.gettempdir() is used instead of
/tmp).

-f reads the names of tests from the file given as f's argument, one
or more test names per line.  Whitespace is ignored.  Blank lines and
lines beginning with '#' are ignored.  This is especially useful for
whittling down failures involving interactions among tests.

-L causes the leaks(1) command to be run just before exit if it exists.
leaks(1) is available on Mac OS X and presumably on some other
FreeBSD-derived systems.

-R runs each test several times and examines sys.gettotalrefcount() to
see if the test appears to be leaking references.  The argument should
be of the form stab:run:fname where 'stab' is the number of times the
test is run to let gettotalrefcount settle down, 'run' is the number
of times further it is run and 'fname' is the name of the file the
reports are written to.  These parameters all have defaults (5, 4 and
"reflog.txt" respectively), so the minimal invocation is '-R ::'.

-M runs tests that require an exorbitant amount of memory. These tests
typically try to ascertain containers keep working when containing more than
2 billion objects, which only works on 64-bit systems. There are also some
tests that try to exhaust the address space of the process, which only makes
sense on 32-bit systems with at least 2Gb of memory. The passed-in memlimit,
which is a string in the form of '2.5Gb', determines howmuch memory the
tests will limit themselves to (but they may go slightly over.) The number
shouldn't be more memory than the machine has (including swap memory). You
should also keep in mind that swap memory is generally much, much slower
than RAM, and setting memlimit to all available RAM or higher will heavily
tax the machine. On the other hand, it is no use running these tests with a
limit of less than 2.5Gb, and many require more than 20Gb. Tests that expect
to use more than memlimit memory will be skipped. The big-memory tests
generally run very, very long.

-u is used to specify which special resource intensive tests to run,
such as those requiring large file support or network connectivity.
The argument is a comma-separated list of words indicating the
resources to test.  Currently only the following are defined:

    all -       Enable all special resources.

    audio -     Tests that use the audio device.  (There are known
                cases of broken audio drivers that can crash Python or
                even the Linux kernel.)

    curses -    Tests that use curses and will modify the terminal's
                state and output modes.

    largefile - It is okay to run some test that may create huge
                files.  These tests can take a long time and may
                consume >2GB of disk space temporarily.

    network -   It is okay to run tests that use external network
                resource, e.g. testing SSL support for sockets.

    bsddb -     It is okay to run the bsddb testsuite, which takes
                a long time to complete.

    decimal -   Test the decimal module against a large suite that
                verifies compliance with standards.

    compiler -  Test the compiler package by compiling all the source
                in the standard library and test suite.  This takes
                a long time.  Enabling this resource also allows
                test_tokenize to verify round-trip lexing on every
                file in the test library.

    subprocess  Run all tests for the subprocess module.

    urlfetch -  It is okay to download files required on testing.

To enable all resources except one, use '-uall,-<resource>'.  For
example, to run all the tests except for the bsddb tests, give the
option '-uall,-bsddb'.

Classes [hide private]
_ExpectedSkips
Functions [hide private]
 
usage(code, msg='')
 
main(tests=None, testdir=None, verbose=0, quiet=False, generate=False, exclude=False, single=False, randomize=False, fromfile=None, findleaks=False, use_resources=None, trace=False, coverdir='coverage', runleaks=False, huntrleaks=False, verbose2=False)
Execute a test suite.
 
findtests(testdir=None, stdtests=['test_grammar', 'test_opcodes', 'test_operations', 'test_buil..., nottests=['test_support', 'test_future1', 'test_future2', 'test_future3'])
Return a list of all applicable test modules.
 
runtest(test, generate, verbose, quiet, testdir=None, huntrleaks=False)
Run a single test.
 
runtest_inner(test, generate, verbose, quiet, testdir=None, huntrleaks=False)
 
cleanup_test_droppings(testname, verbose)
 
dash_R(the_module, test, indirect_test, huntrleaks)
 
dash_R_cleanup(fs, ps, pic)
 
reportdiff(expected, output)
 
findtestdir()
 
removepy(name)
 
count(n, word)
 
printlist(x, width=70, indent=4)
Print the elements of iterable x to stdout.
Variables [hide private]
  newsoft = min(hard, max(soft, 1024* 2048))
  RESOURCE_NAMES = ('audio', 'curses', 'largefile', 'network', '...
  STDTESTS = ['test_grammar', 'test_opcodes', 'test_operations',...
  NOTTESTS = ['test_support', 'test_future1', 'test_future2', 't...
  _expectations = {'aix5': '\n test_aepack\n test_...
  mod = 'test.string_tests'

Imports: os, sys, getopt, random, warnings, re, cStringIO, traceback, resource, test_support


Function Details [hide private]

main(tests=None, testdir=None, verbose=0, quiet=False, generate=False, exclude=False, single=False, randomize=False, fromfile=None, findleaks=False, use_resources=None, trace=False, coverdir='coverage', runleaks=False, huntrleaks=False, verbose2=False)

 

Execute a test suite.

This also parses command-line options and modifies its behavior accordingly.

tests -- a list of strings containing test names (optional) testdir -- the directory in which to look for tests (optional)

Users other than the Python test suite will certainly want to specify testdir; if it's omitted, the directory containing the Python test suite is searched for.

If the tests argument is omitted, the tests listed on the command-line will be used. If that's empty, too, then all *.py files beginning with test_ will be used.

The other default arguments (verbose, quiet, generate, exclude, single, randomize, findleaks, use_resources, trace and coverdir) allow programmers calling main() directly to set the values that would normally be set by flags on the command line.

runtest(test, generate, verbose, quiet, testdir=None, huntrleaks=False)

 
Run a single test.

test -- the name of the test
generate -- if true, generate output, instead of running the test
            and comparing it to a previously created output file
verbose -- if true, print more messages
quiet -- if true, don't print 'skipped' messages (probably redundant)
testdir -- test directory
huntrleaks -- run multiple times to test for leaks; requires a debug
              build; a triple corresponding to -R's three arguments
Return:
    -2  test skipped because resource denied
    -1  test skipped for some other reason
     0  test failed
     1  test passed

printlist(x, width=70, indent=4)

 

Print the elements of iterable x to stdout.

Optional arg width (default 70) is the maximum line length. Optional arg indent (default 4) is the number of blanks with which to begin each line.


Variables Details [hide private]

RESOURCE_NAMES

Value:
('audio',
 'curses',
 'largefile',
 'network',
 'bsddb',
 'decimal',
 'compiler',
 'subprocess',
...

STDTESTS

Value:
['test_grammar',
 'test_opcodes',
 'test_operations',
 'test_builtin',
 'test_exceptions',
 'test_types']

NOTTESTS

Value:
['test_support', 'test_future1', 'test_future2', 'test_future3']

_expectations

Value:
{'aix5': '''
        test_aepack
        test_al
        test_applesingle
        test_bsddb
        test_bsddb185
        test_bsddb3
        test_bz2
...