A.16 SOFTWARE-EVOLUTION-LIBRARY/COMPONENTS/TEST-SUITE

Variable: *process-sleep-interval* [sel/cp/test-suite]

Frequency (in seconds) at which to check if a process has completed.

Generic Function: finish-test [sel/cp/test-suite] test-process

Ensure that test-process either runs to completion or is killed. Return the standard output, error output, and process exit code as strings.

* test-process the ‘process’ associated with the test case.

Method: finish-test [sel/cp/test-suite] test-process

Ensure that test-process either runs to completion or is killed. Return the standard output, error output, and process exit code as strings.

* test-process the ‘process-info’ object returned by ‘launch-program’ associated with the test case.

See also ‘*process-sleep-interval*’ and ‘*process-kill-timeout*’. If test-process is still running when ‘finish-test’ is called, and if ‘*process-kill-timeout*’ is bound to a non-nil value greater than 0, alternate between sleeping for ‘*process-sleep-interval*’ seconds and checking again until either ‘*process-kill-timeout*’ seconds have elapsed or the process completes. After the timeout, send a sigterm signal if test-process is still running, sleep for ‘*process-sleep-interval*’ seconds once more and then if the process is still running, send a sigkill signal.

Generic Function: run-test [sel/cp/test-suite] phenome test-case &rest extra-keys &key &allow-other-keys

Run test-case on phenome and return the results. Return the standard output, error output, and process exit code as strings.

Method: run-test [sel/cp/test-suite] phenome (test-case test-case) &rest extra-keys &key &allow-other-keys

Run test-case on phenome and return the results. Return three values: output printed to standard out, error output, and exit status. Th default behavior is simply to run ‘start-test’ and ‘finish-test’.

Generic Function: start-test [sel/cp/test-suite] phenome test-case &rest extra-keys &key &allow-other-keys

Start an external process to run test-case on phenome. Return the ‘process’.

• phenome the phenome of the ‘software’ object under test.
• test-case the ‘test-case’ to run.
• extra-keys additional keyword arguments.

Method: start-test [sel/cp/test-suite] phenome (test-case test-case) &rest extra-keys &key &allow-other-keys

Start an external process to run test-case on phenome. Return the ‘process’. This is a wrapper around ‘uiop:launch-program’, and any extra-keys will be passed through to that method.

• phenome the phenome of the ‘software’ object under test.
• test-case the ‘test-case’ to run.
• extra-keys additional keyword arguments to pass to ‘uiop:launch-program’.

Some extra-keys that may be useful are:

• :output and :error-output - to specify how output and error streams are handled. By default, these are captured so they can be returned as strings.
• :wait - whether to wait for the process to complete before continuing. The default is to wait; however, some components may elect not to wait and instead to stream results through a named pipe.
• :env - to set environment variables

Class: test-case [sel/cp/test-suite]

Class precedence list: test-case, standard-object, slot-object, t

Slots:

• program-name — initarg: :program-name; reader: software-evolution-library/components/test-suite:program-name; writer: (setf software-evolution-library/components/test-suite:program-name)

  The executable that runs the test case. Contains either a string containing the name of an executable file or special symbol :bin, which will be dynamically replaced by the name of the compiled phenome for the software object under test.

• program-args — initarg: :program-args; reader: software-evolution-library/components/test-suite:program-args; writer: (setf software-evolution-library/components/test-suite:program-args)

  A list of arguments which will be passed to the executable program-name. Each argument must be either a string or symbol :bin, which will be replaced by the name of the compiled phenome.

• fitness — initarg: :fitness; reader: software-evolution-library/software-evolution-library:fitness; writer: (setf software-evolution-library/software-evolution-library:fitness)

  May be used to store the fitness result from a prior run of the test case. This field is not updated automatically since some components may need to perform comparisons between new and old values (e.g., condition synthesis).

Test case object.

Class: test-suite [sel/cp/test-suite]

Class precedence list: test-suite, oid-object, standard-object, slot-object, t

Slots:

• test-cases — initarg: :test-cases; reader: software-evolution-library/components/test-suite:test-cases; writer: (setf software-evolution-library/components/test-suite:test-cases)

  List of ‘test-case’ objects that make up the test suite.

A suite of unit tests.

Variable: *process-kill-timeout* [sel/cp/test-suite]

Optional timeout (in seconds) before killing a process