Evaluation (Software Evolution Library)

5.2.2 Evaluation

Once we are mutating programs, we need to be able to evaluate each variant and keep track of the evaluation results.

We have provided a shell script test.sh, which contains several gcd test cases. Our fitness function will be the number of test cases for which a program produces the correct solution.

Once a fitness value for a sel:software object is established, it can be recorded in the object’s fitness field.

The example code below has two main components:

A test function that harnesses the test.sh script to evaluate the fitness of an individual sel/sw/asm:asm object.
A loop in which ten mutants of the sel/sw/asm:asm object corresponding to the original gcd.s program are generated, their fitness assessed, and the results printed.

(defpackage :example
  (:use :gt/full
        :software-evolution-library
        :software-evolution-library/software/asm))
(in-package :example)

(defvar *orig* (from-file (make-instance 'asm) "test/etc/gcd/gcd.s"))

(defun test (asm)
  "Run the GCD unit tests on ASM. Return the number of passing tests."
  (ignore-errors
    (with-temporary-file (:pathname bin)
      ;; Build executable
      (phenome asm :bin bin)
      (count-if #'identity
                (loop :for i :below 12 :collect
                   (multiple-value-bind (stdout stderr errno)
                       (shell "test/etc/gcd/test.sh ~a ~d" bin i)
                     (declare (ignorable stdout stderr))
                     ;; Collect list of T/NIL indicating if the exit code was 0.
                     ;; Tests whose exit code is 0 are considered successful.
                     (zerop errno)))))))

;; Apply 10 mutations to copies of `*orig*'
(loop :for i :below 10 :do
   (handler-bind
       ((no-mutation-targets
         (lambda (e)
           (declare (ignorable e))
           (invoke-restart 'try-another-mutation))))
     (multiple-value-bind (mutant edit) (mutate (copy *orig*))
       ;; Set fitness to be the number of passing unit tests
       (setf (fitness mutant) (test mutant))
       ;; Print the final fitness and the mutation applied.
       (format t "~2d fitness for edit ~S~%" (fitness mutant) edit))))

Executing this code will print output resembling the following.

 0 fitness for edit (:INSERT 76 38)
 0 fitness for edit (:SWAP 66 77)
 0 fitness for edit (:CUT 50)
10 fitness for edit (:CUT 11)
10 fitness for edit (:SWAP 62 39)
 0 fitness for edit (:INSERT 2 48)
10 fitness for edit (:CUT 66)
 0 fitness for edit (:CUT 73)
10 fitness for edit (:INSERT 73 26)
 0 fitness for edit (:INSERT 71 1)

API components in this example

Class sel/sw/asm:asm
Methods