All participating solvers will be executed on the same set of input problems.

Ranking the solvers on one single problem

Let m be the number of participating solvers. For each input problem we give points to each solver, according to whether the participant yields

  1. a claimed solution that really is a solution
  2. no solution: either the solver terminates normally and declares that he hasn't found a solution, or crashes (forced timeout, segmentation fault, ...)
  3. a claimed solution, which in reality is not a solution to the problem

The number of points a solver gets on a given input problem is calculated as follows:

  • All solvers in case (1) are ordered according to the optimization criterion (best solution first), and a participant in case (1) gets the number of points that corresponds to his positions in that list. In case two solvers have found equally good solutions they get the same number of points. This means that the number of points obtained here is 1 plus the number of solvers that are better. Example with 4 participating solvers s1, s2, s3 and s4: If s1 gives the best solution, followed by s2 and s3 giving an equally good solution, and s4 who gives the worst solution, then s1 gets 1 point, s2 and s3 each get 2 points, and s4 get 4 points.
  • A participant in case (2) gets 2*m points
  • A participant in case (3) gets 3*m points

Note that it is possible that an input problem indeed doesn't have a solution. In that case, (1) above is not possible, so the solvers that correctly say FAIL are ranked best.

It is also possible that the input universe is not consistent. In that case a solver is expected to turn that inconsistent universe into a consistent universe which satisfies the request.

Determining the winner

The total score of a solver is the sum of the number of points obtained for all problems of the problem set. The solver with the minimal score wins.

Breaking ties

If several solvers obtain the best total score then the winner is determined by choosing among those having obtained the best total score the one with the minimal total success time. The total success time of a solver is the sum of execution times of that solvers over all problems in case 1 (correct solution), plus f*timeout where f is the number of problems where the result of that solver is in one of the cases 2 (no solution) or 3 (wrong solution), and timeout is the time limit defined in the execution environment.