###############
Counterexamples
###############

Counterexamples are the translation of models generated by SMT solvers, when a
formula can be shown to be satisfiable, into an execution trace at the SPARK
level for the user. Roughly, how it works:

- During :ref:`Translation to Why3`, annotations (attributes) are added to the
  generated code together with locations.
- Then, during :ref:`Weakest Precondition`, the attributes are kept and still
  associated to appropriate variables.
- Then, a specific transformation for counterexamples will introduce new
  constants for all constants appearing inside the verification condition. This
  constant location will be the one of the VC. This will allow to have more
  counterexample values on the VC line (because no variable is defined at an
  overflow check for example).
- Then, the printer will collect the variables that have a ``model_trace``
  attribute (added during first phase). So, the printer prints the variables to
  smt2 and Why3 records the correspondance with ``model_trace``.
- Then cvc4/z3 executes and gives its output.
- The prover model is parsed and processed to recover values of program
  variables
- This correspondance is amended with information from gnatcounterexamples.ml
  ??? TODO
- Everything is transformed into JSON together with the rest of the results
- Send to gnat2why
- Gnat2why parses the JSON input from counterexamples and do some computation
  to display it.

Perhaps the easier way to explain this is to proceed with layers.

Provers layer
=============

From Why3, the smt solvers are called by transforming the task into an smt2
file by replacement of syntax and change the goal into its negation. That way,
if the prover is sure that the property is unsatisfiable then we are sure that
indeed the task is valid (meaning it was proved).
So, when provers are able to satisfy the task, they answer sat. In this case,
they found a model for the task (i.e. an instantiation of all constants that is
consistent with the task and does not negate the goal). In this case, we can
ask them for this task using:

.. TODO ??? Add smt2 language in pygments using the .el file availables online
.. code-block:: Ada

    (get-model)

After a sat/unknown answer from a ``(check-sat)`` command, when given a
``(get-model)`` command, the prover produce an assignment for every variable (a
model).
The way things are expressed in smt2 is defined in http://smtlib.cs.uiowa.edu/ .
This looks like the following example (S-expr style and prefix operators).
From task:

.. code-block:: Ada

    (declare-const a Int)

    (declare-const b Int)

    (assert (= b 2))

    (assert (= a (- 2 )))

    (assert
      (in_range (* a b)))
    (check-sat)
    (get-model)

yields the following model:

.. code-block:: Ada

     sat
     (model
     (define-fun a () Int (- 2))
     (define-fun b () Int 2)
     )


Parsing and processing of prover models
=======================================

The entry points for parsing and processing of prover models is in the variable
``model_parsers`` of module :download:`Model_parser
<../../why3/src/core/model_parser.ml>`. The model is parsed using function
``model_parser`` by first applying a *raw* model parser, which is implemented in
module :download:`Smtv2_model_parser
<../../why3/src/driver/smtv2_model_parser.ml>` for SMTv2 prover models, and then
processing the resulting preliminary model elements. The following sections
describe the parsing of SMTv2 prover models and their processing.

..
   Here is an overview of the parsing and processing:

   - ``Model_parser.model_parse : string -> Model_parser.model``

     - ``Smtv2_model_parser.parse : string -> Model_parser.model_element list``

       - ``get_model_string : string -> string``

       - ``parse_sexps : string -> Sexp.sexp``

       - ``model_of_sexps : Sexp.sexp -> Smtv2_model_defs.definition Mstr.t``

       - ``Collect_data_model.create_list : Smtv2_model_defs.definition Mstr.t -> Model_parser.model_value Mstr.t``

       - create preliminary model elements ``Model_parser.model_value Mstr.t -> Model_parser.model_element list``

     - ``build_model_rec : Model_parser.model_element list -> Model_parser.model_files``

       - process and filter model elements ``build_model_rec:process_me : model_element -> model_element option``

       - add model elements at relevant locations ``build_model_rec:add_model_elt``

     - clean elements in the model files using ``Model_parser.clean``

     - ``handle_contradictory_vc : model_files -> model_files``

     - complete model ``model_files -> model``

Parsing
-------

The parsing of the prover model is implemented in function ``parse`` of module
:download:`Smtv2_model_parser <../../why3/src/driver/smtv2_model_parser.ml>`.

In a first step, the prover model is cut from the prover output using a regular
expression (``Smtv2_model_parser.get_model_string``) and parsed as an
S-expression. The definition of the S-expression and the parser are implemented
in module :download:`Sexp <../../why3/src/driver/sexp.mli>`. The parsing is
based on using a stack in ``ocamllex``.

Using ``Smtv2_model_parser.model_of_sexps``, the S-expression is then converted
to a mapping from variable names to SMTv2 definitions from module
:download:`Smtv2_model_defs <../../why3/src/driver/smtv2_model_defs.mli>` using
function ``create_list`` from module :download:`Collect_data_model
<../../why3/src/driver/collect_data_model.ml>`. The function ``create_list``
creates a mapping from the names of constants among the SMTv2 definitions to
model values. The model values are obtained by evaluating the SMTv2 expressions
by which the constants are defined.

Finally, the function ``Smtv2_model_parser.parse`` converts the model values to
preliminary model elements. A model element combines a model value with
information about the identifier, the location, and the corresponding Why3 term
from the VC.

Processing
----------

The preliminary model elements obtained from the raw model parser are then
further processed in function ``Model_parser.model_parser``. The bulk of the
processing is done in function ``Model_parser.build_model_rec``, which rebuilds
the model elements by replacing projections and restore single field records,
and adding the element at all relevant locations. The model elements are then
cleaned using ``Model_parser.clean`` (which is customised to
``Gnat_counterexample.clean`` to delay the cleaning of model names). Finally,
contradictory VCs are marked when there are no model elements for the VC.

In SPARK, the model is additionally processed using
``Gnat_counterexample.clean``, which prepares the values and names in the model
element for display in ``gnatprove``: the values of unconstrained arrays and
split discriminants are modified and the names are replaced by their model trace
strings, which contain their GNAT identifiers.

The cleaning of the model is split into ``Gnat_counterexample.clean`` and
``Gnat_counterexample.post_clean`` in ``gnatwhy3``, because ``post_clean``
breaks the types of the model values, but valid types are necessary for the
checking and classification of counterexamples, which is described below.

Checking and classifying proof-failures
=======================================

``gnatprove`` checks counterexamples models and classifies proof failures: The
subprogram from where the VC originates is executed using two types of
runtime-assertion checking (RAC), normal RAC and giant-step RAC (see
``Gnat2Why.Error_Messages.Handle_Result``). In both RAC executions, the
counterexamples provides the values for the arguments of the subprogram and for
values for global variables. The normal RAC is implemented in
``SPARK_RAC.RAC_Execute`` and corresponds to the execution of a binary compiled
with assertions enabled.

The *giant-step* RAC is requested from ``gnatwhy3``. A giant-step
RAC corresponds to a normal RAC execution of the WhyML program, with the
difference that function calls and loops are executed by a single "giant step"
by obtaining return values of function calls and the values of modified
variables directly from the model. (I.e. the model is additionally used as an
*oracle* during the giant-step RAC execution.)

Both RAC executions result in a ``SPARK_RAC.Result``, which contains the kind of
the result (normal termination, assertion failure, incomplete execution, failed
assumption, or not executed) and additionally information about the relevant VC
for assertion failures.

The results of the normal and giant-step RAC executions are combined into a
*verdict* on the counterexample, using the following table

================= ============================== ====================== ================== ==========
\                 Giant-step RAC result
----------------- -----------------------------------------------------------------------------------
Normal RAC result Normal                         Failure                Incomplete         Stuck
================= ============================== ====================== ================== ==========
Failure at VC     Non-conformity
----------------- -----------------------------------------------------------------------------------
Failure elsewhere Bad counterexample
----------------- -----------------------------------------------------------------------------------
Stuck             Invalid assumption
----------------- -----------------------------------------------------------------------------------
Normal            Subcontract-weakness           Bad counterexample     Bad counterexample Incomplete
Incomplete        Non-conformity or sub-weakness Incomplete             Bad counterexample Incomplete
================= ============================== ====================== ================== ==========

This approach to checking and classifying counterexamples is described in the
article *Explaining Counterexamples with Giant-Step Assertion Checking*
(https://eptcs.web.cse.unsw.edu.au/paper.cgi?FIDE2021.10), and more details are
available in the technical report *Giant-step Semantics for the Categorisation
of Counterexamples* (https://hal.inria.fr/hal-03213438).

However, for SPARK the normal RAC is implemented in gnat2why based on the
original Ada program, because the generated Why3 program is not executable. Only
the giant-step RAC execution is based on the WhyML program generated by
``gnat2why``.

The verdict of the counterexample checking from the table above is represented
in the SPARK message about the proof failure as follows:

============================== ============== ======== ===================================================
Verdict                        Counterexample Priority Fix
============================== ============== ======== ===================================================
Incomplete                     discard        medium
------------------------------ -------------- -------- ---------------------------------------------------
Bad CE                         discard        medium
------------------------------ -------------- -------- ---------------------------------------------------
Non-conformity                 show           high
------------------------------ -------------- -------- ---------------------------------------------------
Subcontract weakness           show           medium   "some loop invariant or post condition is too weak"
------------------------------ -------------- -------- ---------------------------------------------------
Non-conformity or sub-weakness show           medium
============================== ============== ======== ===================================================

Normal and giant-step runtime-assertion checking
------------------------------------------------

The engine for normal runtime-assertion (RAC) checking is implemented in
``SPARK_RAC.RAC_Execute``. (*Normal* in contrast to the giant-step RAC engine
implemented in gnatwhy3.)

Debugging
~~~~~~~~~

The results of the two RAC executions are printed by ``gnatprove`` when using
option ``--debug``. When setting the environment variable
``GNAT2WHY_RAC_TRACE=on`` the program nodes are printed as they are executed
during the normal RAC.

To collect debugging information about the giant-step RAC in ``gnatwhy3``,
several environment variables can be activated:

``WHY3DEBUG=<options>``
    ``<options>`` is a comma-separated list of Why3 debug options. E.g.
    ``rac-check-term-result,rac-check-term-sat,check-ce-summary`` when checking
    counterexamples.

``WHY3RACTASKDIR=<dir>``
    Print tasks for checks that cannot be checked during giant-step RAC to files
    in directory ``<dir>``. Enabled only when Why3 debug option
    ``rac-check-term-sat`` is enabled. The filenames are printed in the messages
    from ``rac-check-term-sat``.

``GNATWHY3LOG=<file>``
    Debug output of ``gnatwhy3`` is send to ``<file>``, instead of the standard
    error output of ``gnatwhy3``.

Testing
~~~~~~~

The normal RAC engine is tested in ``TC02-027__rac``, which compares the
behaviour of the RAC engine on the main procedure of a project with the
behaviour of a program compiled with assertions enabled:

1. Compile the project using options ``-gnata -gnatw.A -gnat2022``

2. Execute the compiled program and capture the output

3. Execute the normal RAC engine with ``gnatprove`` using option
   ``--debug-exec-rac`` (which runs the RAC engine on the main procedure and
   exits), and capture the output.

4. Compare the output of the compiled program and of the RAC engine

This is implemented by the ``RACTestDriver`` in
:download:`run-tests <../../testsuite/gnatprove/run-tests>`.

Tracking incompleteness
~~~~~~~~~~~~~~~~~~~~~~~

Using the ``--debug`` flag, additional information (including the final verdict
and its optional reason and the results from the normal and giant-step RAC
execution and their optional reasons and the location) is printed about the
checking of each counterexample as a tab-separated record with a prefix
``VERDICT``:

.. code-block::

   VERDICT\tFinal_verdict\tVerdict_reason>\tNormal_RAC_result\tNormal_reason>\tGiant-step_result\tGiant-step_reason\tLocation

The following Makefile can help to extract statistics from mailserver's ``.out``
files:

.. code-block:: makefile

   VERDICTS.tsv:
   	grep '^VERDICT' *.out \
   	| sed -e 's/\.out:VERDICT//' -e 's/\\x09/	/g' \
   	> $@

   .PHONY: count-final-verdicts count-incomplete-verdicts

   count-final-verdicts: VERDICTS.tsv
   	awk -v FS='\t' '{ print $$2 }' $< | sort | uniq -c | sort -hr

   count-incomplete-verdicts: VERDICTS.tsv
   	awk -v FS='\t' '$$2 == "INCOMPLETE" { print $$3 }' $< \
   	| sort | uniq -c | sort -hr

   count-incomplete-verdicts-clean: VERDICTS.tsv
   	awk -v FS='\t' '$$2 == "INCOMPLETE" { print $$3 }' $< \
   	| awk '/cannot be evaluated/                           { print "cannot decide"; next } \
   	       /cannot import/                                 { print "cannot import value"; next } \
   	       /missing value for return value/                { print "missing return value"; next } \
   	       /missing value for variable/                    { print "missing variable value"; next } \
   	       /No counterexample value for program parameter/ { print "missing parameter value"; next } \
   	       /not supported/                                 { print "unsupported"; next } \
   	       /No VC/                                         { print "no VC"; next } \
   	       /No body/                                       { print "no body"; next } \
   	                                                       { print $0 }' \
   	| sort | uniq -c | sort -hr

   CRASHES:
   	cat *.out \
   	  | grep '\<Pro\>' *.out \
   	  | sed 's/\([^.]*\).out:| Pro 22.0w ([^)]\+) (spark) \(.*\)/\1:\2/' \
   	  | sed 's/for node  [0-9]\+/for some node/' \
   	| sed 's/\.out:/	/' \
   	> $@

   count-crashes: CRASHES
   	cut -d'	' -f2- $< \
   	| awk -v FS='\t' \
   	    '/No VC for node/ { print "no VC"; next } \
   	     /import error/   { print "import error"; next } \
   	                      { print $$0 }' \
   	| sort | uniq -c | sort -hr
