What is this?

This is "bracket", a set of scripts for

 - Doing periodic builds of NetBSD-current
 - Testing the builds using anita (https://www.gson.org/netbsd/anita/)
 - Pinpointing regressions through automated binary search (aka bisection)
 - Reporting build and test failures by email
 - Generating HTML reports and graphs


Why is it called "bracket"?

It is called "bracket" because it pinpoints build and test failures
through a binary search process that iteratively bounds the failure
point from both sides:

  Bracket, v: To bound on both sides, to surround as enclosing
  with brackets.

Also, it serves as a kind of test fixture:

  Bracket, n: A projecting fixture


Disclaimer

Bracket was written for the author's personal use, with no particular
regard to usability by others, maintainability, orthogonality,
modularity, etc.  It is being made publicly available only because
people have requested it.  Bracket comes with no warranty and no
support.


Selecting a host machine

Bracket it typically installed on a NetBSD host, but it is also
possible to install it on other Unix-like systems for cross-building
and cross-testing of NetBSD from those hosts.  NetBSD 9 and Debian 11
systems have been successfully used as hosts.


How to install

Run the configure script, specifying the installation prefix and the
location of your preferred python interpreter, for example:

  ./configure --prefix /usr/local --with-python /usr/pkg/bin/python3.9

Bracket supports Python versions from 3.6 onwards and has been most
recently tested with Python 3.9.  Python 2.7 is no longer supported.

Then:

  make
  sudo make install

The data files (repository copy, checked-out source trees, build
logs, etc) are kept in a separate directory, by default in /bracket.
Make sure the disk /bracket resides on has at least on the order of
100 GB of free space.  At the time of writing, the CVS repository
takes about 30 GB, and build and test directories take several tens of
GB more.  You may want to mount a separate disk on /bracket.

Most of the disk traffic happens during the builds, which are made in
a directory defined by "build_root" in bracket.conf, defaulting to
/bracket/build.  If /bracket is on a magnetic disk, you should point
build_root to a ramdisk or SSD of 32 GB or more as this will greatly
reduce build times.  There is no need to preserve the contents of the
build_root directory across reboots.

Create the /bracket directory (or whatever you chose to call it)
as root and chown it to the user who will be running bracket:

  sudo mkdir /bracket
  sudo chown $USER /bracket

Install prerequisite packages from pkgsrc (this list may not be complete):

  lang/python39
  misc/py-anita
  emulators/qemu
  net/rsync
  graphics/py-matplotlib
  textproc/libxslt
  textproc/py-expat (because of http://gnats.netbsd.org/45345?)
  devel/py-sysctl
  devel/py-gitpython (only when using git as the version control system)

If you are going to publish results on the web, the web server machine
(which may or may not be the same as the build machine) will need a web
server.  If it is running NetBSD, the built-in bozohttpd will do.
Alternatively, you can use www/lighttpd, in which case you will also need
lang/perl5.

Prerequisites on non-NetBSD hosts vary.  On Debian, you need at least

  qemu
  qemu-system-ppc (if you want to test the macppc port)
  m4
  python-pexpect
  rsync
  nullmailer (or another MTA that provides a "sendmail" program)
  cvs*

* You will need a version of CVS that is compatible with the NetBSD
repository; the CVS shipped by Debian will not work.  One option is to
first install the incompatible native cvs to bootstrap pkgsrc, and
then install a compatible CVS from pkgsrc:

  cvs -danoncvs@anoncvs.netbsd.org:/cvsroot checkout pkgsrc
  cd pkgsrc/bootstrap/
  export SH=/bin/bash
  ./bootstrap --unprivileged
  cd ../devel/scmcvs/
  $HOME/pkg/bin/bmake install

Then, when editing bracket.conf below, set the "cvs" configuration
variable to the absolute path $HOME/pkg/bin/cvs expands to.

Create the configuration file bracket.conf:

  cp /usr/local/share/examples/bracket/bracket.conf.sample bracket.conf

and edit it to taste.  The example is set up to test the sparc port,
so unless that's what you intend to test, the first step should be to
search-and-repace the string "sparc" with the name of the port you
want to test.

Bracket will look for bracket.conf in the current working directory.
Therefore, if you are going to test more than one architecture or
configuration, you should make a directory for each, each with its
own bracket.conf.  For example, you could have these two bracket.conf
files for testing amd64 and i386, respectively:

  $HOME/bracket/amd64/bracket.conf
  $HOME/bracket/i386/bracket.conf

To initialize /bracket and create various subdirectories under it, run

  sudo bracket setup

Download a copy of the NetBSD CVS repository using rsync and index it:

  bracket update-repository

This can take a long time; around 24 hours is not unusual.  Also, it
may fail because of issues with anoncvs or your connection to it, such
as those reported in systems ticket [NetBSD.org #160795] "rsync from
anoncvs fails repeatedly".  If that happens, rerun "bracket
update-repository", and if it happens often, uncomment the line

  update_repo_retry="1"

in bracket.conf to retry the rsync automatically.

If you already have a copy of the repository on hand, things will go
much faster if you copy it to /bracket/repo first so that rsync has
less to sync.  The important part is the "src" directory, which should
be copied to /bracket/repo/src.


How to run

Now you should be able to manually start a build and test of -current:

  bracket notify

This will take a few hours to run.  Subsequent runs of this command
will send an email notification to the address given in the
configuration if the build has been broken between the runs (if
enabled in the configuration).

Even if you run "bracket notify" periodically, it probably won't
run often enough to build every commit.  To fill in the gaps, you
can run

  bracket refine

This does a test build of a historic version, preferring versions in
recent gaps where where the build was either broken or fixed.  By
running "bracket refine" repeatedly enough times, every build break
(and fix) will eventually be bisected down to a specific commit.

To build, install, test, or interact with the console of a given
source date, use one of

  bracket build <date>
  bracket install <date>
  bracket test <date>
  bracket interact <date>

Here, <date> can be an RCS-style date like 2019.09.17.15.20.05,
or one of the following special strings:

  latest_successful_build - the latest version to build successfully
  latest_image - the latest version for which there is an already
    installed system image
  current - the newest version on anoncvs (does a repo rsync)
  current_local - the newest version in the local repo mirror
    (does not do a repo rsync)
  "2 days ago" - or similar with weeks/months/years

To see what source date one of the above resolves to, use

  bracket print-rcsdate <date>

To manually update the repository copy or the HTML reports:

  bracket update-repository
  bracket update-reports

To force regeneration of all HTML reports rather than just changed
ones:

  bracket update-reports --full

To generate HTML reports about the amount of machine resources
consumed by the builds and tests:

  bracket update-resource-reports

If web reports are enabled, the resulting reports are are copied to
the web server but are not currently linked from the main report page;
they can be found by appending /resource/ to the URL, for example
http://releng.netbsd.org/b5reports/amd64/resource/.

To print an ASCII "punched card" summary of the ATF test results from
the latest runs:

  bracket print-atf-summary


Running the tests periodically from cron

To run the tests periodically, set up cron entries modelled after the
following.  These run each job four times daily; adjust to taste.

   15 4,10,16,22 * * * cd /home/bracketuser/bracket/sparc && /usr/local/lib/bracket/cronjob.sh notify
   10 3,9,15,21 * * * cd /home/bracketuser/bracket/sparc && /usr/local/lib/bracket/cronjob.sh refine

If you copy these entries to your own crontab, please make some random
changes to the minutes and hours fields to help even out the load on
anoncvs.netbsd.org.

When planning the schedule, keep in mind that the notify and refine
jobs both start by doing a build using all available CPU cores,
followed by installation and test which will only use one core.
Therefore, it makes sense to space the runs such that each build phase
has time to complete before the next run starts, but the install/test
phase may overlap with the next build.


Running the tests continuously

If you are running the tests on a dedicated machine, you may want to
run them continuously instead of at regular interval to take full
advantage of the available machine resources and test as many commits
as possible.  This can be done using the command "bracket schedule",
which will run in an endless loop, alternately updating the repository
copy and doing test builds using "bracket notify".

When schedule.py has free time (such as when a repository update shows
no new commits since the last build), it can be configured to run
other, lower-priority jobs such as "bracket refine", or scripts in
other directories, for example running "bracket notify" for a
different NetBSD port.  These lower-priority jobs are configred using
"background_jobs" in bracket.conf.

The scheduler can be started by hand with

  bracket schedule

or at boot time by putting the following in /etc/rc.local:

  echo -n "bracket scheduler"
  su bracketuser -c 'cd /home/bracketuser/bracket && sh /usr/local/lib/bracket/cronjob.sh schedule' &


Configuring a web server

If you would like to publish HTML reports on the web, you also need to
configure a web server.

The reports contain different file types that need to be mapped to the
appropriate MIME types based on their extensions, and some of them are
stored compressed and have a double extension like .log.gz.

The official testbed on babylon5.netbsd.org achieves this using the
following invocation of the bozohttpd server in the NetBSD base system:

    /usr/libexec/httpd \
        -M .css.gz text/css x-gzip gzip \
        -M .html.gz text/html x-gzip gzip \
        -M .log.gz text/plain x-gzip gzip \
        -M .log.tail.gz text/plain x-gzip gzip \
        -M .svg image/svg+xml - - \
        -M .svg.gz image/svg+xml x-gzip gzip \
        -M .tps.gz 'application/X-atf-tps;version="2"' x-gzip gzip \
        -M .xml.gz application/X-atf+xml x-gzip gzip \
        /bracket/web

If you use this, adjust the final /bracket/web to match the htmldir
defined in your bracket.conf.

If you use a different web server, it will have to be configured in
its own way.  Instructions for lighttpd can be found in
cgi/README.LIGHTTPD in the source tree.


Upgrading bracket

To upgrade from a previous version of bracket, build and install
it as above, and then run

  bracket upgrade


Patches

The patches/ subdirectory contains a set of patches that backport bug
fixes to old versions of NetBSD so that they have a better chance of
building on a modern host.  They include fixes for failures to build
the tools with recent versions of gcc, failures to build the tools
with recent Linux libc header files, random build failures due to race
conditions, etc.  They are not needed to build a current -current, but
are needed when building historic versions from certain periods (which
in turn is needed when bisecting old bugs).

The file name is of the form DATE0-DATE1-DESCRIPTION.patch, where
DATE0 is the first source date for which the patch should be applied,
and DATE1 is the first date on which it is no longer needed.  Some
of the dates are guesswork.

The current set of patches allow a Ubuntu 10.04 host to build he
NetBSD CVS trunk back to CVS date 2006.06.27.21.59.51.  Sadly,
a NetBSD 6.1.5 host is not capable of going back as far.


The results database

The results of every build and test run are stored in the directory
given by the configuration variable "results_root", for example,
/bracket/sparc/results.

There is a subdirectory per year and a sub-subdirectory per source
date tested, using the RCS-format date as the directory name.  For
example, the results from testing CVS date 2015.03.18.05.54.14 on
sparc will be stored in the directory
/bracket/sparc/results/2015/2015.03.18.05.54.14.

In each subdirectory, there is a file "bracket.db" and a number of
compressed log files and reports:

  build.log.tail.gz     Last 1000 or so lines of build log
  install.log.gz        Console output from the install phase
  test.log.gz           Console output from the test phase
  test.tps.gz           Output from atf-run
  test.xml.gz           Output from atf-report -o xml
  test.html.gz          ATF test report in HTML format
  tests-results.css.gz  Style sheet used by HTML test report

Only the 1000 last lines of the build log are saved to conserve disk
space (or a bit more if the build failed), but the other files are
included in their entirety.

The file bracket.db contains summary information about the build and
test results.  It is a simple key-value store where each line begins
by a key name and an equals sign, and the rest of the line is the value.
Values are updated by appending new entries to the file rather than by
replacing existing entries, so there may be multiple entries with the
same key.  For example, if the tests have been run multiple times,
there can be multiple "passed_tests" keys.  Normally, the last entry
should be used.

Here are the contents of a typical bracket.db file:

  build_log_lines=353187
  build_host=guesthouse
  build_status=0
  install_status=0
  test_status=0
  passed_tests=1317
  failed_tests=0
  expected_failure_tests=25
  skipped_tests=45

The various *_status lines record the exit status (in the Unix sense)
of various stages of the build and test, so 0 means success and
anything else means failure.

The results database is easily greppable; for example, to find all build
failures in December 2010, cd to $db_dir and do

  grep build_status=1 2010/2010.12.*/bracket.db

(omit the initial 2010/ if bracket was configured with
use_subdir_per_year="0").


Bisection

In addition to running automated builds and tests and automatically
pinpointing build failures, bracket can also be used to pinpoint other
forms of breakage using binary search, using the "bracket bisect"
command.  Users of "git" may find this somewhat similar to the
functionality provided by the "git bisect" command.

For example, to find which commit broke the build between two
given source dates, you can use a command like

  bracket bisect 2022.08.20.18.40.35 2022.08.20.23.26.02 build

If the build is currently broken, you can find the commit that
broke it simply by running

  bracket bisect --auto build

To find other kinds of breakage, you can use other test names in place
of "build", including the following ones built into the bracket core
(in order of increasing stringency):

  install            The system installed
  boot               The installed system booted
  test_completed     The ATF tests ran to completion
  test               The ATF tests all passed

Many additional tests are defined as external Python scripts.  These
are under tests/ in the source tree, and are installed into
$prefix/lib/bracket/py/tests.  The name of each such Python script
(sans the .py extension) can be used as a test name, and some take
additional arguments.  Many of these are special-purpose regression
tests for individual PRs, but others are more general.  For example,
using "atf_test", you can find the commit that broke a specific ATF
test:

  bracket bisect 2020.04.04.19.46.01 2020.04.05.00.36.25 atf_test kernel/t_pty/pty_queue

That works by running all the ATF tests and then check their output
for the outcome of a single test case.  You can save some time by
running only the test of interest using "atf_test_single":

  bracket bisect 2020.04.04.19.46.01 2020.04.05.00.36.25 atf_test_single kernel/t_pty/pty_queue

When tests fail only randomly, the "repeat" meta-test can be
useful.  For example, this will run the dhcpcd_lease_no_w test
100 times and consider it to pass only when all 100 runs pass:

  bracket bisect 2017.01.01.03.06.06 2019.12.20.12.02.46 repeat 100 dhcpcd_lease_no_w

To find when something was fixed rather than when it broke, use
"bracket bisect --fix ...".


Old-style bisection

There is also an older form of bisection support where the
entire bisection, including both the error condition and the
time interval, is defined as a Python script to be run using
"bracket run".

For example, if you have a bug that needs to be tested for manually,
you can bisect it by running the "breakage.py" script.  This takes the
source date range to bisect and optionally a name you assign to the
bug for record-keeping, for example:

  bracket run /usr/local/share/examples/bracket/bisect/breakage.py \
    2019.08.01.00.10.22 2019.09.03.19.07.50 pr99999

This will build different versions and ask you to test them manually.

If you can write a Python function that returns a boolean indicating
whether things are working, the process can be fully automated.  The
other scripts in /usr/local/share/examples/bracket/bisect/ serve as
examples of such fully automated bisections that have been made in the
past.  You can use these as starting points for your own fully
automated bisections.  For example, if you want to bisect a build
failure, you can copy a script that was used to bisect build failures
in the past, edit the date range in it to cover the new build
failure, and run it:

  cp /usr/local/share/examples/bracket/bisect/build-failure.py .
  vi build-failure.py
  bracket run ./build-failure.py


Security

When you run bracket, you implicitly trust the NetBSD source.  For
example, any time you build NetBSD-current from source, the NetBSD
build will execute many thousands of shell commands from the NetBSD
source tree, and some of those could potentially have harmful effects,
either by accident or as a result of a deliberate attack.  This is the
case whether the build is done by bracket or manually.  Bracket does
not currently attempt to sandbox the build process in any way.

The bracket command line and the configuration in bracket.conf are
also implictly trusted.  Bracket may, for example, execute shell
commands that include expanded configuration variables from
bracket.conf.  Do not invoke bracket with command line arguments from
an untrusted source, or use a bracket.conf you have obtained from an
untrusted source.


Inner workings

If you plan to work on bracket itself rather than just running it,
please also read the file INTERNALS.


Acknowledgements

Parts of the Mercurial support were contributed by coypu and wiz.

The bozohttpd command line was devised by spz.
