Release 13.1 of the PostgreSQL BuildFarm client

 

I have just pushed Release 13.1 of the PostgreSQL BuildFarm client.

This update to Release 13 fixes errors that occur from the new default branch name checking code when used with versions of git that are too old. This code is now disabled if the git version is not capable of running it, and in verbose mode a warning is printed. The warning can be disabled by explicitly setting "skip_git_default_check => 1" in the config file. In either case, the owner will need to update their git installation or remove all branch and mirror repositories when the default branch name changes.

Downloads are available at:

https://github.com/PGBuildFarm/client-code/releases and

https://buildfarm.postgresql.org/downloads

Release 13 of the PostgreSQL BuildFarm client

I have pushed Release 13 of the PostgreSQL BuildFarm client.

Change highlights:

• add tests for a few cases that were previously missing
• more fine-grained control over which TAP test sets run
• --no-keepall can be specified on the command line
• repair a repo if it crashed during a copy operation
• make sure the repo is really clean (including free of ignored files)
• generate stack traces on CYGWIN
• collect stack traces for TAP tests
• Extract MSVC settings at runtime rather than had coding them in theconfig file (see below)
• cross version upgrade tests now run on Windows, both for msys2 and MSVC builds
• add support for inhibit-runs and force-one-run trigger files( see below)
• add experimental module for running arbitrary out of tree TAP tests
• Adjust if an upstream repo changes the default branch name (see below)
• add use_discard_caches caches setting (see below)

MSVC animals are now very much simpler to set up, and to upgrade to a new compiler. Using the new mechanism, as shown in the sample config file, all that's needed is to specify a location where the standard script vcvarsall.bat can be found. The script will then run that script and extract the settings and apply them. Tha means that upgrading to a new version of Visual Studio would entail just a one line change in the config file.

If you put a file called [animalname].inhibit-runs in the build root, all runs will be stopped until the file is removed. If you put a file called [animalname].force-one-run in the build root, each configured branch will forced to run once, and the file will be removed. These only apply if you use the run_branches.pl script.

The client should transparently deal with any change that is made in the upstream repository's default branch name. This avoids the need for a flag day when we eventually change the default branch name for postgresql, as I assume we will do before long. The branch bf_HEAD which the client creates now refers to the upstream default whatever it might be, rather than the hardcoded name 'master'. The code of the SCM module underwent quite a lot of change in order to make this happen; the checkout code had become quite long and convoluted and I had to refactor it somewhat before I was able to make and test this change. The changes have been fairly extensively tested, but I'm still slightly nervous about them. Owners are asked to report any issues promptly.

the use_discard_caches setting reflects a change in the way postgres handles this - it's now a runtime setting rather than a compile time setting. On older branches it sets "-DCLOBBER_CACHE_ALWAYS". If you use this setting don't use that define.

Downloads are available at

https://github.com/PGBuildFarm/client-code/releases and https://buildfarm.postgresql.org/downloads

Running Standalone TAP Tests

Recent discussion set me thinking about what would be involved in setting up standalone TAP tests for Postgres, to do, say, more code coverage than we get from the current core tests. So I started experimenting. And with a tiny bit of Makefile magic it turns out to be absurdly easy.

You need just two things: an entirely boilerplate Makefile and a tiny bit of glue at the top of your TAP test files.

First let's look at the Makefile. Here it is in its entirety:

TAP_TESTS = 1

PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

# install_path for PostgreSQL binaries etc
PG_INSTALL_ROOT := $(dir $(shell $(PG_CONFIG) --bindir))
export PG_INSTALL_ROOT

# where to find PostgresNode and friends
PG_NODE_LOC = $(top_srcdir)/src/test/perl
export PG_NODE_LOC

Then in your TAP test perl file(s) (which should reside in a subdirectory called "t") you put this:

use lib "$ENV{PG_NODE_LOC}";

use PostgresNode;
use Test::More;

local $ENV{PG_REGRESS} = "/bin/true";

my $node_inst = $ENV{PG_INSTALL_ROOT};

# for pre--release-14 releases you would possibly set LD_LIBRARY_PATH
# based on this. For release 14 and up PostgresNode does that for you
# via the install_path parameter.

my $node = PostgresNode->get_new_node('dummy', install_path => $node_inst);

...

That's all you need. Given that you can run your TAP tests with just a Postgres installation, as in this example:

andrew@emma:tests $ make PG_CONFIG=../inst.head.5701/bin/pg_config installcheck
rm -rf '/home/andrew/pgl/tests'/tmp_check
/usr/bin/mkdir -p '/home/andrew/pgl/tests'/tmp_check
cd ./ && TESTDIR='/home/andrew/pgl/tests' PATH="/home/andrew/pgl/inst.head.5701/bin:$PATH" PGPORT='65701' \
  top_builddir='/home/andrew/pgl/tests//home/andrew/pgl/inst.head.5701/lib/postgresql/pgxs/src/makefiles/../..' \
  PG_REGRESS='/home/andrew/pgl/tests//home/andrew/pgl/inst.head.5701/lib/postgresql/pgxs/src/makefiles/../../src/test/regress/pg_regress' \
  REGRESS_SHLIB='/src/test/regress/regress.so' \
  /usr/bin/prove -I /home/andrew/pgl/inst.head.5701/lib/postgresql/pgxs/src/makefiles/../../src/test/perl/ -I ./  t/*.pl
t/dummy.pl .. ok   
All tests successful.
Files=1, Tests=1,  3 wallclock secs ( 0.04 usr  0.01 sys +  1.22 cusr  0.36 csys =  1.63 CPU)
Result: PASS

The Makefile and a demo TAP test are at This Gitlab repo

In the next episode, we'll look at how to leverage this so that your standalone tests are run by your buildfarm animal.

Buildfarm adopts modern git naming

In keeping with modern git practice, I have renamed the default branch on both the client-code and server-code repositories to main. If you have a clone of either of these repositories, you can adapt to the change by doing the following commands in your local repo:

git checkout master
git branch -m main
git fetch
git branch --unset-upstream
git branch -u origin/main
git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main
git fetch -p

There is still a bunch of work to do to ensure that things run smoothly when source repositories such as The PostgreSQL Git repository make this change, as will surely happen at some stage. I will be testing this on a small source repository I control - the plan is that when it happens it will be transparent to buildfarm animals.

 Update (August 4 2021):

I just went to update a couple of machines with very old versions of git. They choked on these lines

git branch --unset-upstream
git branch -u origin/main

The solution is instead to edit the .git/config file directly. In these lines change "master" to "main":

[branch "main"]
    remote = origin
    merge = refs/heads/master

The rest of the recipe worked fine. In some cases you might need a final git pull.