Wednesday, November 18, 2026

New Security Infrastructure for PostgreSQL

Since June 2026, PostgreSQL security fixes have had a home of their own: a private git repository holding every pending fix as a stack of patches, and a private buildfarm that builds and tests that stack across every supported branch, continuously, before anything goes public.

The problem

Security fixes are the one kind of PostgreSQL change that can’t use the normal development process. They can’t be posted to pgsql-hackers, they can’t go into a public commitfest, and they can’t be pushed to postgresql.git and left for the public buildfarm to shake out. They have to stay under embargo until release day.

That left a gap. A fix might be reviewed carefully, but until wrap day it was a patch file passed around among a small group, or stashed in our bug tracker, often with a separate version for each of six or seven branches. Nobody was running it on the range of platforms, compilers and configurations that the buildfarm covers for ordinary commits. Then on wrap day, patches were pushed and the first broad test those fixes got is the public buildfarm, a few hours before tarballs are made.

That’s a lot of risk concentrated in one day, moreover the day when a broken build is most expensive.

The security repository

The core of the new setup is pgsql-security, a private git repository that the security team and committers working on security issues can access. It’s organised as a git quilt: one directory per supported branch, each holding a series file and the patches it lists, in the order they will be applied.

The patches are ordinary git format-patch output, so each carries the author, the commit message and the backpatch information that will end up in postgresql.git. A patch that applies unchanged to several branches can be listed in each branch’s series as a relative path (../master/foo.patch) instead of being copied.

Alongside the quilt, individual fixes in progress can get their own branches, which is where work happens before a fix is ready to join the stack. Work that shouldn’t be built, such as fuzzing harnesses, is kept apart from those.

A few small tools sit at the top of the repository:

  • import-series.pl applies one branch’s series to a working tree with git am. This is the important one; more on it below.
  • apply.pl runs import-series.pl for every branch. With a prefix, it builds local test branches; with an empty prefix, it produces the branches that get pushed on wrap day.
  • series-add-file.pl adds a patch to a series file along with its tracker reference.
  • check.py runs consistency checks across the whole quilt, such as whether each patch is present in every branch its backpatch information says it should be.
  • committers lists the author identities postgresql.git will accept.

The result is that on wrap day, pushing the security fixes is far more automated. It’s one command, run against a stack that has already been tested.

Buildfarm support

The second piece is a separate, private instance of the buildfarm, the SecurityFarm. It runs the same server code as the public buildfarm, with its own branding so nobody confuses the two. Its machines run the standard buildfarm client and report to the private server, not the public one.

A machine can take part in two ways.

Building the fix branches. A machine can point its scmrepo at the security repository and build the individual fix branches directly, using branches_to_build to skip the ones that aren’t meant to be built. This is the closest thing to a CI run for a single fix under development.

Building the patch stack. This is the more interesting mode. The machine builds the normal upstream branches from postgresql.git, and the new PatchStack client module applies the security quilt on top of each one before the build starts, then puts the tree back afterwards. Configuration looks like this:

modules => [ qw( PatchStack ... ) ],

patch_stack => {
    repo => 'https://.../pgsql-security.git',
},

So every patch-stack run answers the question that matters on release day: does current upstream, plus every pending security fix, build and pass the tests on this platform?

A few details make that answer trustworthy.

  • The machine applies the stack with the wrap’s own code. If the patches repository has an import-series.pl at the top, PatchStack runs it instead of its own apply loop. That’s the same script apply.pl uses on wrap day, so the machine tests what the wrap will produce, not an approximation of it.
  • Changes to the stack trigger builds. The normal trigger is a new commit on the upstream branch. PatchStack adds a second one: it hashes the net diff the stack makes to the tree, with line numbers stripped out, and forces a run when that changes. A patch edited, added, dropped or reordered triggers a build. An unrelated upstream commit that merely shifts line numbers does not.
  • A broken stack is reported as broken. If a patch is missing or doesn’t apply, the run stops and reports a PatchStackBroken failure naming the patch, rather than building whatever did apply.
  • The report says what was tested. The build report on the server shows the list of applied patches, so anyone looking at a failure can see exactly which fixes were in the tree.

What this solves

The biggest change is in timing. Problems that used to show up on wrap day now show up weeks earlier, on a private farm, where fixing them costs nothing but a revised patch.

Several of the checks exist because of specific failure modes we ran into while building this:

Failure mode What catches it now
A fix breaks the build or tests on a platform nobody on the security team uses SecurityFarm machines build the full stack on every branch, continuously
Two fixes that are fine on their own conflict with each other The stack is applied and tested as a whole, in wrap order
A patch stops applying because upstream moved underneath it Every run applies the stack to current upstream, so this shows up within hours
A patch applies, but in the wrong place import-series.pl applies with full context. git quiltimport, which we started with, let context shrink to a single line and misapplied patches as a result
A patch listed in series is missing, so a branch quietly ships without it import-series.pl refuses to start, and the machine reports PatchStackBroken. git quiltimport skipped missing entries and exited zero, giving a green build with a fix absent
A patch’s author is a personal address, so gitmaster rejects the push on wrap day import-series.pl checks every From: against committers, so the farm goes red instead
What gets tested drifts from what gets pushed The farm and the wrap run the same script

The last row is the principle behind the rest. Where possible, there is one piece of code that turns the quilt into commits, and both the buildfarm and the wrap run it. If you change how the quilt is applied, the farm starts testing the new behaviour on its next run.

It’s also simply easier to work this way. The current state of every pending fix is in one place, with history. Revising a fix is a commit. And wrap day for security fixes becomes a single apply.pl run and a push.

Beyond security

The PatchStack module is part of the standard buildfarm client, and nothing in it is specific to security. Any quilt-style repository with one directory of patches per branch will work. A repository that doesn’t supply its own import-series.pl gets the module’s built-in git apply loop. If you maintain a set of patches you carry on top of PostgreSQL, you can use it to build and test them on your own animals.

No comments:

Post a Comment