Cascading Dependency Rebuilds for Jackson CI

aka “Github Actions for Fun and Profit!”

Jackson Component Dependencies

Jackson project consists of over 30 distinct components: there components depend on each other in a layered fashion, forming a dependency tree.
Most well-known are the three “core” components:

• jackson-core: Low-level reading and writing of JSON as token streams (and basic API for other(non-JSON) format modules): defines types like JsonParser, JsonGenerator, JsonFactory
• jackson-databind: Functionality for reading/writing Java Objects and JSON Tree (JsonNode) from/to token streams; builds on jackson-core (deserializers operate on JsonParsers and serializers on JsonGenerators). Best-known type is ObjectMapper
• jackson-annotations: Annotations used to configure aspects of jackson-databind (like matching names in JSON to Java class property names)

Core components have simple dependencies:

• jackson-core and jackson-annotations do not depend on any external package (Jackson or otherwise)
• jackson-databind depends on both jackson-core and jackson-annotations

But there are also many extension modules:

• Dataformat modules like jackson-datatype-xml that implement support for other formats, exposed through streaming APIs (JsonParser, JsonGenerator, JsonFactory subtypes) to work with format-agnostic databind (ObjectMapper)
• Datatype modules like jackson-datatype-joda that add JSON (and other supported format) read/write on 3rd party types so that you can — for example — read and write ImmutableList<LocalDateTime> values from/to YAML (by registering Joda and Guava datatype modules and using XmlMapper from jackson-dataformat-xml module)
• JAX-RS/Jakarta-RS providers for Jersey, RESTeasy for integrating Jackson into web-service frameworks
• JVM language support (jackson-module-kotlin, jackson-module-scala) for Kotlin/Scala types (and notations)
• Misc other modules from JAXB annotation module to Afterburner optimizer module

all of which depend on some or all of core components (data formats mostly on jackson-core and datatypes on jackson-databind, for example).

So we have a (subset of a) dependency tree like:

where dependencies are depicted by arrows from dependent packages to their dependencies: jackson-databind depends on jackson-core, jackson-datatype-joda depends on jackson-databind and so on.

Jackson Github Repository Structure (multi-repo)

Unlike some other projects that use so-called mono-repo structure (a single Github repository for all components of a library or framework), Jackson components are split across multiple Github repos — some in a repo of their own (like all 3 core components), others bundled together (like jackson-dataformats-binary which contains format modules for Avro, CBOR, Ion, Protobuf and Smile formats) in a multi-module repo.
We can call this a multi-repo setup.

All Jackson Github repos have Continuous Integration (CI) workflows implemented using Github Actions, to build and test (for all branches and PRs) components, and to publish SNAPSHOT versions of code pushed to branches. This allows use of SNAPSHOT dependencies across components during development.
The main build workflow is under .github/workflows/main.yml , f.ex for FasterXML/jackson-core:.github/workflows/main.yml for jackson-core repo/component.

This structure keeps things modular but has some challenges for making changes that affect modules in different repos: when adding a feature in a dependency (say, jackson-core), needed by a dependent (like jackson-databind) both need to be built, separately (first the dependency, then dependent). This happens for example when adding new functionality in streaming level, to be used by other components (databind, format modules, datatype modules).
When developing in an IDE, such changes can be co-ordinated if all relevant projects/repos are opened together; it is possible to make changes across projects, including refactoring.

But even if it is possible to handle changes wrt. syntax/API compatibility (code compiles, signatures match), it is possible that functionality/behavior changes in a way that causes breakages in dependent modules.
If that happens, unit tests on “upstream” (dependents, components that depend on changed components) often catch such breakages and they are easy to address. The problem is that there is no easy to run these tests automatically— IDEs can re-compile things, but by default they don’t run test suites.
And even if they did, this assumes that you always keep all dependents open along with whatever component you are modifying (dependency).

As a result it occasionally happens that a change made to a lower-level component (usually jackson-core or jackson-databind), successfully passing that component’s own test suite, causes “hidden” breakage(s) of a higher-level component (like a datatype module). And this breakage is only visible when the test suite of THAT dependent component is run, usually when it is being modified.
This is problematic.

As a result, I usually try to re-build dependents most likely to have issues locally, as part of making changes.
But since this is a manual process, I sometimes miss breakages; only finding them after a day (or even a week) when rebuilding the dependent package. Or worse, when someone else (contributor) does that.

Wouldn’t it be nice if it was possible to just trigger a run a build with tests on dependent packages to catch these breakages when (or ideally before…) they occur? If only there was a way to automate this…. Wait a minute! Isn’t this the exact thing Continuous Integration should help with — catch problems eagerly and automatically?

CI To The Rescue!

So what I would like to happen is something like:

1. A Jackson component like jackson-core has a push to its default branch
2. If and when full build succeeds (which it should as merging of PRs is gated by passing of component’s test suite), a rebuild of its direct dependents is triggered
3. If and when each dependency’s build (with tests) succeeds, trigger re-builds of its dependencies, in a cascading (recursive) manner (that is, repeat steps 2 and 3)

We already have some pieces of the puzzle: whenever a change is pushed to a mainline branch (Jackson has minor version branches like 2.1, … 2.18 as well as master for future 3.0 major version update), a full build — including execution of all tests — is run, and if that succeeds, a “snapshot” Maven version (like 2.17.2-SNAPSHOT during development of 2.17.2 version from 2.17 branch) is released (deployed to Sonatype OSS Maven repository).

This is how dependencies work during development: components depend on latest Snapshot builds of each other.
Non-snapshot versions are published (released) as part of official release process: these are versions that code outside of Jackson universe usually depends on (put another way, snapshot versions are essentially internal to Jackson development process).

So why do Snapshot builds matter in this context? Because this means that we only need to publish a new Snapshot version of the newly changed component itself and run build-and-test workflows for its dependents, using existing Snapshot versions of unchanged components (*).

(*) for now ignoring potential problem with concurrent changes — there are edge cases where things could still fail wrt rebuild process.

How to Implement?

Ok so conceptually this is simple: just cascade build-test workflows across all dependent repositories, recursively. A change to jackson-core triggering rebuild of jackson-databind triggering rebuild of jackson-dataformats-binary and so on.

My first inclination was to create a new dedicated Github repository, with a workflow that essentially clones other repositories, building, testing and Maven installing them in dependency order.
(technically this workflow could live on one of existing repos, but it seemed cleaner to just create new one to clearly indicate it is not part of existing components).

But while doing this would certainly be possible, it has at least one major downside: being in another repo, it cannot be directly triggered on pushes to other repos. Instead it would probably need to be run on schedule (cronjob style), once per day or so (or use a mechanism to get events from other repositories… :) )
There are other downsides as well. Such workflow would:

1. have delay of up to 24h to find breakages (depending on how often workflow would be run)
2. not be able to give much indication as to change (merge) that caused breakage (although failure itself probably would be enough to indicate this… with more work and troubleshooting)

Despite downsides I was seriously considering implementing this approach.

But then I learned about Github repository dispatch action: https://github.com/peter-evans/repository-dispatch. This action allows sending custom events from one workflow to another, within or ACROSS repositories. This can be used to “glue” together workflows to do modular rebuild, re-using existing Github workflows and just adding small cookie-cutter pieces to repos for which cascading rebuilds are needed.
Benefits over the initial idea include:

• Immediate triggering of cascading rebuild
• Easier identification of the failure root cause (being triggered on specific change)
• Modular solution: allows easy parallel builds of dependents (over explicit parallelism a custom workflow would require — doable, but once again more work)

How does repository-dispatch action work?

Repository-dispatch action is pretty simple: as per README on action’s repository: “to dispatch an event to a remote repository using a repo scoped (*) Personal Access Token” you will use something like this (example from trigger_dep_builds.yml of jackson-core):

jobs:
  trigger-dep-build:
    ...
    steps:
      - name: Repository dispatch
        uses: peter-evans/repository-dispatch@v3
        with:
          # PAT with repo (or public_repo) access, passed by caller
          token: ${{ secrets.token }}
          repository: FasterXML/jackson-databind
          event-type: jackson-core-pushed
          # Could push information on what was built but not yet
          client-payload: '{"version": "N/A" }'

(*) or: “If you will be dispatching to a public repository then you can use the more limited public_repo scope”

And on receiving end you do not need the action itself: you just add a trigger on job to trigger, like so (from new dep_build.yml workflow of jackson-databind):

name: Re-build on jackson-core push
on:
  repository_dispatch:
    types: [jackson-core-pushed]
jobs:
  build:
    # Build and test across JDKs we want to test for
    ...

Changes to Github Workflows, overview

Ok, so: given that we already had main Build-test-and-deploy-snapshot workflow under .github/workflows/main.yml what do we need to add or change?

We have basically 3 kinds of project repositories wrt. to cascading rebuild:

1. Roots: repos that do not depend on any other component but are dependencies of other repos; jackson-core and jackson-annotations
2. Branches: repos that depend on others and are also dependencies of other repos; for example jackson-databind
3. Leaves: repos that depend on others but do not have dependents; for example jackson-module-kotlin

Of these, (1) and (2) need to be able to trigger re-builds of other repos. Since there are 2 ways a re-build needs to be triggered — either a push event to repo itself OR a re-build event received — we will create a new workflow to call, to avoid code duplication.
We will also need to modify the existing build workflow ( .github/workflows/main.yml) to call this new workflow.

Conversely, (2) and (3) need to be able to receive “trigger rebuild” events — for these, we need to add new .github/workflows/dep_build.yml workflow; and for (2) that needs to call above-mentioned trigger_dep_build.yml workflow as well.

So with that we have 3 types of modifications. Since case (2) is a superset of the three, let’s have a look at specifically how jackson-databind workflows are changed. This forms the base for other 2 cases as well.

Changes to jackson-databind Github workflows

First: to be able to trigger re-build of dependent(s), we need a new workflow to be called from 2 sources: either when change is made to repo itself, resulting in a “push event”, or when repo receives a re-build event from one of its dependencies (jackson-core or jackson-annotation). We will create a “Reusable Workflow” (useful technique for reuse) as .github/workflows/trigger_dep_build.yml.
It looks like this (with some irrelevant portions removed):

name: Trigger downstream builds
on:
  workflow_call:
    secrets:
      token:
        required: true
jobs:
  trigger-dep-build:
    name: Trigger downstream builds
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        repo:
          - 'FasterXML/jackson-modules-base'
          - 'FasterXML/jackson-dataformats-binary'
          - 'FasterXML/jackson-dataformats-text'
          - 'FasterXML/jackson-dataformat-xml'
          - 'FasterXML/jackson-datatypes-collections'
          - 'FasterXML/jackson-datatypes-misc'
          - 'FasterXML/jackson-module-kotlin'
    steps:
      - name: Repository dispatch
        uses: peter-evans/repository-dispatch@v3
        with:
          token: ${{ secrets.token }}
          repository: ${{ matrix.repo }}
          event-type: jackson-databind-pushed

We will also need to make a change to existing “build-test-and-deploy” workflow (under .github/workflows/main.yml).
Modified version looks like so (with some parts removed for clarity):

name: Build and Deploy Snapshot
on:
  push:
    branches:
    - "2.18"
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        java_version: ['8', '11', '17', '21', '22']
    steps:
    - uses: actions/checkout@v4
    - name: Set up JDK
      uses: actions/setup-java@v4
      with:
        distribution: 'temurin'
        java-version: ${{ matrix.java_version }}
        cache: 'maven'
        # ... and some other settings
    - name: Build
      run: ./mvnw -B -ff -ntp clean verify
    - name: Deploy snapshot
      if: ${{ github.event_name != 'pull_request' && matrix.release_build && endsWith(steps.projectVersion.outputs.version, '-SNAPSHOT') }}
      env:
        CI_DEPLOY_USERNAME: ${{ secrets.CI_DEPLOY_USERNAME }}
        CI_DEPLOY_PASSWORD: ${{ secrets.CI_DEPLOY_PASSWORD }}
      run: ./mvnw -B -q -ff -DskipTests -ntp source:jar deploy

  #### AND HERE IS THE ADDED CODE -- EXECUTED IF "build" succeeds:
  trigger-dep-build:
    name: Trigger dep builds
    needs: [build]
    # Only for pushes to default branch, for now
    if: ${{ github.event_name == 'push' && github.ref_name == github.event.repository.default_branch }}
    # Here's how to "call" reusable workflows!
    uses: ./.github/workflows/trigger_dep_builds.yml
    secrets:
      token: ${{ secrets.REPO_DISPATCH }}

where last 7 code lines are the actual addition: nothing else needed to be changed.

One dependency we have here, though, is that we need to create a Personal Access Token (PAT), and create a matching Org-level Secret with name REPO_DISPATCH. As mentioned earlier, this PAT needs to have either general “repo” access (to target repositories) or “public_repo” (if all target repos are public).

And then finally we also want to add a new “build-as-dependent” workflow that we need to build module as a dependant. It looks like this:

name: Re-build on jackson-core push
on:
  repository_dispatch:
    types: [jackson-core-pushed]
permissions:
  contents: read
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        java_version: ['8', '17', '21']
    steps:
    - uses: actions/checkout@v4
    - name: Set up JDK
      uses: actions/setup-java@v4
      with:
        distribution: 'temurin'
        java-version: ${{ matrix.java_version }}
        cache: 'maven'
    - name: Build and test
      run: ./mvnw -B -ff -ntp clean verify
  # And let's proceed recursively...
  trigger-dep-build:
    name: Trigger downstream builds
    needs: [build]
    uses: ./.github/workflows/trigger_dep_builds.yml
    secrets:
      token: ${{ secrets.REPO_DISPATCH }}

Ok. But one thing may not be immediately obvious here — why didn’t we just trigger existing main.yml workflow of the dependent instead of creating new build job? It would be possible, but our needs for dependency rebuild are much more limited than the build for actual changes (for “push” event). Since the component itself did not change, we do not need to:

1. Validate API compatibility with dependencies (like Android SDK) that some components verify
2. Measure and publish Code Coverage metrics
3. Publish new snapshot version

so can reduce both CI running time and resource usage. Especially given that actual build-and-test is a just a one-line Maven invocation.

A counterpoint: it would be possible to alternatively modify main.yml to skip some parts based on whether it was triggered by an event or push — but this might end up being more complicated than modular approach taken here.

Challenges developing CI workflows

Implementation of this new “cascading rebuild” did take a while, although not as long as I feared. But I was reminded of one specific challenge with CI workflow development: something that I think iscommon to all CI systems I am familiar with (Jenkins, Travis, Github Action, Gitlab CI/CD): testability (or lack thereof).

It is difficult to test CI code the way one tests “regular” code. Most of the time you just need to “test in production”, try to see if changes work; or — most frustratingly — fail on silly things like TABs that your editor happened to insert. This is doubly so when developing distributed (even if in a simple way) workflows like here.

If anyone has good ideas of how to actually test Github workflows — or maybe just validate for common gotchas? — I’d be very interested in learning.

Next Steps?

Ok so the re-build system I built is pretty simple and missing things. Some obvious things to improve would be:

1. Add multi-branch capability: it is not necessary to test all maintenance branches (for example), but it would be very useful to have “dual-branch” capability at least — Jackson’s “master” branch is for developing next major version, 3.0, so it would be good to similarly test along with the latest 2.x branch (2.18). Doing this would require passing along branch name, using that to check out specific branch and so on. Completely doable, with bit of time and focus.
2. More information on dependency chain? Although it is easy to see how direct build (on push) fails, it may still be challenging to see how dependent build fails — maybe triggering event should pass along dependency chain used for triggering rebuilds?
   Either way, passing along actual information along with new remote build event would make sense.

One thing that would be nice, but that I probably won’t try to add is the ability to do dependency builds on Pull Requests (PRs): as useful as that would be — and maybe even more so, given that it could prevent breakages before they get merged! — there is one immediate problem: we should not (or perhaps, must not…) build Snapshot versions from PRs. This is both for security reasons (arbitrary PRs could add, at least temporarily, unsafe code) and for stability reasons — state of Snapshot builds would be non-linear.
So to support something like this for PRs would require a bit different approach. Possibly something like “mono-build” I sketched earlier: it is possible then to just use ./mvnw install for local Snapshot build within build environment instead of actual centralized Maven artifact repository.