Thoughts on how to organize / maintain CI jobs/tests/stages

Wondering if anyone has any pointers or wants to share tips on how best to organize CI (continuous integration) jobs/tests when dealing with a mono repo.

Here's what we have

Jenkins job

  • some of these are simple jobs
    • When triggered it will run a specific test suite.
  • some of these are matrix jobs
    • A job that runs multiple test suites in parallel. They are grouped together into a matrix job because they are similar in some way. Either product area, or test type

Different States for each Jenkins job or test suite. At any time each job or test suite will be treated as one of the following

  • Blocking
    • if it fails this is critical and should be fixed
  • Debug
    • Test is flapping or being worked on and is useful to be run, but should not be treated as critical if failed
  • Disabled
    • for some reason or other, this test cannot be deleted but should not be run for the time being

Different test Stages

  • pre-commit tests
    • these are run before code is merged/landed into a branch
  • post-commit tests
    • these are run after each commit into a branch
  • nightly tests
    • these are run less frequently because they take longer or have external / complicated dependencies

How we are managing this :

  • with a largish YAML file that lists all jobs/suites and maps them to different States and Stages
  • a series of scripts that run before each Stage run
    • parse the YAML file and determine for the current Stage, what jobs/suites should be run
    • passes this information downstream to the individual Jenkins jobs that try to do the right thing with this information

Why I'm asking for advice

  • I'm curious to see how other people solve these types of problems
  • Our solution sometimes feels a bit unwieldy and difficult to maintain
  • It is not very intuitive when someone who isn't familiar with it needs to make a change
  • Not easy to test this setup

I'm wondering if we are going about this whole thing wrong and maybe there is an easier way to approach this and still have it meets our needs. Perhaps some tool that does a lot of this for us.

I'm more interested in seeing how others might have implemented a setup like this.

For the time being we cannot get away from having a mono repo, so splitting things out isn't feasible. It might be possible to do some light refactoring, or perhaps breaking the YAML file up into many YAML files and have them live in different parts of the repo.

Thanks in advance for your thoughts.

submitted by
---