What are best practices for minimizing CI Check times within a Code Repository?

Question

Are there things I can do to reduce the amount of time it takes for Code Repo checks to run?

This is a bit of a high level question and it may get close due to general stack overflow community guidelines on ambiguity of the question and potential answers. My recommendation is that you list what is making your CI check slow and what you are running in it (how many libs you are importing, tests that you are running, is this python, java, sql, etc etc...) — fmsf, Mar 04 '22 at 07:47
I would invite you to look at your CI run and check which task is actually slow. — fmsf, Mar 04 '22 at 07:53

score 3 · Answer 1 · answered Mar 08 '22 at 19:39

If you're curious to get more detail about which step in your checks is taking the longest, you can add the following to your transforms-python/build.gradle file at the end:

import java.util.concurrent.TimeUnit
// Log timings per task.
class TimingsListener implements TaskExecutionListener, BuildListener {
    private long startTime
    private timings = []
    @Override
    void beforeExecute(Task task) {
        startTime = System.nanoTime()
    }
    @Override
    void afterExecute(Task task, TaskState taskState) {
        def ms = TimeUnit.MILLISECONDS.convert(System.nanoTime() - startTime, TimeUnit.NANOSECONDS);
        timings.add([ms, task.path])
        task.project.logger.warn "${task.path} took ${ms}ms"
    }
    @Override
    void buildFinished(BuildResult result) {
        println "Task timings:"
        for (timing in timings) {
            if (timing[0] >= 50) {
                printf "%7sms  %s\n", timing
            }
        }
    }
    @Override
    void buildStarted(Gradle gradle) {}
    @Override
    void projectsEvaluated(Gradle gradle) {}
    @Override
    void projectsLoaded(Gradle gradle) {}
    @Override
    void settingsEvaluated(Settings settings) {}
}
gradle.addListener new TimingsListener()

This can often reveal which step is the slowest, which when used in combination with the other answer can decrease your check times in a targeted fashion.

score 1 · Answer 2 · answered Mar 04 '22 at 08:01

This is a very high level question, so I'm just dropping some bullet points as an answer, since there are so many things that can contribute to a long CI Check run. Once you make your question more deterministic I'll edit this.

Out of the top of my head here are the some common reasons why this happens:

You are importing a wide range of libraries, some with cross dependencies, bloating up your conda library resolution and making it take forever. A good resolution here would be to manually pin versions, to bypass the conda library resolution making it substancially faster.
Are you changing libraries to experiment with things? This will invalidate your conda cache forcing a cache rebuild which takes longer.
Are you on prem or within a Tennant with restricted internet access? Some steps of the CI download packages from the web, this can take time if your institution deployed foundry with slow internet access.
Your tests take a long time to run? Refactor your tests or optimize them somehow.
Are you in a monorepo? These often bloat in size and the CI takes longer because there is so much stuff in them.

What are best practices for minimizing CI Check times within a Code Repository?

2 Answers2