This page details and compares available CI-as-a-Service (CISaaS) platforms for open source communities. Though there is no free SaaS provider for Jenkins, it is included in the comparison as that is what most LFN projects are currently using.
Work in progress as I (Trevor Bramwell ) compile research and slides from multiple presentations.
SCM Support by Platform
This table details the support for CI systems to report back the status of builds to SCM providers.
Note: a similar approach could be used for other SCMs provided there is an API for updating CI status
Build Resources and Minutes
Builds, agents, runners: these are all different names providers use to refer to the compute machine CI/CD builds run on.
|Builder Size||ARM Builders||External Builders||Platform Support|
|CircleCI||2 vCPUs, 4-8 Gb RAM, larger sizes||Yes/No||All|
|Azure Pipelines||2 vCPUs, 7 Gb RAM||Yes||All|
|Github Actions||2 vCPUs, 7 Gb RAM||Yes||All|
|GitLab-CI||1 vCPU, 3.75 Gb RAM||Yes||Linux, Windows|
Platform Support 'All' means support for Linux, OSX, Windows.
External Builders 'Yes/No' means external hardware is supported in the self-hosted (non SaaS) version.
Plans and Pricing Models
Details on 'free for open source' plans and pricing models for communities who may exceed those limits.
Minutes (per Month)
Gitlab-CI: Open source projects need to apply to the "Gold" level to bump the minute cap.
Notes on Other Systems
There are several other CI SaaS platforms out there not included in this comparison for various reasons.
After the majority of senior staff was laid off when the company was acquired, Travis-CI has essentially been put into standby mode. No new major feature or changes are expected.
- Bitbucket Pipelines
Bitbucket Pipelines requires code be hosted in Bitbucket and only provides 50 free minutes per-month for open source projects.
Originally the only place to get free Windows builds. There's little need to use them when Azure Pipelines provides all-if-not-more of the same services.
Platform Feature Comparison
This is not an exhaustive list of features across all the projects but a subset that maybe relevant to LFN projects.
Bring Your Own
Test Result Reports
Cross Project Pipelines
SSH Debug Support
Jenkins: SSH Debug Support: once a pipeline is paused users can SSH to the build machine
Azure Pipelines: Container Register is not free; Cross Project Pipelines provide no indication in the UI
Github Actions: Cross Project Pipelines provide no indication in the UI
Gitlab-CI: Matrix builds are created with dynamic child pipelines
Registry hosted by the provider to store container build images
Support for top package formats (Ex: Maven, NPM, Helm)
External machines (VMs, hardware, containers) can be connected to the platform
Test Result Reporting
Insights and statistics for test results
Reduced CI development time by supporting verifying CI configuration and builds locally
Support for caching packages or files between builds (Ex: Maven, PyPI, NPM)
Reduces job configuration needed to build the combination of multiple versions across multiple platforms
Cross Project Pipelines
Dependent builds can be easily traced across projects (not just within a project)
Pausing the execution of a build to login and debug build failures