GitLab CI has become a powerful tool widely adopted over the past decade, thanks to its robust capabilities. However, despite its growing user base and extensive online resources, developers often struggle to differentiate between best practices and common pitfalls in GitLab CI. This confusion can hinder their ability to effectively utilize GitLab CI.
This article aims to provide a comprehensive set of best practices to help developers overcome these challenges and enhance their GitLab CI proficiency. By following these guidelines, developers can avoid common anti-patterns and improve their workflows.
Table of Contents
1. Utilize Versioned Public CI Docker Images
The foundation of a containerized job in GitLab CI relies on a Docker image and an instantiated container. Many package managers and tools offer well-maintained public Docker images with pre-configured settings and dependencies. Regularly update these images to avoid using the latest tag, which can introduce unexpected breaking changes and lead to significant troubleshooting efforts.
Common Anti-pattern: Custom Docker Images
Creating custom Docker images may seem like a straightforward solution, but it involves setting up a Docker registry, configuring runners to authenticate with the registry, and managing the registry to handle disk usage. This approach can lead to maintenance challenges, as custom images become black boxes, requiring careful planning for tool upgrades and changes.
Recommended Approach: Minor Installations at Runtime
Instead of creating custom images, split jobs based on their core goals and choose images that align with the main ecosystem. Install any missing tools at runtime. This approach minimizes execution time and reduces reliance on custom images. Configure runners to use local images to avoid Docker Hub rate limits and consider using a Docker registry proxy to reduce network latency.
2. Start New Projects in a Mono-repo
When starting a new project, developers face the choice between using multiple repositories (multi-repo) or a single repository (mono-repo).
Multi-repos:
- ✅ Basic pipeline setup knowledge required
- ❌ Multiple merge requests for a single functionality
- ❌ Advanced versioning strategy needed
- ❌ Limited continuous integration and delivery without advanced tooling
- ❌ No cache sharing between modules
Mono-repos:
- ❌ Advanced pipeline setup knowledge required
- ✅ Single merge request per functionality
- ✅ No versioning strategy needed
- ✅ Enables continuous integration and delivery without advanced tooling
- ✅ Allows cache sharing between modules
Mono-repos simplify the management of common code and testing changes, making them a preferable choice for many projects.
3. Start CI with Local GitLab CI YAML Files
Instead of relying on centralized GitLab templates, start by building your CI YAML files locally. Centralization can introduce inefficiencies and challenges, such as slow feedback loops and managing template versions. Begin with local YAML files and only centralize them after multiple identical use cases emerge.
Must read: Speed Up GitLab CI/CD: 16 Effective Tips
4. Use Raw Commands in Scripts
While encapsulating commands within scripts can provide consistency between local and remote testing, it can obscure the execution process and make error identification difficult. Use raw commands in your CI/CD pipelines to maintain transparency and understanding. Minimize the number of commands per job and leverage appropriate Docker images and packaging tools to avoid lengthy job scripts.
5. Utilize workflow:rules and rules
To avoid running unnecessary jobs, use the workflow:rules keyword to determine when a pipeline is triggered and the rules keyword for fine-grained control over individual jobs. Avoid using the deprecated only and except keywords, as rules provides all the necessary functionality.
6. Abstract Duplicated Code without YAML Anchors
GitLab CI supports job templates (hidden jobs) to avoid code duplication. Use the extends and !reference keywords for abstraction. Avoid YAML anchors, which are technically complex and limited to the current YAML file. Focus on readable and maintainable CI/CD code.
7. Properly Use Artifacts and Cache
Understand the difference between artifacts and cache in GitLab CI:
- Artifacts: Transmit small files between jobs in the same pipeline.
- Cache: Handle large files and speed up pipelines.
Use artifacts for guaranteed file transmission and cache for optimizing repeated downloads. Avoid misusing these features to ensure efficient pipeline performance.
8. Split Jobs Wisely
When designing your pipeline, balance between single and multiple jobs. Consider the following:
Single Job:
- Faster pipeline
- Harder progress tracking
- Risk of hitting runner’s log limit
- Less understanding of failure causes
Multiple Jobs:
- Easier progress tracking
- Avoid custom Docker images
- More understanding of failure causes
Optimize your pipeline by splitting jobs based on requirements and avoiding over-splitting.
9. Use the needs Keyword Wisely
The needs keyword allows jobs to start as soon as the required jobs complete, enabling parallel job chains. Use this feature to improve pipeline efficiency, but avoid full needs chains that can result in unstable applications. Use synchronization jobs to manage complex pipelines.
10. Avoid Downstream Pipelines
a. Avoid Child Pipelines
Child pipelines run separately from the main pipeline, making them complex to manage. Issues include a clunky UI, limited parent pipeline control, and complex synchronization of artifacts. Avoid using child pipelines whenever possible.
b. Avoid Multi-Project Pipelines
Multi-project pipelines trigger pipelines in other projects but have limited control over triggered pipelines and access to artifacts. This approach is often unnecessary and can be avoided by using a mono-repo product.
Conclusion
By adhering to these best practices, developers can avoid common anti-patterns and optimize their GitLab CI workflows. If you have additional best practices or insights, please share them in the comments. Let’s continue to improve our GitLab CI proficiency together.
