GitLab Runner: Argument List too long because of Self Signed Certificate

The other day I was configuring a new GitLab instance and added a new GitLab Runner to the instance to be able to run Terraform pipelines on Azure.  After all Firewalls and Proxies allowed the general communication, I finally had the GitLab Runner listed as available in GitLab.

I quickly did setup a small pipeline in my .gitlab-ci.yml to do a quick test.

image:
  name: hashicorp/terraform:0.15.3

stages:
  - test

test:
  stage: test
  script:
    - terraform version

My GitLab instance did connect to the runner, which was an Azure VM running Ubuntu. The GitLab Runner did pull the required image but then got stuck with the following error:

Running with gitlab-runner 13.11.0 (7f7a4bb0)
  on azure Lgbh_aXr
  
Preparing the "docker" executor
Using Docker executor with image hashicorp/terraform:0.15.3 ...
Pulling docker image hashicorp/terraform:0.15.3 ...
Using docker image sha256:944d[...]7d44 for 
hashicorp/terraform:0.15.3 with digest 
hashicorp/terraform@sha256:ac06[...]0f1c ...

Preparing environment
standard_init_linux.go:219: exec user process caused: argument list too long
ERROR: Job failed (system failure): prepare environment: exit code 1. 
Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading 
for more information

I tried to debug the issue. Using the first link statet in the error message just says to check if there's a .bash_logout  or .bashrc  file for the user. This wasn't the case here. The documentation recommends adding the variable CI_DEBUG_TRACE to the pipelines stage, but this wouldn't help here as the pipeline is crashing during the preparation - a step before stage would start.

So I did start the GitLab Runner in Debug mode by stopping the regular process and hitting the following command into the cli:

gitlab-runner --debug run

This helped me to see in real time what's going at each step. After a few job runes I discovered that the settings from  config.toml, especially the contents of the tls-ca-file parameter, get passed as environment variables to the Docker environment.

This litterally means, that the content of the certificate file gets set as an environment variable. But wait a minute, isn't there a limit for environment variables? Yes, there is a limit! A single environment variable has the maximum length of 32,760 characters. Now guess what. The ca-bundle I added was longer than that. As a single certificate easily reaches 5,300 characters you get to the limit with only 6 certificates in your bundle.

The solution is quite simple. Just reduce the length of your certificate file by removing all not urgently required CA certificates and the irrelevant informations like the common -----BEGIN CERTIFICATE-----  and -----END CERTIFICATE----- entries.

I hope this helps someone out there debugging the same and saving a lot of hours investigating.

tl;dr

Your root-ca-bundle-file or a environment variable, you've set in config.toml, is too big! Reduce the length.