GitLab Runner: Argument List too long because of Self Signed Certificate
The other day I was configuring a new GitLab instance and added a new GitLab Runner to the instance to be able to run Terraform pipelines on Azure. After all Firewalls and Proxies allowed the general communication, I finally had the GitLab Runner listed as available in GitLab.
I quickly did setup a small pipeline in my
.gitlab-ci.yml to do a quick test.
My GitLab instance did connect to the runner, which was an Azure VM running Ubuntu. The GitLab Runner did pull the required image but then got stuck with the following error:
I tried to debug the issue. Using the first link statet in the error message just says to check if there's a
.bashrc file for the user. This wasn't the case here. The documentation recommends adding the variable
CI_DEBUG_TRACE to the pipelines stage, but this wouldn't help here as the pipeline is crashing during the preparation - a step before stage would start.
So I did start the GitLab Runner in Debug mode by stopping the regular process and hitting the following command into the cli:
This helped me to see in real time what's going at each step. After a few job runes I discovered that the settings from
config.toml, especially the contents of the
tls-ca-file parameter, get passed as environment variables to the Docker environment.
This litterally means, that the content of the certificate file gets set as an environment variable. But wait a minute, isn't there a limit for environment variables? Yes, there is a limit! A single environment variable has the maximum length of 32,760 characters. Now guess what. The ca-bundle I added was longer than that. As a single certificate easily reaches 5,300 characters you get to the limit with only 6 certificates in your bundle.
The solution is quite simple. Just reduce the length of your certificate file by removing all not urgently required CA certificates and the irrelevant informations like the common
-----BEGIN CERTIFICATE----- and
-----END CERTIFICATE----- entries.
I hope this helps someone out there debugging the same and saving a lot of hours investigating.
Your root-ca-bundle-file or a environment variable, you've set in
config.toml, is too big! Reduce the length.