Why you should add security very first in your CI/CD pipeline

For several years now, we have been experiencing an ever-increasing shift from classic, monolithic application development to cloud-based architecture based on microsegmentation, container technology and Kubernetes. As with every technology change, security is often implemented only at the end - if at all. Security is supposedly inconvenient, not sexy enough and the new MVP should have been released yesterday.

But what many people don't know is that in the world of containers not only the program code has to be written differently. After all, we give developers virtually full access to the system! I have never seen an administrator who willingly gives all developers root access to his machines without signing a 100 page form with blood. So why should we suddenly do this for containers?

How to implement security in CI/CD

The CI/CD pipeline is a central element of agile development. Due to its flexibility, it is increasingly susceptible to many weaknesses. Continuous Integration, CI for short, describes the software development process in which code changes are merged into one build several times a day. Continuous Delivery, or CD for short, automates the entire software release process. By using CI/CD pipelines, developers are able to provide a "constant stream of updates" fully automatically and deploy them directly into the cloud.

CI/CD pipelines help companies automate their software delivery process by initiating code builds, performing automated testing and delivering software in a staging or production environment. Due to the enormous number of different compliance requirements, test tools and the resulting complexity, automation is essential. Automated pipelines reduce manual errors, provide standardized and reproducible development feedback loops, and enable fast release cycles.

It is in the nature of things that the way orchestration works in Kubernetes can cause serious security problems at all stages of development. In addition to the root access mentioned above, there is also the problem of unauthorized integration of unknown third-party software. This problem must be countered with a real runtime security strategy that provides end-to-end protection for containers and Kubernetes systems.

Why you have to put your security on autopilot

In the same breath with Kubernetes always comes the word "DevOps" and so we end up back at automation. Automation is a major part of the DevOps culture that drives CI/CD. If we put security at the end of the development process we will definitely slow it down considerably.

An example: Imagine you are a developer. You develop a new application over several weeks. You have built it hundreds of times, tested it, told all your friends about it and are ready to show your work to the world. You want to go into production - now! Suddenly the guys from security appear out of nowhere and say: "This can't go out like this. You have to do a pentest first."

You wouldn't find that funny, would you?

The early bird catches the worm

What is known and accepted in the vernacular is no different in security. The earlier we start with security in CI/CD, the sooner we detect a possible security problem. The question many people ask themselves is "How should I start with it?"

If we look at a standard CI/CD pipeline, we have at least 5 levels which we can roughly summarize.

Step 1: Development

Unfortunately, there are only a few methods in software development that are otherwise used in classic security threat modeling. For me it turned out that a simple role play with the developers leads to the greatest success and understanding for security. We sit together with the development team and think like hackers. Because nobody knows his own problems better than the developers themselves! We ask ourselves questions such as "How would I hack my application?" or "How would I hack this feature?" . We write down the scenarios and write down what we have to do to protect our application. Support is also provided by security plug-ins that are available for the most common IDEs like Eclipse, IntelliJ, VisualStudio Code and other modern editors. These plugins help to identify potential problems while the developers are still writing the code. Additionally, there is the possibility to add pre-commit hooks to check the code against code standards and rules. Linter and other code formatters are especially useful for this.

Step 2: Commit & Build

With static code analysis, we can examine the repository before it becomes an image. Using our own case studies from the development phase "How would I hack my application?" we are able to write specific security unit tests and thus further reduce the risk.

Dependency management is particularly important, which allows us to merge third-party software with our own software. By doing without "latest" dependencies and the use of an npm/rpm/maven proxy or cache, increased security can be guaranteed. Also, foreign code should generally be scanned, since up to 90% of an application consists of third-party components. We never know who wrote the code - maybe someone who will soon attack our company?

Step 3: Acceptance

Now that we have all the components together and have successfully completed tests, we are ready for the first deployment. While Infrastructure as Code - defining a runtime environment using configuration parameters - can be a nightmare, it is a boon to security. Since the infrastructure automatically changes with every configuration change, we are able to test for vulnerabilities automatically. Is SSH root access still allowed or RDP open? Are too many replicas started? Is a memory limit set?

Step 4: Production

In the production environment we basically repeat all tests again to eliminate possible human error sources. It is especially important here that the production environment cannot be changed manually.

Step 5: Operations

Operation is monitoring. Like automation, monitoring is an essential part of the DevOps world. We have to ask ourselves how we can prove that something has improved, that real added value has been delivered to the customer and the company. If we have created a solution that nobody uses - it is not the solution.

This should include full penetration testing. For example, whenever a completely new application is released for the first time or we have rewritten the entire stack or substructure. It is not new that there is no 100 percent certainty. Therefore, in addition to the penetration tests, there should always be a Plan-B in the drawer in case we have been hacked.

But the most important thing, apart from all the safety precautions, is the human being. We should never pillory anyone for this just because a security error has been made. It brings no added value at all. On the contrary, people then only start to cover up mistakes and thereby increase the damage even further. We all learn from our mistakes.

Which tool should I choose?

A common question I am asked is which tools should be used. Of course there is a wide range of different manufacturers and solutions. But which one is the right one?

A tale: Every year in January we witness how countless people prepare for the next world championship in ultramarathon. Packed with their New Year's resolutions, they storm the fitness shops and buy the most expensive and supposedly best equipment the market has to offer. Thinking, "It was expensive, so I'm sure I'm faster and better than everyone else." But after a few weeks they stop running and all they've done is waste a lot of money.

In terms of security it is similar. There are many free solutions, some are even OpenSource, so you can start quickly and easily. A good basic set is for example the Open Policy Agent, Notary and Falco. On this basis one can build up super. A development team needs time to integrate the new checks and rules into the daily development process; they get used to it. So there is nothing wrong with starting small. It helps to show the management later that security works and that there is a real added value for the company. It is also easier to evaluate future tools and to find the right tool for the teams and for the company.

What is your experience in implementing security? Have you already started or do you not know where to start? Let me know in the comments!