Securing CICD pipelines with StackRox and Sigstore

October 9, 2023
Tags: , , , ,

How can we ensure the security of our supply chain and verify that all container images deployed in our Kubernetes clusters are both signed and protected, preventing the deployment of malicious ones? What methods can we adopt to sign and verify container images within our CI/CD pipelines, thereby bolstering the security of our DevOps workflows?

This is the third blog post of the series of DevSecOps and Securing CICD Pipelines.

Check the earlier posts in:

  1. Securing the integrity of Software Supply Chains (part 1) – https://www.opensourcerers.org/2022/11/21/securing-the-integrity-of-software-supply-chains/
  2. Building Trust in the Software Supply Chain (part 2) – https://www.opensourcerers.org/2023/04/24/building-trust-in-the-software-supply-chain/

Let’s dig in!

1. Overview

Let’s explore how open-source tools such as StackRox and Quay, combined with Sigstore, can enhance the security of our CI/CD and DevOps pipelines. Similar to our previous blog posts, we’ll begin with a standard CI/CD pipeline where we build, bake, and deploy container images in our Kubernetes/OpenShift clusters.

In this process, we’ll leverage various components to safeguard this pipeline:

  •  Cosign: Open Source tool within Sigstore project that provides image-signing and verification, so we can sign our container images, to ensure that our container image stored in our registry is signed properly and also the signature is stored within the Registry and always available to be verified.
  •  Quay: Open Source container registry that stores, builds, and deploys container images. It analyzes your images for security vulnerabilities, identifying potential issues that can help you mitigate security risks. NOTE: we will use the SaaS option, Quay.io in this blog post but it’s totally possible to reproduce this demo in the non SaaS Quay.
  •   StackRox: StackRox Kubernetes Security Platform performs a risk analysis of the container environment, delivers visibility and runtime alerts, and provides recommendations to proactively improve security by hardening the environment. Red Hat offers their supported product based on StackRox, Red Hat Advanced Security for Kubernetes.
  • ArgoCD 
  • Tekton Pipelines

Finally all the files and code used in the demo are available in the K8S sign-acs repository.

NOTE: The demonstration will be deployed on an OpenShift 4 cluster. However, it’s worth noting that vanilla Kubernetes can also be used. This can be achieved by leveraging OLM and modifying the namespaces as necessary.

2. Installing ArgoCD and Tekton

As we plan to install all components through ArgoCD and GitOps, our first step is to bootstrap the ArgoCD cluster. In this scenario, we will be utilizing OpenShift GitOps, which is built on ArgoCD.

  • Clone the repository and checkout into the proper branch:
git clone https://github.com/rcarrata/ocp4-network-security.gitgit checkout sign-acs && cd sign-imagesCode language: PHP (php)
  • Install OpenShift GitOps / Pipelines (ArgoCD and Tekton):
until kubectl apply -k bootstrap/; do sleep 2; doneCode language: JavaScript (javascript)
  • After couple of minutes check the OpenShift GitOps / ArgoCD and Pipelines / Tekton components are up && running:
ARGOCD_ROUTE=$(kubectl get route openshift-gitops-server -n openshift-gitops -o jsonpath='{.spec.host}{"\n"}')
curl -ks -o /dev/null -w "%{http_code}" https://$ARGOCD_ROUTECode language: PHP (php)

NOTE: The OpenShift Pipelines needs to be installed.

3. Quay.io Repository Setup

We require a repository to store the container images generated in our pipeline. To accomplish this, we plan to create an empty public repository on quay.io. Subsequently, we will implement the necessary Role-Based Access Control (RBAC) and security measures to enable their seamless utilization within our pipelines.

  • Add a Public Quay Repository in Quay.io (we’ve used pipelines-vote-api repository as an example):
  • In the Settings repository, add a Robot Account user and assign Write or Admin permissions to this Quay Repository:
  • Grab the QUAY_TOKEN and the USERNAME that is shown within the Robot Account generated in the previous step:

and export into your shell:

export USERNAME=<Robot_Account_Username>export QUAY_TOKEN=<Robot_Account_Token>Code language: HTML, XML (xml)

3.1 Configure Quay credentials and RBAC in K8S/OCP Cluster

Now that we’ve created a Quay account and configured the appropriate Role-Based Access Control (RBAC) with a robot account, we need to integrate these credentials into the ${Namespace} in our Kubernetes/OpenShift (OCP) cluster. This setup is essential for pulling images seamlessly within the Tekton Pipelines.

  • Export the token for the GitHub Registry / quay.io:
export QUAY_TOKEN=""export EMAIL="xxx"export USERNAME="rcarrata+acs_integration"export NAMESPACE="demo-sign"Code language: JavaScript (javascript)
  • Create the namespace for the demo-sign:
kubectl create ns ${NAMESPACE}Code language: PHP (php)
  • Generate a docker-registry secret with the credentials for Quay Registry to push/pull the images and signatures:
kubectl create secret docker-registry regcred \  --docker-server=quay.io \  --docker-username=${USERNAME} \  --docker-password=${QUAY_TOKEN} \  --docker-email=${EMAIL} \  -n ${NAMESPACE}Code language: PHP (php)
  • Add the imagePullSecret to the ServiceAccount “pipeline” in the namespace of the demo:
export SERVICE_ACCOUNT_NAME=pipelinekubectl patch serviceaccount $SERVICE_ACCOUNT_NAME \ -p "{\"imagePullSecrets\": [{\"name\": \"regcred\"}]}" -n $NAMESPACEoc secrets link pipeline regcred -n demo-signCode language: PHP (php)

4. Deploy Tekton Demo Tasks and Pipelines using GitOps

Let’s deploy now the several demo tasks and pipelines that are needed for this demo. We can deploy it manually… or use GitOps to deploy them automatically! 

  • Create an ArgoCD App for deploying all the Tekton Pipelines, Tasks and other CICD components needed for these demo:
cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: signed-pipelines-app
  namespace: openshift-gitops
spec:
  destination:
    namespace: demo-sign
    server: https://kubernetes.default.svc
  project: default
  source:
    path: sign-images/manifests
    repoURL: https://github.com/rcarrata/ocp4-network-security
    targetRevision: sign-acs
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
EOF
Code language: JavaScript (javascript)

After a couple of minutes our ArgoCD app will deploy every manifest that is needed within our Kubernetes / OpenShift cluster:

5. Install StackRox / RHACS using GitOps

It’s time to install StackRox/RHACS in our Kubernetes/OpenShift cluster. Once again, we’ll utilize GitOps to deploy all the necessary components.

cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: acs-operator
  namespace: openshift-gitops
spec:
  destination:
    namespace: openshift-gitops
    server: https://kubernetes.default.svc
  project: default
  source:
    path: apps/acs
    repoURL: https://github.com/rcarrata/acs-gitops
    targetRevision: develop
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
    - PruneLast=true
EOF
Code language: JavaScript (javascript)

* After the Argo App is fully synched and finished properly, check the StackRox / ACS route:

ACS_ROUTE=$(kubectl get route -n stackrox central -o jsonpath='{.spec.host}')curl -Ik https://${ACS_ROUTE}Code language: JavaScript (javascript)

NOTE: Check that you’re getting a 200.

  • Furthermore you can check that your cluster is secured and managed by StackRox / RHACS within the ${ACS_ROUTE}/main/clusters:

5.1 Generate Roxctl API Token within StackRox

Now that our StackRox cluster is operational and safeguarding our Kubernetes/OpenShift environment, we’re integrating it with the roxctl CLI. This integration is crucial as we’ll be utilizing the roxctl CLI within the Tekton Pipelines’ image-check tasks, connecting to the StackRox/RHACS Central API.

To facilitate this integration, we’ll generate an API Token within StackRox. Navigate to Platform Configuration -> Integrations -> Authentication Tokens -> API Token, and create a new API Token:

  • Grab the token generated, and export into the ROX_API_TOKEN variable:
export ROX_API_TOKEN="xxx"Code language: JavaScript (javascript)
roxctl --insecure-skip-tls-verify image check --endpoint $ACS_ROUTE:443 --image quay.io/centos7/httpd-24-centos7:centos7
Code language: PHP (php)
WARN:   A total of 2 policies have been violatedERROR:  failed policies found: 1 policies violated that are failing the checkERROR:  Policy "Fixable Severity at least Important" - Possible remediation: "Use your package manager to update to a fixed version in future builds or speak with your security team to mitigate the vulnerabilities."ERROR:  checking image failed after 3 retries: failed policies found: 1 policies violated that are failing the checkCode language: JavaScript (javascript)

The output of the command will show that two policies are violated, so the roxctl image check is working as expected:

NOTE: For further information around this check the ACS Integration with CI Systems guide.

5.2 Integrate Tekton Pipeline with StackRox/ACS using API Token Secrets

Now that StackRox/ACS is installed and the API Token has been generated, the next step is to integrate StackRox/ACS with our Tekton Pipelines.

To achieve this, we’ll create a Secret containing the StackRox API Token credentials. This Secret will be used by the Tekton Pipelines Tasks.

Let’s start!

  • As we discussed before, to be able to authenticate from the Tekton Pipelines towards the StackRox / ACS API, the roxctl Task used in the pipelines, needs to have both ROX_API_TOKEN (generated in one step before) and the ACS Route as well inside a K8s Secret:
cat > /tmp/roxsecret.yaml << EOF
apiVersion: v1
data:
  rox_api_token: "$(echo $ROX_API_TOKEN | tr -d '\n' | base64 -w 0)"
  rox_central_endpoint: "$(echo $ACS_ROUTE:443 | tr -d '\n' | base64 -w 0)"
kind: Secret
metadata:
  name: roxsecrets
  namespace: ${NAMESPACE}
type: Opaque
EOF

kubectl apply -f /tmp/roxsecret.yaml
Code language: JavaScript (javascript)

Now we have a K8s secret that will be used for the Tekton Tasks that will use roxctl and the ROXCTL API Token / ACS Route to connect to the StackRox API.

5.3 Integrate Quay Registry within StackRox / ACS

One last step in ACS/StackRox section, is to add the Quay registry credentials. 

  • Go to Integrations and add a new Generic Docker Registry, adding the quay.io endpoint and the Quay Robot Account credentials generated earlier: 
  • Click Test, and Save the integration within StackRox / ACS.

This will allow roxctl and ACS to analyze the images uploaded into the Quay Registry and grab the vulnerability scans analyzed and produced from Clair integrated within Quay.io.  

6. Cosign and StackRox / ACS

Now that we have in place and integrated Tekton Pipelines / Tasks, ArgoCD and StackRox / ACS, it’s time to generate and integrate our Cosign/Sigstore signatures.

6.1 Generating KeyPair with Cosign

To sign content using Cosign, a public/private keypair needs to be generated. Cosign allows the use of keys stored in Kubernetes Secrets for signing and verifying signatures.

  • To generate a secret, you must use the cosign generate-key-pair command with a k8s://[NAMESPACE]/[NAME] URI, specifying the namespace and secret name:
cosign generate-key-pair k8s://${NAMESPACE}/cosignCode language: JavaScript (javascript)
  • Once the key pair is generated, Cosign will store it in a Kubernetes secret using your current context. This secret will contain both the private and public keys, along with the password required to decrypt the private key.
kubectl get secret -n $NAMESPACE cosign -o yaml
apiVersion: v1
data:
  cosign.key: xxxx
  cosign.password: xxxx
  cosign.pub: xxxx
immutable: true
kind: SecretCode language: PHP (php)

The cosign command mentioned above prompts the user to enter the password for the private key. Users have the option to manually input the password or set an environment variable COSIGN_PASSWORD.

During image signature verification using cosign verify, the key is automatically decrypted using the password stored in the Kubernetes secret under cosign.password.

When executing cosign generate-key-pair, a cosign.pub file (the public key of the Cosign key-pair) is generated in the same folder. Additionally, the public key is available in the cosign secret within our ${NAMESPACE}, as confirmed.

6.2 Add Signature Integration within StackRox / ACS

  • Add the Cosign signature into the StackRox / ACS console using Integrations submenu. Go to Integrations, Signature, New Integration and add the following:
Integration Name - Cosign-signature
Cosign Public Name - cosign-pubkey
Cosign Key Value - Content of cosign.pub generated beforeCode language: CSS (css)

This will make available the cosign public signature generated in the step before, and that will enable ACS/StackRox to check through the System Policies, the verification check of the images that we will produce / deploy in our K8s / OCP cluster.

6.3 Add StackRox/ACS Policy Image Signature Verification

Now that we have the cosign public key present in the StackRox / ACS cluster, we need to generate a System Policy that verifies the Image Signature of the container images that we will build within our Tekton Pipelines and/or deploy in our ${NAMESPACE} inside of our K8s cluster.  

  • After it’s imported, check the policy generated and select the response method as Inform and Enforce:
  • In the policy scope restrict the Policy Scope of the Policy to the specific cluster and namespace (in my case demo-sign) and save the policy:
  • Check with the roxctl image check command the new image against the cosign public key generated:
roxctl --insecure-skip-tls-verify image check --endpoint $ACS_ROUTE:443 --image  quay.io/centos7/httpd-24-centos7:centos7 | grep -A2 Trusted
| Trusted_Signature_Image_Policy |   HIGH   |      -       | Alert on Images that have not  |      - Image signature is      | All images should be signed by ||                                |          |              |          been signed           |           unverified           |   our cosign-demo signature    |+--------------------------------+----------+--------------+--------------------------------+--------------------------------+--------------------------------+Code language: PHP (php)

With this, the new policy is ready and generates alerts in the StackRox / ACS cluster, checking that the Container Image checked is not signed with the Cosign public key that we defined before.

NOTE: For more information around this check the StackRox / ACS official guide around signature verification.

7. Running Tekton Pipelines with Image Signature Verification

We’re ready to run our first Tekton Pipeline that will include Image Signature Verification from ACS/StackRox!

  • Run the pipeline for build the image, push to the Quay registry, sign the image with cosign, push the signature of the image to the Quay registry:
kubectl create -f run/sign-images-pipelinerun.yaml
  • The steps within the Tekton Pipeline will be as depicted below:

1. Clone Repository

2. Build Container Image and Push to Quay

3. Sign the Container Image generated with the Cosign Private Key and push the signature to Quay

4. Deploy the k8s deployment for the application

5. Update the k8s deployment for the application with the signed image tag

  • Check in Quay that effectively the Container Image (with v2 tag) pushed is signed properly as you can check in the picture:

as we can check the message in the pic says: “This Tag has been signed via cosign”. So our pipeline generated the Container Image, signed with Cosign and pushed both, the Image and the Signature to the Quay registry.

  • This Tekton Pipeline will deploy the signed image and also will be validated against StackRox/ACS Trusted Image Verification system policy:
kubectl get deploy -n workshop pipelines-vote-apiNAME                 READY   UP-TO-DATE   AVAILABLE   AGEpipelines-vote-api   1/1     1            1           29hCode language: JavaScript (javascript)

It works!

8. Trying to hack our Tekton Pipeline with unsigned Container Images

Let’s simulate a hacking attempt on our pipeline! Imagine a rogue user or hacker gaining access to our source code, introducing CVEs that could potentially be exploited to access our systems.

Without proper Sign and Verify steps in our pipelines and Kubernetes/OpenShift clusters for our container images, the hacker could create a malicious container image using the modified source code or a rogue Dockerfile/Containerfile. This image could mimic our legitimate application, even sharing the exact same name as our deployed application. Such security breaches introduce severe risks to our supply chain security.

So, how do we fix this vulnerability? Let’s harness the power of Signing and Verifying container images using StackRox/ACS and Sigstore to fortify our security measures!

  • Run the pipeline for build the image and push to the Quay registry, but this time without sign with cosign private key:
kubectl create -f run/unsigned-images-pipelinerun.yaml
  • The pipeline will fail because ACS/StackRox through the roxtctl image check task will detect that an unsigned image is built during the process and will be used for the app k8s deployment:
  • As we can see in the logs, the step of check-image failed, because StackRox / ACS policy blocked the pipeline due to a policy failure (enforced by the Trusted Signature Image Policy).
  • As you can check in the pipeline we have the full output of the image check with the rationale of the policy violation:

StackRox / ACS saved the day, blocking the pipeline that tried to hack our CICD / container supply chain! 

8.1 Checking the Violations in StackRox / ACS

Let’s finally check the violations detected by ACS, related to the Container Image Verification system policies.

  • If we go to the StackRox / ACS console, in the Violations dashboard we can check a Violation related with our Tekton Pipeline that generates a Container Image that was not properly signed in our cluster: 
  • Digging a bit deeper we can verify that the Policy that blocked our Tekton Pipeline and failed our CICD process was the Trusted Signature Image Policy:

Every image within our namespace, once built, must be signed using our Cosign private key, and the resulting signature needs to be pushed to the Quay registry. Although we are currently using Quay, it’s worth noting that this approach can be extended to other registries as well.

And with that ends the third blog post around Securing the software supply chain with Open Source tooling.

Happy Hacking!

Authors

Roberto Carratalá

EMEA Senior Cloud Services Black Belt in Red Hat specializing in Container Orchestration Platforms (OpenShift & Kubernetes), Cloud Services, DevSecOps, and CICD. Although I work for Red Hat I still consider myself a Linux Geek and a Unix fanboy, always searching for new tech to learn and try!

Rodrigo Alvares

EMEA Senior Specialist Solutions Architect in Red Hat specializing in Security, Containers and Kubernetes. Brazilian who has been living abroad for quite a while and working with Open-Source software since 1998. Apart from my professional career, I am also a Brazilian Jiujitsu black belt, which reflects my dedication and passion towards martial arts.

Leave a Reply

close

Subscribe to our newsletter.

Please select all the ways you would like to hear from Open Sourcerers:

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please visit our website.

We use Mailchimp as our newsletter platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.