Skip to content

GitOps and ArgoCD


CP4D services are defined, configured and changed using a GitOps model. GitOps puts git repositories and git commands such as git push (to request a change) and git merge (to approve a change) at the heart of configuration management. A GitOps approach helps an organization implement best practices in version control and release governance based on a widely used open standard -- git.

The following diagram outlines the major components in a GitOps CD process:


ArgoCD (OpenShift GitOps) applications watch a Git config repository for changes built as a result of successful pipeline runs. This repository identifies the latest version of the application using information stored in Git and an image repository. OpenShift GitOps applies the Kubernetes resources thus identified to the cluster, resulting in new or updated Kubernetes resources that represent the changed CP4D resources and its ecosystem, such as pods, routes etc. OpenShift GitOps changes are durable; they remain as defined unless and until they are explicitly changed or deleted in the GitOps repository. Moreover, if the cluster resources drift from their Git config values, ArgoCD will restore them to their desired values; only changes that are applied to the Git config repository affect the long term state of the cluster.

In this section of the tutorial, we're going to set up the GitOps repository, and install ArgoCD.

In this topic, we're going to:

  • Create a Github Organization
  • Download a sample GitOps repository and briefly review it
  • Install ArgoCD
  • Customize ArgoCD
  • Login in to the ArgoCD UI

By the end of this topic we'll have a all the basic components in place to perform GitOps in our cluster.


Before attempting this section, you must have completed the following tasks:

  • You have created an OCP cluster instance.


In macOS, many of these utilities can be installed using Homebrew

  • You have installed on your local machine the oc command that matches the version of your cluster. Use these instructions to get the latest version of oc. Use oc version to confirm that you have
    • Client Version: 4.8 or higher
    • Server Version: 4.8 or higher
  • You have also installed git, tree, jq and gh commands.

Please see these instructions for more information about how to install these prerequisites.

Setting up GitHub

In this section we're going to create the GitOps organization that contains the GitOps repositories that are used by ArgoCD to determine the state of our cluster. We'll copy set of sample GitOps repositories as a starting point; we'll make our own copy and review its contents. Later in the tutorial, we'll customize it for our cluster.

(If you'd like to re-familiarize yourself with git, GitHub or managing repositories, you'll find the following short video and short read helpful.)

  1. Create a new GitHub organization

    We're going to create a dedicated GitHub organization for the repositories used in this tutorial. This is good practice as it allows you to keep your production reference work separate from other git work, or try out multiple configurations using different organizations.

    In your browser, navigate to to create a new GitHub organization:


    Click on New organization and select the Free plan:


    Complete this page as follows:

    • A Organization account name such as tutorial-org-123
    • A Contact email such as
    • Select My personal account to associate this with your personal GitHub account
    • Click Next to create the GitHub organization

  2. Set up environment variable for your GitHub Organization

    This tutorial uses environment variables to save typing and reduce errors. We will use the $GIT_ORG environment variable to store your GitHub organization name. We'll use this variable in many subsequent commands.


    You'll find it helpful to open a new terminal window for this topic.

    In the following command, replace <git-organization> with the GitHub organization name you've just chosen (e.g. tutorial-org-123):

    export GIT_ORG=<git-organization>

    You can verify your $GIT_ORG as follows:

    echo $GIT_ORG

  3. Set the value of your working git folder

    We recommend that you keep all repositories related to your Git organization under a common folder, which is usually a child folder of your home folder, e.g. $HOME/git.

    The $GIT_ROOT environment variable will be used to identify this folder as the root folder for your cloned git repositories.

    export GIT_ROOT=$HOME/git/$GIT_ORG-root

    You can verify your $GIT_ROOT folder as follows:

    echo $GIT_ROOT

    Feel free to use a different value for $GIT_ROOT if you'd like to do so.

  4. Change to your working git folder

    The set of repositories will be cloned under the $GIT_ROOT. Let's create this folder if required, and move to it.

    Issue the following command to create the $GIT_ROOT folder and move to it:

    mkdir -p $GIT_ROOT
    cd $GIT_ROOT

    You'll now be in your working folder, for example:

    /Users/anthonyodowd/git/tutorial-org-123-root %

Clone sample repositories

Now that we have a GitHub organization and a local folder, we can populate them with the sample GitOps repositories. Later, we'll customize these repositories for our cluster.

  1. Install and authenticate with the gh cli

    To install gh cli

      brew install gh
    To validate your github authentication follow below steps:

      gh auth login -h
    What is your preferred protocol for Git operations?

    Authenticate Git with your GitHub credentials?

    How would you like to authenticate GitHub CLI?

      Login with a web browser
    First copy your one-time code: ***

    Press Enter to open in your browser...


    Enter one-time code then click on continue button


    When you see authorize GitHub CLI page click on authorize github


  2. The script

    The production reference guide provides a script that creates the GitOps repositories from a set of GitHub templates and then clones them to your local machine under the $GIT_ROOT folder.

    Issue the following command to get the script:

    curl -O && chmod +x

    You'll see the being downloaded, stored on the local file system and made executable.

      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  9006  100  9006    0     0  27779      0 --:--:-- --:--:-- --:--:-- 28681

    You can use the ls command to verify that the script is now in your local folder:

    ls -alt

    to show the files in your working folder:

    total 24
    drwxr-xr-x   3 anthonyodowd  staff    96  7 Feb 17:32 .
    -rwxrwxr-x   1 anthonyodowd  staff  8374  8 Feb 15:20
    drwxr-xr-x  17 anthonyodowd  staff   544  7 Feb 15:53 ..

    Notice that file mode for has x set to indicate that it is executable.

  3. Run the script

    Let's run the script. It will:

    • create the repositories in your GitHub organization
    • clone those repositories to your local machine for customization

    Issue the following command to run the script:

    OUTPUT_DIR=. ./

    You'll see four repositories created in your $GIT_ORG organization and cloned to your local machine:

    Client Version:
    Creating GitHub repositories and local clones in folder: tutorial-org-123-root
    Github user/org is tutorial-org-123
    Repository multi-tenancy-gitops found but not cloned... cloning repository
    Cloning into 'multi-tenancy-gitops'...
    remote: Enumerating objects: 403, done.
    remote: Counting objects: 100% (403/403), done.
    remote: Compressing objects: 100% (194/194), done.
    remote: Total 403 (delta 259), reused 328 (delta 198), pack-reused 0
    Receiving objects: 100% (403/403), 3.29 MiB | 4.98 MiB/s, done.
    Resolving deltas: 100% (259/259), done.
    Already on 'master'
    Your branch is up to date with 'origin/master'.
    Repository multi-tenancy-gitops-infra found but not cloned... cloning repository
    Cloning into 'multi-tenancy-gitops-infra'...
    Repository multi-tenancy-gitops-services found but not cloned... cloning repository
    Cloning into 'multi-tenancy-gitops-services'...
    Creating repos for Cloud Pak examples
    Repository multi-tenancy-gitops-apps found but not cloned... cloning repository
    Cloning into 'multi-tenancy-gitops-apps'...
    Your branch is up to date with 'origin/master'.

    See how the output shows that the sample repositories have been successfully created and cloned. Let's now check that this has happened.

  4. Verify the new repositories in your GitHub organization

    Let's examine the newly created repositories in your GitHub organization.

    The following command will print the URL of your Git organization:


    for example:

    Use your browser to locate this URL. See how your organization has the following repositories:


    These are the GitOps repositories that are used to sync the cluster:

    • multi-tenancy-gitops contains the YAML files that describe a set of ArgoCD applications. It's these ArgoCD applications that synchronize the cluster with the following resources:
      • infrastructure components such as Kubernetes namespaces, whose YAMLs are defined in multi-tenancy-gitops-infra
      • service components such as Artifactory, Cloud Pak services or Sonarqube, whose YAMLs are defined in multi-tenancy-gitops-services
      • application components such as queue managers, databases, workflow engines, integration servers... whose YAMLs are defined in multi-tenancy-gitops-apps
    • multi-tenancy-gitops-infra contains a library of infrastructure component YAMLs that are referenced by the ArgoCD applications. There are a fixed number of these YAMLs and they come preloaded with this repository.
    • multi-tenancy-gitops-services contains a library of service component YAMLs that are referenced by the ArgoCD applications. There are a fixed number of these and they come preloaded with this repository.
    • multi-tenancy-gitops-apps contains a the set of application component YAMLs that are references by the ArgoCD applications. There are a variable number of these, corresponding to the number of application components that are being deployed in a given configuration. This repository is populated by the Tekton pipelines using the source repositories for these components. For example, a source repository containing a database configuration might be used to build the database YAMLs for deployment of that database to the cluster.

    Feel free to examine these repositories; we'll look at them in much more detail later.

  5. Verify the new repositories cloned to your local machine

    These repositories have also been cloned to your local machine under the $GIT_ROOT folder.

    Issue the following command:

    tree . -L 1

    to see the four repositories newly cloned repositories:

    ├── multi-tenancy-gitops
    ├── multi-tenancy-gitops-apps
    ├── multi-tenancy-gitops-infra
    └── multi-tenancy-gitops-services

  6. Review the GitOps folder structure

    The multi-tenancy-gitops repository will be the focus of our attention as we configure the cluster.

    Issue the following command to show its structure:

    cd multi-tenancy-gitops
    tree . -L 1

    Notice the top level folders and files:

    ├── 0-bootstrap
    ├── LICENSE
    ├── doc
    ├── scripts
    └── setup

    The main folders perform the following roles:

    • 0-bootstrap contains different profiles that can be used to initialize the cluster ready. We will choose one of the available profiles -- single-cluster.
    • scripts and setup contain useful scripts that we'll use during this tutorial.

    Feel free to explore these folders and files. We'll examine it in much more detail throughout this tutorial, and you'll become very familiar with it by trying it out.

Installing ArgoCD

Now that we've had an initial view of the main GitOps repository multi-tenancy-gitops, let's install ArgoCD to make use of it.

We will install ArgoCD using the Red Hat OpenShift GitOps operator. We will also configure it with the appropriate access rights to limit the resources it can create in the cluster. In so doing, we ensure that ArgoCD acts like a well-governed administrator of the cluster, only doing what it needs to do. Also, we will customize our ArgoCD instance by adding necessary custom checks.

  1. Ensure you're logged in to the cluster


    Ensure your terminal window is in the multi-tenancy-gitops folder.

    Log into your OCP cluster, substituting the --token and --server parameters with your values:

    oc login --token=<token> --server=<server>

    If you are unsure of these values, click your user ID in the OpenShift web console and select Copy Login Command.

    On successful login, you'll see some basic information:

    Logged into "" as "" using the token provided.
    You have access to 63 projects, the list has been suppressed. You can list all projects with 'oc projects'
    Using project "default".

    This shows some details about your user within this cluster. (This user is different to your GitHub user.)

  2. Install ArgoCD into the cluster

    We use the Red Hat GitOps operator to install ArgoCD into the cluster. The cloned multi-tenancy-gitopsrepository contains the YAML necessary to do this. We’ll examine the YAML later; first let’s use it.

    Issue the following command to create an ArgoCD subscription in your cluster:

    oc apply -f setup/ocp4x/

    The response confirms that the below resources has been created: created created created created

    Most importantly, this command verifies that it has registered a subscription for ArgoCD to be installed in the cluster.

  3. Wait for the ArgoCD installation to complete

    Installation of ArgoCD is an asynchronous process; that's because while the previous script created the subscription for ArgoCD almost immediately, the installation of the ArgoCD operator takes some time, and the once the operator has been installed, the default instance of ArgoCD also takes a little time to create.

    We can now issue two commands which, in turn, will wait for the ArgoCD operator to be installed and for the default instance of ArgoCD to be started.

    Issue the following commands:

    while ! oc wait crd --timeout=-1s --for=condition=Established  2>/dev/null; do sleep 30; done
    while ! oc wait pod --timeout=-1s --for=condition=Ready --all -n openshift-gitops 2>/dev/null; do sleep 30; done

    After a while, you should see the following message informing us that operator installation is complete (it may tale a few minutes): condition met

    The ArgoCD operator is now installed and the default instance has been successfully created.

  4. ArgoCD clusterrole and clusterrolebinding

    ArgoCD runs under a dedicated service account. For good governance, we use a custom clusterrole and clusterrolebinding to control the specific operations this service account can perform on different resources in the cluster.

    The cluster role and the cluster role binding YAMLs are also setup along with the subscription.

    oc get clusterrole custom-argocd-cluster-argocd-application-controller
    oc get clusterrolebinding openshift-gitops-argocd-application-controller
    oc get clusterrolebinding openshift-gitops-cntk-argocd-application-controller

    You'll see the resources are created in the cluster:

    $ oc get clusterrole custom-argocd-cluster-argocd-application-controller
    NAME                                                  CREATED AT
    custom-argocd-cluster-argocd-application-controller   2021-08-27T13:35:13Z
    $ oc get clusterrolebinding openshift-gitops-argocd-application-controller
    NAME                                             ROLE                                                              AGE
    openshift-gitops-argocd-application-controller   ClusterRole/custom-argocd-cluster-argocd-application-controller   8m43s
    $ oc get clusterrolebinding openshift-gitops-cntk-argocd-application-controller
    NAME                                                  ROLE                                                              AGE
    openshift-gitops-cntk-argocd-application-controller   ClusterRole/custom-argocd-cluster-argocd-application-controller   8m45s

    The clusterrole custom-argocd-cluster-argocd-application-controller defines a specific set of specific resources that are required by the ArgoCD service account and access rights over them.

    The cluster role bindings openshift-gitops-argocd-application-controller and openshift-gitops-cntk-argocd-application-controller binds the ArgoCD service account to the cluster role above.

    In combination, these definitions limit ArgoCD to perform the minimum set of operations required to manage the cluster. This is important; it means that ArgoCD behaves as a well governed administrator of the cluster.

    We'll also see later in the tutorial how ArgoCD creates resource in the cluster.

  5. Delete default ArgoCD instance

    As we'll see in a moment, the default instance of ArgoCD, created when we install the operator, isn't sufficient for our tutorial; we have to create a new one. But before we do this, we have to delete the default instance of ArgoCD.

    Issue the following command to delete the default ArgoCD instance:

    oc delete gitopsservice cluster || true

    You will see the resources being deleted as follows: "cluster" deleted

    Now that we've deleted the default instance, we can create a customized ArgoCD instance with additional capabilities.

Creating a custom instance

The default instance of ArgoCD provides some built-in health checks for validating standard Kubernetes resources. However, these checks are not sufficient to check the health of the custom resources added by the IBM Cloud Paks; we need to create a custom instance of ArgoCD that implements some IBM Cloud Pak specific health checks.

  1. The custom ArgoCD instance

    You can explore these custom health checks in the ArgoCD YAML that we'll use to create the new ArgoCD instance:

    cat setup/ocp4x/argocd-instance/argocd-instance.yaml

    Notice, for example, a custom health check for queue managers:
      health.lua: |
        hs = {}
        if obj.status ~= nil then
          if obj.status.phase ~= nil then
            hs.message = obj.status.phase
            if obj.status.phase == "Running" then
              hs.status = "Healthy"
              hs.status = "Progressing"
            return hs
        hs.status = "Progressing"
        hs.message = "Unknown"
        return hs

    This logic is used by ArgoCD to determine when a queue manager is healthy.

    You can learn more about the ArgoCD Resource Health here and ArgoCD custom Lua health checks here.

  2. Create a custom ArgoCD instance

    Now let's create the custom ArgoCD instance using this YAML.

    Issue the following command to create a custom ArgoCD instance:

    oc apply -f setup/ocp4x/argocd-instance/ -n openshift-gitops

    The response confirms that the below resources has been created: created

    Issue the below command; it will wait to complete until ArgoCD instance pod is up and running:

    while ! oc wait pod --timeout=-1s --for=condition=ContainersReady -l -n openshift-gitops > /dev/null; do sleep 30; done

  3. Associate a certificate with the ArgoCD instance


    If you are using the Firefox browser, you will not be able to access the ArgoCD console until a valid certificate has been associated with it. Other browsers are unaffected; you can omit this step if you wish.

    Some browsers will not allow you to access the ArgoCD instance unless it has been configured with a valid certificate. The following commands will associate a valid certificate directly with the ArgoCD instance.

    mkdir $$tmp$$
    cd $$tmp$$
    ingress_secret_name=$(oc get ingresscontroller.operator default \
    --namespace openshift-ingress-operator \
    -o jsonpath='{}')
    oc extract secret/$ingress_secret_name -n openshift-ingress
    oc create secret tls -n openshift-gitops openshift-gitops-cntk-tls --cert=tls.crt --key=tls.key --dry-run=client -o yaml | oc apply -f -
    cd ..
    rm -rf $$tmp$$

  4. Launch ArgoCD

    ArgoCD can be accessed via an OpenShift route. Using a browser, navigate to the URL returned by following command:

    oc get route openshift-gitops-cntk-server -n openshift-gitops -o jsonpath='{"https://"}{}{"\n"}'

    This will list the route to the ArgoCD instance we've just created, for example:

    Copy the URL from your terminal output into your browser to launch the ArgoCD web console.

    (You can safely ignore any browser certificate warnings.)

    You should get to the ArgoCD login page:


  5. Login to ArgoCD


    When logging into ArgoCD you can select one of two different authentication mechanisms. You can select either mechanism, but if you want to have full access to do things like synching or refreshing ArgoCD Applications, you should use the mechanism that uses administrator credentials (first option below).

    You can see that the ArgoCD login page provides two different mechanism for authentication:

    1. The first mechanisms uses the ArgoCD service account credentials created at installation time. If you use this mechanism, you will be authenticated as an administrator with all associated privileges. (You don't need administrator privileges to complete the tutorial.)

      To log in using the ArgoCD administrator credentials, use admin for the username and retrieve the password from the appropriate Kubernetes secret. Use the following command to retrieve the password:

      oc extract secret/openshift-gitops-cntk-cluster -n openshift-gitops --keys="admin.password" --to=-
    2. The second mechanism uses your OpenShift Single Sign On mechanism to authenticate you. If you use this mechanism, you will be authenticated as your user, rather than an administrator, with appropriate permissions. These permissions are sufficient to complete the tutorial, but you will not be able to perform a full range of ArgoCD operations.

      You may be prompted to grant the ArgoCD Service Account access your OpenShift user account information details.


      If so, grant access to retrieve your OpenShift user information.

    Once the UI launches, you'll see:


    See how there are no ArgoCD applications active at the moment.

    In the next section of the tutorial, we'll configure ArgoCD to create the ArgoCD applications that will in turn spin up infrastructure, service, and application resources to apply to the cluster.


You've created the GitOps repository for your cluster and examined its high level structure. You also installed ArgoCD. You created a specific clusterrole and clusterrolebinding for the ArgoCD service account to ensure that it manages the cluster in a well governed manner. Finally, you launched the UI for ArgoCD; you'll make extensive use of it during this tutorial.

In the next topic of this chapter, we're going to customize the GitOps repository for your cluster and use ArgoCD to create and manage the Kubernetes resources for our CP4D deployment.