Installing services with ArgoCD¶

Audience: Architects, Application developers, Administrators

Timing: Configuration 90 minutes - Waiting 60 minutes

Overview¶

In this topic, we are going to:

Deploy services to the cluster using GitOps.
Explore the ArgoCD applications that manage these services.
Review how ArgoCD projects work.
See how infra and services ArgoCD applications manage the infrastructure and services layers in our architecture.

By the end of this topic we will have a fully configured cluster which is ready for us to deploy ACE applications.

Introduction¶

In the previous section of this chapter, we used GitOps to create the ArgoCD applications that installed and managed the ci, tools and dev namespaces in the cluster.

In this section we are going to complete the installation of all the necessary services required by our IBM AppConnect Enterprise (ACE) CI/CD process:

cluster-tech

We will examine these highlighted components in more detail throughout this section of the tutorial; here is an overview of their function:

Tekton is used for Continuous Integration. Often, the Tekton pipeline will perform its changes under a pull-request (PR) to provide an explicit approval mechanism or through a push based on the requirements for cluster changes.
ArgoCD applications will be created for each of these resources. Specifically, ArgoCD applications will keep the cluster synchronized with the application YAML definitions stored in Git.

Note how these application services are installed in the ci and tools namespaces we created in the previous topic.

This section will reinforce our understanding of GitOps. We will then be ready to create ACE applications that use the infrastructure we previously created and these services we are going to install further.

Pre-requisites¶

Before attempting this section, you must have completed the following tasks:

You have created an OCP cluster instance.
You have installed on your local machine the oc command that matches the version of your cluster.
You have installed npm, git and tree commands.
You have completed the tutorial section to customize the GitOps repository, and install ArgoCD.
You have completed the tutorial section to create the ci, tools and dev namespaces using GitOps.

Please see the previous sections of this guide for information on how to do these tasks.

Video Walkthrough¶

This video demonstrates how to install Tekton. It also shows how to use the GitOps repository to set up different service related components.

This is a video walkthrough and it takes you step by step through the below sections.

Post cluster provisioning tasks¶

Red Hat OpenShift cluster¶

An OpenShift v4.7+ cluster is required.

CLI tools¶

Install the OpenShift CLI oc (version 4.7+) . The binary can be downloaded from the Help menu from the OpenShift Console.

Download oc cli

Log in from a terminal window.

oc login --token=<token> --server=<server>

IBM Entitlement Key¶

The IBM Entitlement Key is required to pull IBM Cloud Pak specific container images from the IBM Entitled Registry. To get an entitlement key,
1. Log in to MyIBM Container Software Library with an IBMid and password associated with the entitled software.
2. Select the View library option to verify your entitlement(s).
3. Select the Get entitlement key to retrieve the key.
In the following command, replace <entitlement_key> with the value of IBM Entitlement Key retrieved in the previous step.
```
export IBM_ENTITLEMENT_KEY=<entitlement_key>
```

A Secret containing the entitlement key is created in the tools namespace.

oc new-project tools || true
oc create secret docker-registry ibm-entitlement-key -n tools \
--docker-username=cp \
--docker-password="$IBM_ENTITLEMENT_KEY" \
--docker-server=cp.icr.io

which confirms that the ibm-entitlement-key secret has been created:

secret/ibm-entitlement-key created

Installing Tekton for GitOps¶

Tekton is made available to your Red Hat OpenShift cluster through the Red Hat OpenShift Pipelines operator. Let's see how to get that operator installed on your cluster.

Ensure environment variables are set

Tip

If you're returning to the tutorial after restarting your computer, ensure that the $GIT_ORG, $GIT_BRANCH and $GIT_ROOT environment variables are set.

(Replace <your organization name> appropriately:)
```
export GIT_BRANCH=master
export GIT_ORG=<your organization name>
export GIT_ROOT=$HOME/git/$GIT_ORG-root
```
You can verify your environment variables as follows:
```
echo $GIT_BRANCH
echo $GIT_ORG
echo $GIT_ROOT
```
Ensure you're logged in to the cluster

Log into your OCP cluster, substituting the --token and --server parameters with your values:
```
  oc login --token=<token> --server=<server>
```
If you are unsure of these values, click your user ID in the OpenShift web console and select "Copy Login Command".
Locate your GitOps repository

If necessary, change to the root of your GitOps repository, which is stored in the $GIT_ROOT environment variable.

Issue the following command to change to your GitOps repository:
```
cd $GIT_ROOT
cd multi-tenancy-gitops
```

Install Tekton into the cluster

We use the Red Hat Pipelines operator to install Tekton into the cluster. The sample repository contains the YAML necessary to do this. We’ll examine it later, but first let’s use it.

Open 0-bootstrap/single-cluster/2-services/kustomization.yaml and uncomment the below resources:

- argocd/operators/openshift-pipelines.yaml

Your kustomization.yaml for services should match the following:

resources:
# IBM Software

## Cloud Pak for Integration
#- argocd/operators/ibm-ace-operator.yaml
#- argocd/operators/ibm-apic-operator.yaml
#- argocd/instances/ibm-apic-instance.yaml
#- argocd/instances/ibm-apic-management-portal-instance.yaml
#- argocd/instances/ibm-apic-gateway-analytics-instance.yaml
#- argocd/operators/ibm-aspera-operator.yaml
#- argocd/operators/ibm-assetrepository-operator.yaml
#- argocd/operators/ibm-cp4i-operators.yaml
#- argocd/operators/ibm-datapower-operator.yaml
#- argocd/operators/ibm-eventstreams-operator.yaml
#- argocd/operators/ibm-mq-operator.yaml
#- argocd/operators/ibm-opsdashboard-operator.yaml
#- argocd/operators/ibm-platform-navigator.yaml
#- argocd/instances/ibm-platform-navigator-instance.yaml

## Cloud Pak for Business Automation
#- argocd/operators/ibm-cp4a-operator.yaml
#- argocd/operators/ibm-db2u-operator.yaml
#- argocd/operators/ibm-process-mining-operator.yaml
#- argocd/instances/ibm-process-mining-instance.yaml

## Cloud Pak for Data
#- argocd/operators/ibm-cp4d-watson-studio-operator.yaml
#- argocd/instances/ibm-cp4d-watson-studio-instance.yaml
#- argocd/operators/ibm-cpd-platform-operator.yaml
#- argocd/operators/ibm-cpd-scheduling-operator.yaml
#- argocd/instances/ibm-cpd-instance.yaml

## Cloud Pak for Security
#- argocd/operators/ibm-cp4s-operator.yaml
#- argocd/instances/ibm-cp4sthreatmanagements-instance.yaml

## IBM Foundational Services / Common Services
#- argocd/operators/ibm-foundations.yaml
#- argocd/instances/ibm-foundational-services-instance.yaml
#- argocd/operators/ibm-automation-foundation-core-operator.yaml
#- argocd/operators/ibm-automation-foundation-operator.yaml

## IBM Catalogs
#- argocd/operators/ibm-catalogs.yaml


# Required for IBM MQ
#- argocd/instances/openldap.yaml
# Required for IBM ACE, IBM MQ
#- argocd/operators/cert-manager.yaml
#- argocd/instances/cert-manager-instance.yaml

# Sealed Secrets
#- argocd/instances/sealed-secrets.yaml

# CICD
#- argocd/operators/grafana-operator.yaml
#- argocd/instances/grafana-instance.yaml
#- argocd/instances/artifactory.yaml
#- argocd/instances/chartmuseum.yaml
#- argocd/instances/developer-dashboard.yaml
#- argocd/instances/swaggereditor.yaml
#- argocd/instances/sonarqube.yaml
#- argocd/instances/pact-broker.yaml
# In OCP 4.7+ we need to install openshift-pipelines and possibly privileged scc to the pipeline serviceaccount
- argocd/operators/openshift-pipelines.yaml

# Service Mesh
#- argocd/operators/elasticsearch.yaml
#- argocd/operators/jaeger.yaml
#- argocd/operators/kiali.yaml
#- argocd/operators/openshift-service-mesh.yaml
#- argocd/instances/openshift-service-mesh-instance.yaml

# Monitoring
#- argocd/instances/instana-agent.yaml
#- argocd/instances/instana-robot-shop.yaml

# Spectrum Protect Plus
#- argocd/operators/spp-catalog.yaml
#- argocd/operators/spp-operator.yaml
#- argocd/instances/spp-instance.yaml
#- argocd/operators/oadp-operator.yaml
#- argocd/instances/oadp-instance.yaml
#- argocd/instances/baas-instance.yaml

patches:
- target:
    group: argoproj.io
    kind: Application
    labelSelector: "gitops.tier.layer=services,gitops.tier.source=git"
  patch: |-
    - op: add
      path: /spec/source/repoURL
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops-services.git
    - op: add
      path: /spec/source/targetRevision
      value: master
- target:
    group: argoproj.io
    kind: Application
    labelSelector: "gitops.tier.layer=applications,gitops.tier.source=git"
  patch: |-
    - op: add
      path: /spec/source/repoURL
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops-apps.git
    - op: add
      path: /spec/source/targetRevision
      value: master
- target:
    group: argoproj.io
    kind: Application
    labelSelector: "gitops.tier.layer=services,gitops.tier.source=helm"
  patch: |-
    - op: add
      path: /spec/source/repoURL
      value: https://charts.cloudnativetoolkit.dev
- target:
    name: ibm-automation-foundation-operator
  patch: |-
    - op: add
      path: /spec/source/helm/parameters/-
      value:
        name: spec.channel
        value: v1.1

Commit and push changes to your git repository:

git add .
git commit -s -m "Install tekton using Red Hat OpenShift Pipelines Operator"
git push origin $GIT_BRANCH

The changes have now been pushed to your GitOps repository:

Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 524 bytes | 524.00 KiB/s, done.
Total 6 (delta 5), reused 0 (delta 0)
remote: Resolving deltas: 100% (5/5), completed with 5 local objects.
To https://github.com/prod-ref-guide/multi-tenancy-gitops.git
   85a4c46..61e15b0  master -> master

Activate the services in the GitOps repo

Access the 0-bootstrap/single-cluster/kustomization.yaml:

cat 0-bootstrap/single-cluster/kustomization.yaml

Let us only deploy services resources to the cluster. Open 0-bootstrap/single-cluster/kustomization.yaml and uncomment 2-services/2-services.yaml as follows:

resources:
- 1-infra/1-infra.yaml
- 2-services/2-services.yaml
# - 3-apps/3-apps.yaml
patches:
- target:
    group: argoproj.io
    kind: Application
    labelSelector: "gitops.tier.layer=gitops"
  patch: |-
    - op: add
      path: /spec/source/repoURL
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops.git
    - op: add
      path: /spec/source/targetRevision
      value: master
- target:
    group: argoproj.io
    kind: AppProject
    labelSelector: "gitops.tier.layer=infra"
  patch: |-
    - op: add
      path: /spec/sourceRepos/-
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops.git
    - op: add
      path: /spec/sourceRepos/-
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops-infra.git
- target:
    group: argoproj.io
    kind: AppProject
    labelSelector: "gitops.tier.layer=services"
  patch: |-
    - op: add
      path: /spec/sourceRepos/-
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops.git
    - op: add
      path: /spec/sourceRepos/-
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops-services.git
- target:
    group: argoproj.io
    kind: AppProject
    labelSelector: "gitops.tier.layer=applications"
  patch: |-
    - op: add
      path: /spec/sourceRepos/-
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops.git
    - op: add
      path: /spec/sourceRepos/-
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops-apps.git

Once we push this change to GitHub, it will be seen by the bootstrap-single-cluster application in ArgoCD, and the resources it refers to will be applied to the cluster.

Push GitOps changes to GitHub

Let’s make these GitOps changes visible to the ArgoCD bootstrap-single-cluster application via GitHub.

Add all changes in the current folder to a git index, commit them, and push them to GitHub:

git add .
git commit -s -m "Deploying services"
git push origin $GIT_BRANCH

The changes have now been pushed to your GitOps repository:

Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 8 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 431 bytes | 431.00 KiB/s, done.
Total 5 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
To https://github.com/prod-ref-guide/multi-tenancy-gitops.git
   533602c..85a4c46  master -> master

This change to the GitOps repository can now be used by ArgoCD.

The bootstrap-single-cluster application detects the change and resyncs

Once these changes to our GitOps repository are seen by ArgoCD, it will resync the cluster to the desired new state.

Switch to the ArgoCD UI Applications view to see the start of this resync process:

Notice how the bootstrap-single-cluster application has detected the changes and is automatically synching the cluster.

(You can manually sync the bootstrap-single-cluster ArgoCD application in the UI if you don't want to wait for ArgoCD to detect the change.)
The new ArgoCD applications

After a short while, you'll see openshift-pipelines ArgoCD application have been created as follows:
Wait for the Tekton installation to complete

Installation is an asynchronous process, so we can issue a command that will complete when installation is done.

Wait 30 seconds for the installation to get started, then issue the following command:
```
sleep 30; oc wait --for condition=available --timeout 60s deployment.apps/openshift-pipelines-operator -n openshift-operators
```
After a while, you should see the following message informing us that operator installation is complete:
```
deployment.apps/openshift-pipelines-operator condition met
```
Tekton is now installed and ready to use. We’ll explore Tekton in later topics.

If you see something like:
```
Error from server (NotFound): deployments.apps "openshift-pipelines-operator" not found
```
then re-issue the command; the error occurred because the subscription took a little longer to create than expected.

Deploy services to the cluster¶

We've just had our first successful GitOps experience, using an ArgoCD application to create the ci, tools and dev namespaces in our cluster. There are few more components to create: IBM ACE, IBM Platform navigator, IBM foundations, IBM automation foundations, IBM Catalog and Sealed secrets.

These components are part of the services layer in our architecture, and that requires us to access /0-bootstrap/single-cluster/2-services within our GitOps repository.

How to deploy services

As we saw earlier, the bootstrap-single-cluster application uses the contents of the /0-bootstrap/single-cluster/2-services folder to determine which Kubernetes resources should be deployed in the cluster.

Issue the following command to see what's currently deployed in the cluster:

tree 0-bootstrap/single-cluster/2-services/ -L 3

The resources currently deployed to the cluster map directly to this folder structure:

0-bootstrap/single-cluster/
├── 1-infra
│   ├── 1-infra.yaml
│   ├── argocd
│   │   ├── consolelink.yaml
│   │   ├── consolenotification.yaml
│   │   ├── infraconfig.yaml
│   │   ├── machinesets.yaml
│   │   ├── namespace-ci.yaml
│   │   ├── namespace-dev.yaml
│   │   ├── namespace-ibm-common-services.yaml
│   │   ├── namespace-istio-system.yaml
│   │   ├── namespace-openldap.yaml
│   │   ├── namespace-openshift-storage.yaml
│   │   ├── namespace-prod.yaml
│   │   ├── namespace-sealed-secrets.yaml
│   │   ├── namespace-staging.yaml
│   │   ├── namespace-tools.yaml
│   │   └── storage.yaml
│   └── kustomization.yaml
├── 2-services
│   ├── 2-services.yaml
│   ├── argocd
│   │   ├── instances
│   │   └── operators
│   └── kustomization.yaml
├── 3-apps
│   ├── 3-apps.yaml
│   ├── argocd
│   │   ├── ace
│   │   ├── apic
│   │   ├── bookinfo
│   │   ├── mq
│   │   └── soapserver
│   └── kustomization.yaml
├── bootstrap.yaml
└── kustomization.yaml

13 directories, 23 files

This structure shows how only infrastructure ArgoCD applications -- and therefore their managed resources such as ci, tools and dev namespaces -- are deployed in the cluster.

Review ArgoCD services folder

Let’s examine the 0-bootstrap/single-cluster/2-services/kustomization.yaml to see how ArgoCD manages the resources deployed to the cluster.

Issue the following command:

cat 0-bootstrap/single-cluster/2-services/kustomization.yaml

We can see the contents of the kustomization.yaml:

resources:
# IBM Software
#- argocd/operators/ibm-ace-operator.yaml
#- argocd/operators/ibm-apic-operator.yaml
#- argocd/operators/ibm-aspera-operator.yaml
#- argocd/operators/ibm-assetrepository-operator.yaml
#- argocd/operators/ibm-cp4i-operators.yaml
#- argocd/operators/ibm-datapower-operator.yaml
#- argocd/operators/ibm-eventstreams-operator.yaml
#- argocd/operators/ibm-mq-operator.yaml
#- argocd/operators/ibm-opsdashboard-operator.yaml
#- argocd/operators/ibm-process-mining-operator.yaml
#- argocd/instances/ibm-process-mining-instance.yaml
#- argocd/operators/ibm-platform-navigator.yaml
#- argocd/instances/ibm-platform-navigator-instance.yaml

# IBM Foundations / Common Services
#- argocd/operators/ibm-foundations.yaml
#- argocd/instances/ibm-foundational-services-instance.yaml
#- argocd/operators/ibm-automation-foundation-core-operator.yaml
#- argocd/operators/ibm-automation-foundation-operator.yaml

# IBM Catalogs
#- argocd/operators/ibm-catalogs.yaml

# Required for IBM MQ
#- argocd/instances/openldap.yaml
# Required for IBM ACE, IBM MQ
#- argocd/operators/cert-manager.yaml
#- argocd/instances/cert-manager-instance.yaml

# Sealed Secrets
#- argocd/instances/sealed-secrets.yaml

# CICD
#- argocd/operators/grafana-operator.yaml
#- argocd/instances/grafana-instance.yaml
#- argocd/instances/artifactory.yaml
#- argocd/instances/chartmuseum.yaml
#- argocd/instances/developer-dashboard.yaml
#- argocd/instances/swaggereditor.yaml
#- argocd/instances/sonarqube.yaml
#- argocd/instances/pact-broker.yaml
# In OCP 4.7+ we need to install openshift-pipelines and possibly privileged scc to the pipeline serviceaccount
#- argocd/operators/openshift-pipelines.yaml


# Service Mesh
#- argocd/operators/elasticsearch.yaml
#- argocd/operators/jaeger.yaml
#- argocd/operators/kiali.yaml
#- argocd/operators/openshift-service-mesh.yaml
#- argocd/instances/openshift-service-mesh-instance.yaml

patches:
- target:
    group: argoproj.io
    kind: Application
    labelSelector: "gitops.tier.layer=services"
  patch: |-
    - op: add
      path: /spec/source/repoURL
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops-services.git
    - op: add
      path: /spec/source/targetRevision
      value: master

Add the services layer to the cluster

Note

The IBM Platform Navigator instance requires a RWX storageclass and it is set to managed-nfs-storage by default in the ArgoCD Application 0-bootstrap/single-cluster/2-services/argocd/instances/ibm-platform-navigator-instance.yaml. This storageclass is available for Red Hat OpenShift on IBM Cloud cluster provisioned from IBM Technology Zone with NFS storage selected.

Open 0-bootstrap/single-cluster/2-services/kustomization.yaml and uncomment the below resources:

- argocd/operators/ibm-ace-operator.yaml
- argocd/operators/ibm-platform-navigator.yaml
- argocd/instances/ibm-platform-navigator-instance.yaml

# IBM Foundations / Common Services
- argocd/operators/ibm-foundations.yaml
- argocd/instances/ibm-foundational-services-instance.yaml
- argocd/operators/ibm-automation-foundation-core-operator.yaml
#- argocd/operators/ibm-automation-foundation-operator.yaml

# IBM Catalogs
- argocd/operators/ibm-catalogs.yaml

# Sealed Secrets
- argocd/instances/sealed-secrets.yaml

If you take a diff, and just look for the added services:

git diff  | grep "^\+" | grep -v "^\+++"

You will have the following

+- argocd/operators/ibm-ace-operator.yaml
+- argocd/operators/ibm-platform-navigator.yaml
+- argocd/instances/ibm-platform-navigator-instance.yaml
+- argocd/operators/ibm-foundations.yaml
+- argocd/instances/ibm-foundational-services-instance.yaml
+- argocd/operators/ibm-automation-foundation-core-operator.yaml
+- argocd/operators/ibm-catalogs.yaml
+- argocd/instances/sealed-secrets.yaml

which shows the resources to be deployed for services.

Commit and push changes to your git repository:

git add .
git commit -s -m "Initial boostrap setup for services"
git push origin $GIT_BRANCH

The changes have now been pushed to your GitOps repository:

Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 564 bytes | 564.00 KiB/s, done.
Total 6 (delta 5), reused 0 (delta 0)
remote: Resolving deltas: 100% (5/5), completed with 5 local objects.
To https://github.com/prod-ref-guide/multi-tenancy-gitops.git
   b49dff5..533602c  master -> master

The intention of this operation is to indicate that we'd like the resources declared in 0-bootstrap/single-cluster/2-services/kustomization.yaml to be deployed in the cluster. Like the infra ArgoCD application, the resources created by the services ArgoCD application will manage the Kubernetes relevant services resources applied to the cluster.

Optional: Activate the services in the GitOps repo

Run this step if you skip installing tekton in the previous section. If you did not skip that step go to Step 7.

Access the 0-bootstrap/single-cluster/kustomization.yaml:

cat 0-bootstrap/single-cluster/kustomization.yaml

Let us deploy services resources to the cluster. Open 0-bootstrap/single-cluster/kustomization.yaml and uncomment 2-services/2-services.yaml as follows:

resources:
- 1-infra/1-infra.yaml
- 2-services/2-services.yaml
# - 3-apps/3-apps.yaml
patches:
- target:
    group: argoproj.io
    kind: Application
    labelSelector: "gitops.tier.layer=gitops"
  patch: |-
    - op: add
      path: /spec/source/repoURL
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops.git
    - op: add
      path: /spec/source/targetRevision
      value: master
- target:
    group: argoproj.io
    kind: AppProject
    labelSelector: "gitops.tier.layer=infra"
  patch: |-
    - op: add
      path: /spec/sourceRepos/-
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops.git
    - op: add
      path: /spec/sourceRepos/-
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops-infra.git
- target:
    group: argoproj.io
    kind: AppProject
    labelSelector: "gitops.tier.layer=services"
  patch: |-
    - op: add
      path: /spec/sourceRepos/-
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops.git
    - op: add
      path: /spec/sourceRepos/-
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops-services.git
- target:
    group: argoproj.io
    kind: AppProject
    labelSelector: "gitops.tier.layer=applications"
  patch: |-
    - op: add
      path: /spec/sourceRepos/-
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops.git
    - op: add
      path: /spec/sourceRepos/-
      value: https://github.com/prod-ref-guide/multi-tenancy-gitops-apps.git

Once we push this change to GitHub, it will be seen by the bootstrap-single-cluster application in ArgoCD, and the resources it refers to will be applied to the cluster.

Push GitOps changes to GitHub

Let’s make these GitOps changes visible to the ArgoCD bootstrap-single-cluster application via GitHub.

Add all changes in the current folder to a git index, commit them, and push them to GitHub:

git add .
git commit -s -m "Deploying services"
git push origin $GIT_BRANCH

The changes have now been pushed to your GitOps repository:

Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 8 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 431 bytes | 431.00 KiB/s, done.
Total 5 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
To https://github.com/prod-ref-guide/multi-tenancy-gitops.git
   533602c..85a4c46  master -> master

This change to the GitOps repository can now be used by ArgoCD.

The bootstrap-single-cluster application detects the change and resyncs

Once these changes to our GitOps repository are seen by ArgoCD, it will resync the cluster to the desired new state.

Switch to the ArgoCD UI Applications view to see the start of this resync process:

Notice how the bootstrap-single-cluster application has detected the changes and is automatically synching the cluster.

(You can manually sync the bootstrap-single-cluster ArgoCD application in the UI if you don't want to wait for ArgoCD to detect the change.)
The new ArgoCD applications

After a short while, you'll see lots of new ArgoCD applications have been created to manage the services we have deployed by modifying kustomization.yaml under 0-bootstrap/single-cluster/2-services folder:

See how most ArgoCD applications are Synced almost immediately, but some spend time in Progressing.

After a few minutes you'll see that all ArgoCD applications become Healthy and Synced:

Notice how many more Synched ArgoCD applications are now in the cluster; these are as a result of the newly added services layer in our architecture.
The services application

Let's examine the ArgoCD application that manage the services in our reference architecture.

In the ArgoCD UI Applications view, click on the icon for the services application:

We can see that the services ArgoCD application creates several ArgoCD applications, each of which is responsible for applying specific YAMLs to the cluster according to the folder the ArgoCD application is watching.

It’s the services ArgoCD application that watches the 0-bootstrap/single-cluster/2-services/argocd folder for ArgoCD applications that apply service resources to our cluster. It was the services application that created the artifactory ArgoCD application which manages the Artifactory instance that is used for application configuration management that we will be exploring later in this section of the tutorial.

We’ll continually reinforce these relationships as we work through the tutorial. You might like to spend some time exploring the ArgoCD UI and ArgoCD YAMLs before you proceed, though it’s not necessary, as you’ll get lots of practice as we proceed.

The services ArgoCD project

As we've seen in the ArgoCD UI, the services ArgoCD application is responsible for creating the ArgoCD applications that manage the services within the cluster. Let's examine their definitions to see how they do this.

Issue the following command:

cat 0-bootstrap/single-cluster/2-services/2-services.yaml

The following YAML may initially look a little intimidating; we'll discuss the major elements below:

---
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: services
  labels:
    gitops.tier.layer: services
spec:
  sourceRepos: [] # Populated by kustomize patches in 2-services/kustomization.yaml
  destinations:
  - namespace: tools
    server: https://kubernetes.default.svc
  - namespace: ibm-common-services
    server: https://kubernetes.default.svc
  - namespace: redhat-operators
    server: https://kubernetes.default.svc
  - namespace: openshift-operators
    server: https://kubernetes.default.svc
  - namespace: openshift-marketplace
    server: https://kubernetes.default.svc
  - namespace: ci
    server: https://kubernetes.default.svc
  - namespace: dev
    server: https://kubernetes.default.svc
  - namespace: staging
    server: https://kubernetes.default.svc
  - namespace: prod
    server: https://kubernetes.default.svc
  - namespace: sealed-secrets
    server: https://kubernetes.default.svc
  - namespace: istio-system
    server: https://kubernetes.default.svc
  - namespace: openldap
    server: https://kubernetes.default.svc
  - namespace: instana-agent
    server: https://kubernetes.default.svc
  - namespace: openshift-gitops
    server: https://kubernetes.default.svc
  clusterResourceWhitelist:
  # TODO: SCC needs to be moved to 1-infra, here for now for artifactory
  - group: "security.openshift.io"
    kind: SecurityContextConstraints
  - group: "console.openshift.io"
    kind: ConsoleLink
  - group: "apps"
    kind: statefulsets
  - group: "apps"
    kind: deployments
  - group: ""
    kind: services
  - group: ""
    kind: configmaps
  - group: ""
    kind: secrets
  - group: ""
    kind: serviceaccounts
  - group: "batch"
    kind: jobs
  - group: ""
    kind: roles
  - group: "route.openshift.io"
    kind: routes
  - group: ""
    kind: RoleBinding
  - group: "rbac.authorization.k8s.io"
    kind: ClusterRoleBinding
  - group: "rbac.authorization.k8s.io"
    kind: ClusterRole
  - group: apiextensions.k8s.io
    kind: CustomResourceDefinition
  roles:
  # A role which provides read-only access to all applications in the project
  - name: read-only
    description: Read-only privileges to my-project
    policies:
    - p, proj:my-project:read-only, applications, get, my-project/*, allow
    groups:
    - argocd-admins
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: services
  annotations:
    argocd.argoproj.io/sync-wave: "200"
  labels:
    gitops.tier.layer: gitops
spec:
  destination:
    namespace: openshift-gitops
    server: https://kubernetes.default.svc
  project: services
  source: # repoURL  and targetRevision populated by kustomize patches in 2-services/kustomization.yaml
    path: 0-bootstrap/single-cluster/2-services
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Notice how this YAML defines three ArgoCD resources: a services project which manages all the necessary services that are needed by the applications.

Notice how the destinations for the services project are limited to the ci, tools and dev namespaces -- as well as a few others that we'll use in the tutorial. These destinations restrict the namespaces where ArgoCD applications in the services project can manage resources.

The same is true for clusterResourceWhiteList. It limits the Kubernetes resources that can be managed to configmaps, deployments and rolebindings amongst others.

In summary, we see that the service project is used to group all the ArgoCD applications that will manage the services in our cluster. These ArgoCD applications can only perform specific actions on specific resource types in specific namespaces. See how ArgoCD is acting as a well-governed administrator.

The similar structure of services and infra ArgoCD applications

Even though we didn't closely examine the infra ArgoCD application YAML in the previous topic, it has has a very similar structure to the ArgoCD services applications we've just examined.

Type the following command to list the ArgoCD infra app YAML.

cat 0-bootstrap/single-cluster/1-infra/1-infra.yaml

Again, although this YAML might look a little intimidating, the overall structure is the same as for services:

---
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: infra
  labels:
    gitops.tier.layer: infra
spec:
  sourceRepos: [] # Populated by kustomize patches in 1-infra/kustomization.yaml
  destinations:
  - namespace: ci
    server: https://kubernetes.default.svc
  - namespace: dev
    server: https://kubernetes.default.svc
  - namespace: staging
    server: https://kubernetes.default.svc
  - namespace: prod
    server: https://kubernetes.default.svc
  - namespace: sealed-secrets
    server: https://kubernetes.default.svc
  - namespace: tools
    server: https://kubernetes.default.svc
  - namespace: ibm-common-services
    server: https://kubernetes.default.svc
  - namespace: istio-system
    server: https://kubernetes.default.svc
  - namespace: openldap
    server: https://kubernetes.default.svc
  - namespace: instana-agent
    server: https://kubernetes.default.svc
  - namespace: openshift-gitops
    server: https://kubernetes.default.svc
  clusterResourceWhitelist:
  - group: ""
    kind: Namespace
  - group: ""
    kind: RoleBinding
  - group: "security.openshift.io"
    kind: SecurityContextConstraints
  - group: "console.openshift.io"
    kind: ConsoleNotification
  - group: "console.openshift.io"
    kind: ConsoleLink
  roles:
  # A role which provides read-only access to all applications in the project
  - name: read-only
    description: Read-only privileges to my-project
    policies:
    - p, proj:my-project:read-only, applications, get, my-project/*, allow
    groups:
    - argocd-admins
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: infra
  annotations:
    argocd.argoproj.io/sync-wave: "100"
  labels:
    gitops.tier.layer: gitops
spec:
  destination:
    namespace: openshift-gitops
    server: https://kubernetes.default.svc
  project: infra
  source: # repoURL  and targetRevision populated by kustomize patches in 1-infra/kustomization.yaml
    path: 0-bootstrap/single-cluster/1-infra
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

As with the 2-services.yaml, we can see

An ArgoCD project called infra. ArgoCD applications defined in this project will be limited by the destinations: and clusterResourceWhitelist: specified in the YAML.
An ArgoCD app called infra. This is the ArgoCD application that we used in the previous section of the tutorial. It watches the path: 0-bootstrap/single-cluster/1-infra folder for ArgoCD applications that it applied to the cluster. It was these applications that managed the ci, tools and dev namespaces for example.

The installation of the operators will take approximately 30 - 45 minutes.

Verify the deployment of IBM App Connect operator
```
sleep 30; oc wait --for condition=available --timeout 60s deployment.apps/ibm-appconnect -n openshift-operators
```
After a while, you should see the following message informing us that operator installation is complete:
```
deployment.apps/ibm-appconnect condition met
```
If you see something like:
```
Error from server (NotFound): deployments.apps "ibm-appconnect" not found
```
then re-issue the command; the error occurred because the subscription took a little longer to create than expected.

Validate the deployment of IBM Cloud Pak for Integration Platform Navigator operator

sleep 30; oc wait --for condition=available --timeout 60s deployment.apps/ibm-integration-platform-navigator-operator -n openshift-operators

After a while, you should see the following message informing us that operator installation is complete:

deployment.apps/ibm-integration-platform-navigator-operator condition met

If you see something like:

Error from server (NotFound): deployments.apps "ibm-integration-platform-navigator-operator" not found

then re-issue the command; the error occurred because the subscription took a little longer to create than expected.

Verify the IBM Platform Navigator has been deployed successfully.
```
oc get platformnavigator -n tools -o=jsonpath='{ .items[*].status.conditions[].status }'
```
Expected output = True
The Platform Navigator console has been installed as part of the IBM Cloud Pak for Integration. We use the Platform Navigator to view our App Connect Enterprise dashboard.
Access the IBM Platform Navigator console
1. Retrieve the URL for the IBM Platform Navigator console:
```
oc get route integration-navigator-pn -n tools
```
2. Copy the URL from the HOST/PORT column and paste into a browser, prefixed with https://.
3. Retrieve login credential:
```
oc extract secret/ibm-iam-bindinfo-platform-auth-idp-credentials -n ibm-common-services --to=-
```
4. Log in to the IBM Platform Navigator console with the admin credentials.

Other important ArgoCD features¶

In this final section of this chapter, let's explore ArgoCD features you may have noticed as you explored different YAML files in this chapter:

SyncWave¶

Using SyncWave to control deployment sequencing

When ArgoCD uses a GitOps repository to apply ArgoCD applications to the cluster, it applies all ArgoCD applications concurrently. Even though Kubernetes use of eventual consistency means that resources which depend on each other can be deployed in any order, it often makes sense to help deploy certain resources in a certain order. This can avoid spurious error messages or wasted compute resources for example.

Let's compare two ArgoCD YAML files.

Firstly, let's examine the namespace-tools in the infrastructure layer:
```
cat 0-bootstrap/single-cluster/1-infra/argocd/namespace-tools.yaml
```
which shows the ArgoCD application YAML:
```
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: namespace-tools
  labels:
    gitops.tier.layer: infra
  annotations:
    argocd.argoproj.io/sync-wave: "100"
...
```
Then examine the sonarqube in the services layer:
```
cat 0-bootstrap/single-cluster/2-services/argocd/instances/sonarqube.yaml
```
which shows the ArgoCD application YAML:
```
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sonarqube
  annotations:
    argocd.argoproj.io/sync-wave: "250"
  labels:
    gitops.tier.group: cntk
    gitops.tier.layer: services
    gitops.tier.source: git
  finalizers:
    - resources-finalizer.argocd.argoproj.io
...
```
Notice the use of sync-wave: "100" for namespace-tools and how it contrasts with sync-wave: "250" for sonarqube. The lower 100 will be deployed before the higher sync-wave 250 because it only makes sense to deploy the pipelines into the tools namespace, after the tools namespace has been created.

You are free to choose any number for sync-wave. In our deployment, we have chosen 100-199 for infrastructure, 200-299 for services and 300-399 for applications; it provides alignment with high level folder numbering such as 1-infra and so on.

In our tutorial, we incrementally add infrastructure, services and applications so that you can understand how everything fits together incrementally. This makes sync-wave less important than in a real-world system, where you might be making updates to each of these deployed layers simultaneously. In such cases the use of sync-wave provides a high degree of confidence in the effectiveness of a deployment change because all changes go through in the correct order.

Congratulations!

You have installed tekton and configured the key services in the cluster to support continuous integration and continuous delivery. These have included services like Sonarqube and Artifactory. We installed them into the tools namespace we created previously.

This chapter is now complete. In the following chapters, we are going to use these components to deploy MQ applications and queue managers to the cluster.