Building Configurations¶

Audience: Architects, Application developers, Administrators

Timing: 1 hour 45 minutes

Overview¶

In this topic, we will build Configurations required by our AppConnect Integration Server.

We will start by going over an overview of Configuration objects. We will then describe the Tekton pipeline that builds the Configuration objects from configuration sources.

We will create the needed Git repositories, one that contains the sources of our configurations and one that contains the templates of the Configuration custom resources.

We will then install CI/CD ArgoCD application.

To install the pipeline, we will first install the needed Tekton tasks and then install the pipeline itself.

Before we can execute the pipeline we will create the needed resources, which includes SSH keys to access our Git repositories and the PVC resource.

And finally we will conclude by executing the pipeline and verifying the result.

Pre-requisites¶

Before attempting this topic, you should have successfully completed the Cloud Native AppConnect Enterprise GitOps.

Configuration objects¶

Environment-specific settings can be decoupled from an Integration Server or a BAR file so that it can can be deployed in various environments. These settings, which includes contents of server.conf.yaml, setdbparms, policy projects, keystores, etc. are represented as Configuration custom Kubernetes resources (CR) and processed by ACE operator.

For example, consider the following a server.conf.yaml, a plain text YAML file:

ResourceManagers:
  JVM:
    truststoreType: 'JKS'
    truststoreFile: '/home/aceuser/truststores/ibm-ca.jks'
    truststorePass: 'setdbparms::truststore'

    keystoreType: 'JKS'
    keystoreFile: '/home/aceuser/keystores/ibm-ace-server.jks'
    keystorePass: 'setdbparms::keystore'

  HTTPSConnector:
    ReqClientAuth: true
    CipherSpec: 'TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384'

Defaults:
  Policies:
    HTTPSConnector: 'HTTPS'

which configures an Integration server with TLS and requires client authentication to set up Mutual Authentication. The settings and the referred keystore (and truststore) can be decoupled from the Integration Server image by creating a Configuration CR (and referred in an IntegrationServer CR).

The Configuration CR of the above server.conf.yaml is created by encoding the content with base64 and creating a YAML of:

apiVersion: appconnect.ibm.com/v1beta1
kind: Configuration
metadata:
  name: ibm-example-serverconf
spec:
  type: serverconf
  data: UmVzb3VyY2VNYW5hZ2VyczoKICBKVk06CiAgICB0cnVzdHN0b3JlVHlwZTogJ0pLUycKICAgIHRydXN0c3RvcmVGaWxlOiAnL2hvbWUvYWNldXNlci90cnVzdHN0b3Jlcy9pYm0tY2EuamtzJwogICAgdHJ1c3RzdG9yZVBhc3M6ICdzZXRkYnBhcm1zOjp0cnVzdHN0b3JlJwoKICAgIGtleXN0b3JlVHlwZTogJ0pLUycKICAgIGtleXN0b3JlRmlsZTogJy9ob21lL2FjZXVzZXIva2V5c3RvcmVzL2libS1hY2Utc2VydmVyLmprcycKICAgIGtleXN0b3JlUGFzczogJ3NldGRicGFybXM6OmtleXN0b3JlJwoKICBIVFRQU0Nvbm5lY3RvcjoKICAgIFJlcUNsaWVudEF1dGg6IHRydWUKICAgIENpcGhlclNwZWM6ICdUTFNfRUNESEVfRUNEU0FfV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQnCgpEZWZhdWx0czoKICBQb2xpY2llczoKICAgIEhUVFBTQ29ubmVjdG9yOiAnSFRUUFMnCg==

Similarly, policy project configurations can be decoupled from a BAR, and stored in a Configuration CR.

There are also configurations that contains sensitive information, such a setdbparms, keystores and truststores. Contents of these sensitive configurations are instead stored in Secrets, with the help of SealedSecrets, and the secret is then referred in the Configuration CR.

To automate the process converting integration configuration to Configuration CR, we can store the configurations to a Git repository. The templates for the Configuration CR are also stored in Git repository. We then make use of the configuration pipeline to create the Configuration CR on the GitOps repository.

Subsequently the GitOps operator deploys the Configuration CR on to a OpenShift cluster.

For more information about Configuration, see Configuration reference.

Configuration pipeline¶

The pipeline has 6 tasks:

1. Clone config repo
2. Clone infra repo

3. Clone gitops repo

   Based on git-clone cluster task, execution of these tasks will clone the required repositories:

   • Configuration Repository: Contains the configuration sources in plain text. For instance, a serverconf for the Integration Server is maintained as a YAML file. The pipeline needs read-only access to this repository.

   • Infrastructure Repository: Contains templates for Configuration custom resources. These are plain YAML files, with templated place holder. The pipeline needs read-only access to this repository.

   • GitOps repository: The multi-tenancy apps repository. This is the repository the GitOps operator watches for any desire state of the application. Configurations for ACE application concern, and therefore the CRs are maintained in the the multi-tenancy apps repository. The pipeline will write the generated Configurations to this GitOps repository. The pipeline needs write access to this repository.

   The tasks will clone the repositories on shared workspace. Subsequent tasks will make use the workspace to access content of the repositories.

   The tasks will require secure access to the repositories. We will use SSH based authentication by using GitHub's deploy key.

4. Resolve props

   Pipeline properties list the Configurations and their metadata that are to be deployed from the configuration repository. These properties are maintained in pipeline_properties.yaml file. The resolve props custom task parses these properties from the properties file and emit them as results to be used by subsequent tasks.

   The tasks makes use of yq command line (CLI) tool. We will create a custom image, sh-tools which will contain the yq CLI tool.

   As this is a custom task, we need to deploy it before the pipeline is deployed.

5. Create config

   This custom task have three steps:

   1. Create the Configurations YAML files: The configuration sources from the configuration repository is converted to Configurations YAML based on the templates maintained in the infrastructure repository. The steps makes use of yq and zip CLI. Custom image, sh-tools, will contains these tools.

   2. Update the kustomization.yaml file: Configurations that are to be deployed by GitOps operator is driven by kustomize. Once the pipeline creates the Configurations, the kustomization.yaml lists the resources that are to be deployed. Therefore, the pipeline need update the kustomization.yaml file. The steps makes use of yq and realpath CLI. Custom image, sh-tools, will contains these tools.

   3. Push the changes to GitOps repository: The Configurations resources and the updated kustomization.yaml file is then pushed to the GitOps repository, under a new branch. bitnami/git image is used for this step.

   Before the pipeline can execute, this custom task needs to be deployed.

6. Create Pull Request

   A custom task that create a pull request to pull the changes to GitOps repository. The task is from tekton catalog, github-open-pr, but it is not part of the ClusterTask that comes with OpenShift pipeline operator.

Fork repos¶

We will create Configuration and Infrastructure git repositories from cloud-native-toolkit-demos organization to you organization.

The GitOps repository, which in our case is the multi-tenancy-gitops-apps repository should have been created from the template as part of the GitOps, Tekton & ArgoCD section.

In the subsequent steps, we will modify our versions of the repositories.

1. Create the Configuration repository in your GitHub organization

   Let's start by making our own copy of the Configuration repository.

   In your browser, navigate to https://github.com/cloud-native-toolkit-demos/ace-config

   Click on Use this template and select your organization, name the repository and click Create repository from template.

   

   After a few seconds the repository will be present in your chosen account.

2. Review the Configuration repository

   In this section we do not need to make any changes to the Configuration repository. So instead of cloning the repository on our local workspace, we can take a look at it in the github.com on the browser with vscode.

   If you press dot (.) key while you are in your github repository, it will open vscode on the browser. See The dot for a demo how it works.

   On the ace-config repository within your organization, press the dot (.) key:

   

   As we can see, there are three important files to consider:

   • backendurl.policyxml: a policy file, defining the URL of the backend service.

     <?xml version="1.0" encoding="UTF-8"?>
     <policies>
       <policy policyType="UserDefined" policyName="backendurl" policyTemplate="UserDefined">
         <soapEndpoint>http://soapserver.tools.svc:8080/ws</soapEndpoint>
       </policy>
     </policies>
     

     This defines a user-defined policy to configure the backend soapserver endpoint. Instead of hard-coding the endpoint in our AppConnect message flow, we de-coupled the endpoint as configuration, so that we can point to different soapserver endpoint.

     The file is stored in a folder, create-customer-policy. The folder name would be the name of our policy project.

   • create-customer-serverconf.yaml: is the serverconf configuration

     Defaults:
       policyProject: 'create-customer-policy'
     

     In our message flow, we are using the policy project as the default policy project. This way, in the message flow, we do not need to fully qualify the policy.

     Note

     There is nothing wrong in fully qualifying the policy project in a message flow. In this tutorial we have chosen to make use of default policy project to demonstrate a configuration using serverconf.

   • pipeline_properties.yaml

     integrationServer:
         configurations:
             serverconf:
             - create-customer-serverconf.yaml
             policyproject:
             - create-customer-policy
     

     Lists the configurations and their type. The pipeline will parse this YAML file and deploy only the listed configurations.

3. Create the Infrastructure repository in your GitHub organization

   In your browser, navigate to https://github.com/cloud-native-toolkit-demos/ace-infra

   Click on Use this template and select your organization, name the repository and click Create repository from template. After a few seconds the repository will be present in your chosen account.

4. Review the Infrastructure repository

   Similar to the Configuration repository, open the Infrastructure repository in the browser with vscode by pressing the dot (.) key:

   

   For this section, we are only concerned with the Configuration.yaml template file.

   The template will be used by the pipeline to render Configurations custom resources.

Set Environment variables¶

Throughout this chapter we will make use of two environment variables:

• GIT_ORG: an environment variable that identifies your GitHub organization.

• GIT_ROOT: an environment variable that identifies the root folder for your cloned git repositories.

We will make use of these environment variables to save typing and reduce errors. We have made use of these variable in the chapter GitOps, Tekton & ArgoCD.

Start a terminal window, we need to keep this terminal and the shell running throughout this chapter.

In the following command, replace <git-organization> with the GitHub organization name you chose (e.g. tutorial-org-123):

export GIT_ORG=<git-organization>

Next, set GIT_ROOT to the root folder for your cloned git repositories. For example:

export GIT_ROOT=$HOME/git/$GIT_ORG-root

You can verify your environment variables as follows:

echo $GIT_ORG 
echo $GIT_ROOT

Install ArgoCD Application¶

We will install the ArgoCD application which watches over resources related to CI/CD from the GitOps-apps repository. Specifically, it watches ace/config/argocd/ci sub-folder from the multi-tenancy-gitops-apps repository.

1. Change to root level GitOps repository

   Issue the following command to change to your GitOps repository:

   cd $GIT_ROOT/multi-tenancy-gitops
   

2. Install ACE ArgoCD Application

   Open 0-bootstrap/single-cluster/3-apps/kustomization.yaml and uncomment argocd/ace/cicd.yaml resource.

   resources:
   - argocd/ace/cicd.yaml
   #- argocd/ace/dev.yaml
   #- argocd/ace/stage.yaml
   #- argocd/ace/prod.yaml
   [...]
   

   Commit and push your changes:

   git add .
   git commit -m "Install ACE ArgoCD app"
   git push
   

3. On the ArgoCD UI, verify the reconciliation

   Soon after ArgoCD will reconcile the new changes. You should see the commit message that made the change, as well as the apps-ace-ci application:

   

   Open the apps-ace-ci application (either by going to Applications > apps-ace-ci or by clicking on the external link icon from the apps-ace-ci application icon):

   

   You should see the apps-ace-ci, which in turn install two ArgoCD applications:

   

Deploy Tekton tasks¶

There are three custom tasks to deploy:

• Resolve properties, resolve-props.
• Create configurations, create-config.
• Create pull request, github-open-pr.

Let's get the tasks deployed.

The create-config custom task makes use of yq and zip utilities. This requires custom image to be created. Therefore, before we can deploy the task, we will create the image.

We will make use of OpenShift Build to create the custom image first, and then deploy the create-config custom task along the other ones.

1. Change to CI environment directory

   cd $GIT_ROOT/multi-tenancy-gitops-apps/ace/environments/ci
   

2. Update kustomization enable build of image stream, sh-tools

   Open kustomization.yaml ($GIT_ROOT/multi-tenancy-gitops-apps/ace/environments/ci/kustomization.yaml) and uncomment:

   - imagestreams/sh-tools.yaml
   - buildconfig/sh-tools.yaml
   - roles/custom-images-builder-role.yaml
   - roles/custom-images-builder-rolebinding.yaml
   

   We are enabling OpenShift buildconfig and imagestream resources to build the yq-zip image. We also need to enable a role and rolebinding so that ArgoCD service account can manipulate build configs and image stream.

3. Commit the change

   git commit -am "Deploy build for sh-tools"
   

4. Update kustomization to enable tekton tasks

   Open kustomization.yaml ($GIT_ROOT/multi-tenancy-gitops-apps/ace/environments/ci/kustomization.yaml) and uncomment:

   - tasks/resolve-props.yaml
   - tasks/create-config.yaml
   - tasks/github-open-pr.yaml
   

5. Commit the and push the change

   git add .
   git commit -m "Deploy tekton tasks"
   git push
   

6. Verify the deployment of resources

   You can verify that ArgoCD deployed the resources using the ArgoCD UI:

   

   If you do not see change right away

   try Refreshing

   

   Once the build finished the image stream will be created. You can verify it in the OpenShift Web Console UI. Navigate to Builds > ImageStreams, making sure project ci selected in the Project dropdown:

   

   Or from command line:

   oc get is -n ci
   

   NAME       IMAGE REPOSITORY                                               TAGS     UPDATED
   sh-tools   image-registry.openshift-image-registry.svc:5000/ci/sh-tools   latest   4 minutes ago
   

7. Check the deployment of the tasks in ArgoCD UI

   Under "apps-ace-rest-ci-1" project, you should see that the tasks have been deployed:

   

   You can verify the tasks are deployed using OpenShift Web UI.

   

   You can also do it with oc CLI:

   oc get tasks -n ci
   

   and expect the following output:

   NAME             AGE
   create-config    4m11s
   github-open-pr   4m11s
   resolve-props    4m11s
   

Deploy Tekton pipeline¶

We are now ready to deploy the configuration pipeline.

1. Change to CI environment directory

   cd $GIT_ROOT/multi-tenancy-gitops-apps/ace/environments/ci
   

2. Update kustomization to enable resolve-props tasks

   Open kustomization.yaml ($GIT_ROOT/multi-tenancy-gitops-apps/ace/environments/ci/kustomization.yaml) and uncomment:

   - pipelines/ace-config.yaml
   

3. Commit the and push the change

   git add . 
   git commit -m "Deploy ace-config pipeline"
   git push
   

4. Check the deployment of the pipeline in ArgoCD UI

   Under "apps-ace-rest-ci-1" project, you should see that the task has been deployed:

   

5. Check the deployment of the pipeline in OpenShift UI

   You can also verify the deploy on OpenShift UI. From Pipelines > Pipelines, make sure Project ci is selected from the drop down menu:

   

Set up SealedSecret¶

Our configuration pipeline would need to access Git repositories for source of the configuration files and the templates for the Configurations objects. In this tutorial we will assume that our configuration repository is a private repository. As noted in this blog, Private Git Repositories, the preferred mechanism for accessing a private Git repository is to create a unique repository SSH key. In this section we will set up this mechanism.

Create SSH key pairs¶

We will create three pairs for SSH keys, one for the configuration repository, one for the infrastructure repository and one for our multi-tenancy-gitops-apps repository.

1. Change to a directory outside the git repositories

   We will generate a pair of SSH keys (public and private) but we do not want to store these in a git repositories. In this tutorial, we will assume to create and store the SSH keys in $HOME/keys directory.

   Create the directory and change working directory:

   mkdir -p $HOME/keys
   cd $HOME/keys
   

2. Create ssh key pairs

   Using ssh-keygen, we can create ssh key pair for the configuration repository:

   ssh-keygen -C "gitops-pipeline/ace-config@github.com/ace-prod-ref" -f ace-config-at-ace-prod-ref-github -N ''
   

   Filenames for the keys will start with ace-config-at-ace-prod-ref-github and we are not setting a passphrase.

   Similarly, for the infrastructure repository:

   ssh-keygen -C "gitops-pipeline/ace-infra@github.com/ace-prod-ref" -f ace-infra-at-ace-prod-ref-github -N ''
   

   And for the multi-tenancy-gitops-apps:

   ssh-keygen -C "gitops-pipeline/multi-tenancy-gitops-apps@github.com/ace-prod-ref" -f multi-tenancy-gitops-apps-at-ace-prod-ref-github -N ''
   

   If you list the files that are created in the current directory, you should see:

   tree .
   

   .
   ├── ace-config-at-ace-prod-ref-github
   ├── ace-config-at-ace-prod-ref-github.pub
   ├── ace-infra-at-ace-prod-ref-github
   ├── ace-infra-at-ace-prod-ref-github.pub
   ├── multi-tenancy-gitops-apps-at-ace-prod-ref-github
   └── multi-tenancy-gitops-apps-at-ace-prod-ref-github.pub
   
   0 directories, 6 files
   

Upload the public ssh-key¶

1. Upload the public ssh-key to the Configuration Git repository

   The public keys of our pairs need to be uploaded to our Git repositories, as Deploy keys.

   Visit your repository, go to Settings, and then to choose Deploy keys on the left hand side menu. For example, for ace-config repository:

   

   Click on Add deploy key button, give a title (say "gitops-pipeline") and copy the content of public key (for example, ace-config-at-ace-prod-ref-github.pub for ace-config repository).

   Use pbcopy in MacOS

   If you are using MacOS, you can use pbcopy CLI to copy the public key on your clipboard:

   cat ~/keys/ace-config-at-ace-prod-ref-github.pub | pbcopy
   

   

   Click on Add key button to the add the deploy key.

2. Upload the public ssh-key to the Infrastructure Git repository

   Similarly, for ace-infra repository, use content of ace-infra-at-ace-prod-ref-github.pub:

   

3. Upload the public ssh-key to the GitOps Git repository

   And finally, for multi-tenancy-gitops-apps repository, use content of multi-tenancy-gitops-apps-at-ace-prod-ref-github.pub. Make sure to check "Allow write access". The pipeline will write to this repository

   

Create a "machine user"¶

The pipeline will not only make use of git operations, but it will also make use of GitHub APIs. Once the Configurations objects are created and pushed to GitOps repository on a specific branch, the pipeline will make a pull request to merge the changes to the main (master) branch.

This requires pipeline to make use of GitHub APIs. We would need a GitHub user for this. It is recommend to make use of a "Machine User" for this purpose. The Machine User will have limited access to your repositories. Create a machine user following GitHub documentation.

In the following we will assume our "Machine User" is cicd-acme.

Create access token¶

The machine user needs an access token. This can be done by logging into github.com as the machine user, going to Settings > Developer Settings > Personal access tokens and clicking Generate new token:

On the "New personal access token" page, name your token and give repo access:

Store the access token in a safe place for now.

Give access to GitHub API¶

1. Browse to the GitOps repository, multi-tenancy-gitops-apps, Settings page

   

2. Add Machine User

   Click on Add People button.

   Use your Machine User. For instance, we will choose cicd-acme as our Machine User:

   

   Make sure that the Machine User has write permission. Our pipeline will use this user to create pull request to the repository.

Create sealed secrets¶

The private part of the SSH key need to be used by our pipeline to authenticate itself when cloning the repository. Moreover, the personal access token of the machine user will also be used the pipeline.

The keys and the token can be accessed via a kubernetes secret object. The secret object can be created by GitOps operator, but instead of storing the SSH key in a Secret manifest yaml, we will store it as SealedSecret.

A Secret manifest encodes it's data, but does not encrypts it. Therefore it is not a good idea to store a plain Secret manifest yaml in a git repository. Instead, we will use SealedSecret manifest, which encrypts a Secret manifest and generates a encrypted version of its data content. We can store this encrypted SealedSecret manifest in a Git repository. Once GitOps operator reconciles this object on to the cluster, SealedSecret operator will decrypt it and store it as a secret in the namespace. Only SealedSecret operator decrypt the SealedSecret manifest.

1. Create sealed secret for the private SSH key

   Sealed Secret operator from bitnami has already been deployed as part of work in section, Deploy services to the cluster. We will now create a sealed secret for the private SSH key.

   For this, we need to have kubeseal installed on our local environment. If you don't have have it installed, on MacOs, you can use homebrew:

   brew install kubeseal
   

   Note

   For more details (and installation for Linux), see kubeseal releases

2. Change to GitOps apps repository

   cd $GIT_ROOT/multi-tenancy-gitops-apps
   

   This repository maintains application manifests. The private key of our SSH key-pair is an application specific artifact, and therefore it is to be maintained in the application repository.

   Change to the directory where sealed secrets are stored:

   cd ace/environments/ci/secrets
   

3. Run the gitops-repo-pk-secret.sh script to generate SealedSecrets for SSH deploy keys

   gitops-repo-pk-secret.sh script make use of three environment variables:

   • GITOPS_PK_SECRET_NAME: Name of the secret. The name is also used in creating the SealedSecret manifest yaml file

   • GIT_BASEURL: Base URL of the git repository. This is needed for the tekton specific annotation on the Secret object. It is also needed to generate the host keys.

   • SSH_PRIVATE_KEY_PATH: full path to the private key.

   Assuming our private key is kept in $HOME/keys directory, then to generate the SealedSecret manifest, we can execute the script as follows for private key of ace-config deploy key:

   GITOPS_PK_SECRET_NAME=ace-config-at-github \
       GIT_BASEURL=github.com \
       SSH_PRIVATE_KEY_PATH=$HOME/keys/ace-config-at-ace-prod-ref-github \
       ./gitops-repo-pk-secret.sh
   

   A new file, named git-ssh-pk-ace-config-at-github.yaml will be created on the ace/environments/ci/secrets directory.

   Convention over configuration

   Throughout this guide we will make use of certain naming convention. For example the secret for storing private key ace-config is called ace-config-at-github. In later sections, some scripts will make use of this naming convention to require less configurations. If you choose to use different naming conventions, you may need to update those scripts.

   Similarly, for private key of ace-infra deploy key:

   GITOPS_PK_SECRET_NAME=ace-infra-at-github \
       GIT_BASEURL=github.com \
       SSH_PRIVATE_KEY_PATH=$HOME/keys/ace-infra-at-ace-prod-ref-github \
       ./gitops-repo-pk-secret.sh
   

   And, for private key of multi-tenancy-gitops-apps deploy key:

   GITOPS_PK_SECRET_NAME=multi-tenancy-gitops-apps-at-github \
       GIT_BASEURL=github.com \
       SSH_PRIVATE_KEY_PATH=$HOME/keys/multi-tenancy-gitops-apps-at-ace-prod-ref-github \
       ./gitops-repo-pk-secret.sh
   

4. Run the gitops-repo-pk-secret.sh script to generate SealedSecrets for GitHub access token

   gitops-repo-pk-secret.sh script make use of two environment variables:

   • TOKEN_SECRET_NAME: Name of the secret. The name is also used in creating the SealedSecret manifest yaml file
   • TOKEN: The Personal Access Token that will be used to authenticate a user to GitHub.

   To generated the SealedSecret for our Machine User's access token, run:

   TOKEN_SECRET_NAME=cicd-acme \
       TOKEN=<PUT_ACCESS_TOKEN_HERE> \
       ./gitops-repo-token-secret.sh
   

   replacing <PUT_ACCESS_TOKEN_HERE> with the token you have created for the Machine User earlier.

5. Check git status

   If you check git status with

   git status
   

   you should be able to confirm that:

   On branch master
   Your branch is up to date with 'origin/master'.
   
   Untracked files:
   (use "git add <file>..." to include in what will be committed)
       github-user-token-secret-cicd-acme.yaml
       gitops-repo-pk-secret-ace-config-at-github.yaml
       gitops-repo-pk-secret-ace-infra-at-github.yaml
       gitops-repo-pk-secret-multi-tenancy-gitops-apps-at-github.yaml
   
   nothing added to commit but untracked files present (use "git add" to track)
   

6. Commit the SealedSecret manifest to application repository

   git add gitops-repo-pk-secret-ace-config-at-github.yaml
   git commit -m 'Add SealedSecret for ace-config private key'
   

   git add gitops-repo-pk-secret-ace-infra-at-github.yaml
   git commit -m 'Add SealedSecret for ace-infra private key'
   

   git add  gitops-repo-pk-secret-multi-tenancy-gitops-apps-at-github.yaml
   git commit -m 'Add SealedSecret for multi-tenancy-gitops-apps private key'
   

   git add github-user-token-secret-cicd-acme.yaml
   git commit -m 'Add SealedSecret for access token for cicd.acme'
   

7. Add the SealedSecrets on apps-ace-rest-ci-1 ArgoCD application

   CI/CD related manifests are configure by apps-ace-rest-ci-1 ArgoCD application, which is configured by ace/environments/ci/kustomization.yaml. To add the SealedSecret on the cluster, add the following item under resources field:

   - secrets/github-user-token-secret-cicd-acme.yaml
   - secrets/gitops-repo-pk-secret-ace-config-at-github.yaml
   - secrets/gitops-repo-pk-secret-ace-infra-at-github.yaml
   - secrets/gitops-repo-pk-secret-multi-tenancy-gitops-apps-at-github.yaml
   

   If you take a difference of changes on the repository:

   git diff --unified=0
   

   You should see the following:

   diff --git a/ace/environments/ci/kustomization.yaml b/ace/environments/ci/kustomization.yaml
   index 5f7b008..db70c64 100644
   --- a/ace/environments/ci/kustomization.yaml
   +++ b/ace/environments/ci/kustomization.yaml
   @@ -24,0 +25,4 @@ resources:
   +- secrets/github-user-token-secret-cicd-acme.yaml
   +- secrets/gitops-repo-pk-secret-ace-config-at-github.yaml
   +- secrets/gitops-repo-pk-secret-ace-infra-at-github.yaml
   +- secrets/gitops-repo-pk-secret-multi-tenancy-gitops-apps-at-github.yaml
   

   Go to the root of the repo and then commit and push the change:

   cd $GIT_ROOT/multi-tenancy-gitops-apps
   git add .
   git commit -m 'Enable sealed secrets for repositories access'
   git push
   

Verify the Secret¶

Open up apps-ace-rest-ci-1 on ArgoCD UI:

As you can see, we have the four SealedSecrets deployed. The SealedSecret operator then decrypted them and created kubernetes Secret objects (which are owned by the corresponding SealedSecrets objects).

You can also verify that the secret is created from the OCP UI. Go to Workloads > Secrets, make sure ci project is selected. Select Label to filter the secrets and put in created-by=pipeline is the Search box:

Create PVC¶

1. Change to CI environment directory

   cd $GIT_ROOT/multi-tenancy-gitops-apps/ace/environments/ci
   

2. Update kustomization to enable ace-config pvc

   Open kustomization.yaml ($GIT_ROOT/multi-tenancy-gitops-apps/ace/environments/ci/kustomization.yaml) and uncomment:

   - pvc/ace-config-pvc.yaml
   

   Commit nd push the change:

   git add .
   git commit -m "Deploy ace-config pvc"
   git push
   

3. Check the deployment of the task in ArgoCD UI

   Under "apps-ace-rest-ci-1" project, you should see that the task has been deployed:

   

4. Check the deployment of the PVC in OpenShift UI

   You can also verify the deploy on OpenShift UI. From Storage > PersistentVolumeClaims, make sure Project ci is selected from the drop down menu:

   

   wait until the PVC bound.

Enable apps-ace-rest-dev-1¶

We need to activate apps-ace-rest-dev-1 ArgoCD application so that ArgoCD watches for changes for the development folder (ace/environments/overlays/dev) of the multi-tenancy-gitops-apps repository. Our pipeline will store custom resources and activate them in the ace/environments/base folder which referred in the ace/environments/overlays/dev/kustomization.yaml.

1. Change to root level GitOps repository

   Issue the following command to change to your GitOps repository:

   cd $GIT_ROOT/multi-tenancy-gitops
   

2. Install ACE Dev ArgoCD Application

   Open 0-bootstrap/single-cluster/3-apps/kustomization.yaml and uncomment

   - argocd/ace/dev.yaml
   

   Commit and push your changes:

   git add .
   git commit -m "Install ACE Dev"
   git push
   

3. On the ArgoCD UI, verify the reconciliation

   Soon after ArgoCD will reconcile the new changes. You should see the commit message that made the change, as well as the applications application:

   

   Open the apps-ace-dev application (either by going to Applications > apps-ace-ci or by clicking on the external link icon from the apps-ace-dev application icon):

   

   You should see the apps-ace-ci, which in turn installs the apps-ace-rest-dev-1 application:

   

   and when you open apps-ace-rest-dev-1 application, it will have an empty deployment for now:

   

Run the pipeline¶

Now that all the tasks, pipeline, and required resources are deployed, we can run the pipeline. We do this by creating a PipelineRun resource which refers to a Pipeline resource. A PipelineRun resource is a running instance of a Pipeline.

Our configuration pipeline has the following parameters:

The pipeline has the following workspaces:

A PipelineRun instantiates a Pipeline by setting its parameters and workspaces. Some of the parameters have default values (mostly based on values used in this tutorial), but they can be overridden when creating the PipelineRun instance.

One way to instantiates a Pipeline is to use the OpenShift Web Console. Alternatively you can manually create a PipelineRun resource YAML and apply it on the cluster (using oc apply -f), or you can make use of tkn CLI.

Run pipeline with OpenShift Web Console

To start the pipeline, on OpenShift Web Console, navigate to Pipelines > Pipelines and then click on ace-config pipeline. This will open the pipeline details page. Choose Start from the Actions dropdown:

Set the parameters to the appropriate values. If you are following this tutorial, you would need update the is-config-repo-url, is-infra-repo-url , git-ops-repo-url and gitops-apps-repo-full-name value. Also if you gave a different name to the PR token secret, make sure you give the name in the github-pr-token-secret-name parameter.

As for workspaces:

• For shared-workspace choose PersistentVolumeClaim from the dropdown and then select ace-config from the dropdown.

• For ace-config-repo-secret choose Secret from the dropdown and then select ace-config-at-github from the dropdown.

• For ace-infra-repo-secret choose Secret from the dropdown and then select ace-infra-at-github from the dropdown.

• For gitops-repo-secret choose Secret from the dropdown and then select gitops-repo-at-github from the dropdown.

Click Start to start the pipeline.

If everything goes well, the pipeline will successfully run:

If you now go to your multi-tenancy-gitops-apps repository, you will see a pull request created:

If you click on the pull request and check the commit, you should see the Configurations object that being pushed and update to the kustomization.yaml file.

If you merge the pull request, ArgoCD will update the change to and deploy the Configuration.

And you can also verify the Configurations object in OpenShift Web UI. On the OpenShift Web Console, navigate to Home > Search, select Configuration on the Resources dropdown, and make sure project dev selected in the Project dropdown:

This concludes the building of Configurations. In the next section we will build IntegrationServer.