Skip to content

Performance Testing


Audience: Architects, Application developers

Performance testing is the process of testing the behavior of an application under load. During performance testing, attributes like the system's scalability, reliability, and resource usage are assessed. Basically, it is a non-functional testing approach used to measure system parameters according to their responsiveness and stability under various workload conditions.

In this topic, we're going to look into the below:

  • Performance Testing Pipeline
  • Performance Testing for QueueManager
  • Performance Testing for MQ Sample application

The contents in this guide are only examples. We demonstrated some of the practices using some sample usecases. For real world scenarios, it should be used in conjunction with existing performance practices and this content is not a replacement. For assistance, CSMs should reach out to performance SMEs.

Performance Testing Pipeline

Performance test pipeline


  • Clones the source repositories.
  • Validates if the deliverable is Queue Manager.
  • Performs load testing on Queue Manager using cphtestp tool.


  • Clones the source repositories.
  • Grabs the route of the deliverable.
  • Performs load testing on the MQ sample spring application using jmeter test scripts.


  • Returns the name of the deliverable.


  • Clones the gitops apps repository.
  • Copies the deliverables from staging to prod.
  • Updates the helm chart with latest resources.
  • Generate a Pull Request with the changes to gitops apps repository.

Performance Test Task

In our sample performance test task, we are using

  • IBM MQ C Performance Harness (MQ-CPH) tool to perform tests on the QueueManager. The live data from MQ-CPH tool can be visualized using charts.
  • Jmeter to perform tests on the sample Spring Boot application. The live data from Jmeter will be fed into a datastore. Based on this data, the metrics can be visualized using Grafana dashboard.

We will take a deep look into the task definition in the coming sections.

Performance Testing for QueueManager

IBM MQ C Performance Harness

IBM MQ C Performance Harness Tool is a native MQI interface performance test tool. It is built on top of C/C++ and is largely based on the function and externals of the JMSPerfHarness Performance Harness for Java Message Service application. Basically, MQ-CPH is designed to test the native, proprietary IBM MQ interface (MQI).

  • It provides a complete set of performance messaging functionality as well as many other features such as throttled operation (a fixed rate and/or number of messages), multiple destinations, live performance reporting.
  • It will print the current throughput rate on a user-selected periodic basis.
  • It also output summary statistics at the end of a test.

Sample tests

Our tests for the Queue Manager include the below configurations.

  • MQ_QMGR_NAME, Queue Manager Name - name of the queue manager
  • MQ_QMGR_HOSTNAME, Hostname where QM is running - queue manager host name
  • MQ_QMGR_PORT, Port where QM listener is running - queue manager port
  • MQ_QMGR_CHANNEL, Channel name to use to connect to QM - channel name
  • MQ_RESPONDER_THREADS, Number of responder threads to run - 64
  • MQ_NON_PERSISTENT, QOS to be used by connecting clients - 0 or 1
  • MQ_RESULTS_CSV, Log results to csv file and send to stdout at end - true or false
  • MQ_RESULTS, Log results to stdout at end of tests - true or false

Running tests

This tool runs as a container. For more details on how to build it, check cphtestp repository.

For running this cphtestp performance tests inside Red Hat OpenShift container platform, we defined the tests as a kubernetes job and the details can be found in the next section.

Task definition for cphtestp performance tests

In the performance test task definition, the following step defines the performance tests for the Queue Manager.

- name: cphtestp
  workingDir: $(params.source-dir)
  script: |
    #!/usr/bin/env bash
    set -ex
    SRC_ENV=$(echo $(params.src-environment) | cut -d "-" -f2)
    if [ ! -f /source/route.txt ]; then
      QMGR_NAME=$(oc get qmgr -n ${SRC_ENV} | awk 'NR>1 { printf sep $1; sep=" "}')
      MQ_QM_NAME=$(oc get qmgr ${QMGR_NAME} -o jsonpath='{}' -n ${SRC_ENV})
      QMGR_HOST_NAME=$(oc get svc ${QMGR_NAME}-ibm-mq -o jsonpath='{}' -n ${SRC_ENV})
      QMGR_HTTP_PORT=$(oc get svc ${QMGR_NAME}-ibm-mq -o jsonpath='{.spec.ports[1].port}' -n ${SRC_ENV})
      oc delete job cphtestp -n ${SRC_ENV} || true
    cat <<EOF | oc apply -f -
    apiVersion: batch/v1
    kind: Job
        run: cphtestp
      name: cphtestp
      namespace: ${SRC_ENV}
            run: cphtestp
          - env:
            - name: LICENSE
              value: accept
            - name: MQ_QMGR_HOSTNAME
              value: ${QMGR_HOST_NAME}
            - name: MQ_QMGR_PORT
              value: "${QMGR_HTTP_PORT}"
            - name: MQ_USERID
            - name: MQ_PASSWORD
            - name: MQ_QMGR_CHANNEL
              value: IBM.APP.SVRCONN
            - name: MQ_RESPONDER_THREADS
              value: "64"
            - name: MQ_NON_PERSISTENT
              value: "0"
            - name: MQ_QMGR_NAME
              value: ${MQ_QM_NAME}
            - name: MQ_DATA
            - name: MQ_RESULTS_CSV
              value: "true"
            - name: MQ_RESULTS
              value: "false"
            imagePullPolicy: Always
            name: cphtestp
                memory: "2Gi"
                cpu: "2"
                memory: "2Gi"
                cpu: "2"
            terminationMessagePath: /dev/termination-log
              - name: cphdata
                mountPath: /var/cph
          dnsPolicy: ClusterFirst
          restartPolicy: OnFailure
          securityContext: {}
          terminationGracePeriodSeconds: 30
          - name: cphdata
            emptyDir: {}
      oc get pods -n ${SRC_ENV}
      oc get jobs -n ${SRC_ENV}
      while true; do
      if oc wait --for=condition=complete --timeout=0 job/cphtestp -n ${SRC_ENV} 2>/dev/null; then
      if oc wait --for=condition=failed --timeout=0 job/cphtestp -n ${SRC_ENV} 2>/dev/null; then
      sleep 3
    if [[ $job_result -eq 1 ]]; then
        echo "Job failed!"
        exit 1
    echo "Job succeeded"
    oc logs job/cphtestp -n ${SRC_ENV}
      echo "Skipping tests ... This is not a QueueManager"

The details for the above task are as follows:

  • cphtestp is the name of the step in the tekton task.
  • The base image being used for this task is
  • This step initially validates if the workload is QueueManager. If so, this task will be run. Otherwise, this task will be skipped.
  • Later this will grab queue manager name, queue name, queue manager host and queue manager port.
  • Then the test job in the staging environment will be removed if it is already existing.
  • Currently, the job definition for cphtestp tool only works for a non-secure queue manager. Based on the security definition of the queue manager params.qm-security, existing job will be created in the staging environment if the security is disabled. Otherwise, these tests will be skipped.

Work is under progress for enabling the tests on secure queue manager as well. This section will soon be updated once we have it in place.

  • This job is responsible for performing the load tests on the queue manager residing in the staging environment.
  • Once the job is created, it keeps waiting till its completion.
  • If the job is successful, we can see the test results in the logs. If the job fails, pipeline will halt.

Test Results

For your reference, when the above task runs as part of our performance test pipeline, the results in the logs are as follows.

# CSV Results
# TLS Cipher:
# Thu Jan 27 16:03:34 UTC 2022
# Persistence, Msg Size, Threads, Rate (RT/s), Client CPU, IO Read (MB/s), IO Write (MB/s), Net Recv (Gb/s), Net Send (Gb/s), QM CPU
cph testing finished--------------------

Performance Testing for MQ Sample application

Apache Jmeter

Jmeter is an open source java based tool. It helps us to measure the performance of the application. We can basically analyze the load functional behavior of the application.

  • Jmeter simulates the load, for instance it simulates a group of users and keep sending requests to the target server.
  • We can define a test plan script. All the necessary configurations will be provided here.
  • Once we run the Jmeter tests, results will be available.

Sample test plan

Our sample app test plan can be accessed here.

If you have a look at the test plan, you will see different configuration elements.

For example, you can see __P(route, in one of the ThreadGroup elements. The default route here is, but the route url is parameterized. We will be passing this information as part of pipeline.

Running jmeter tests

Below is the jmeter command we used as part of our tests.

    -n -t $(params.test-plan) \
    -Jroute=$route \
    -l /source/report.jtl

Here we are providing the test plan script along with the route to the Jmeter. Once the command is executed successfully, it loads the results into /source/report.jtl file.

Task definition for Jmeter performance tests

In the performance test task definition, the following step defines the performance tests for the sample Spring Boot application.

- name: jmeter-it
  workingDir: $(params.source-dir)
  script: |
    #!/usr/bin/env bash
    set -ex
    if [ ! -f /source/route.txt ]; then
      echo "This is a QueueManager ... Skipping this step"
      exit 0
    route=$(cat /source/route.txt)
    echo $route
    if [ -e $test_plan ]; then
      # note: the export does not like the - sign
      ${JMETER_BIN}/ \
        -n -t $(params.test-plan) \
        -Jroute=$route \
        -l /source/report.jtl
      cat /source/report.jtl
      echo "checking for the existence of errors .."
      LINES=$(grep false /source/report.jtl | wc -l)
      TOTAL_LINES=$(cat /source/report.jtl | wc -l)
      if [ $LINES -eq 0 ] ; then
        echo "no errors in jmeter test -> ready to promote the code"
        exit 0
        echo "oops, errors were found."
        exit 1
      echo "Skipping tests ... Jmeter test plan does not exist"

The details for the above task are as follows:

  • jmeter-it is the name of the step in the tekton task.
  • The base image being used for this task is
  • This task initially validates if the workload is QueueManager. If so, this task will be skipped.
  • Later this task will grab the route of the sample spring application.
  • The test plan (jmx file) will be passes as a param params.test-plan.
  • It also validates if the test plan file is existing. If it does not exist, this tests will be skipped.
  • Jmeter uses this test plan to validate the load on the sample spring application and saves the results to /source/report.jtl.
  • Based on that report, this task will check for any existence of errors.
  • If errors exist, pipeline will halt. Otherwise, it moves to the next step.

Test Results

For your reference, when the above task runs as part of our performance test pipeline, the logs are as follows.


+ '[' '!' -f /source/route.txt ']'
++ cat /source/route.txt
+ echo
+ test_plan=jmeter/mq-spring-app.jmx
+ ls
+ '[' -e jmeter/mq-spring-app.jmx ']'
+ /opt/apache-jmeter-5.4.1/bin/ -n -t jmeter/mq-spring-app.jmx -l /source/report.jtl
Jan 18, 2022 12:02:22 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Creating summariser <summary>
Created the tree successfully using jmeter/mq-spring-app.jmx
Starting standalone test @ Tue Jan 18 12:02:22 GMT 2022 (1642507342857)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
Warning: Nashorn engine is planned to be removed from a future JDK release
summary +   1488 in 00:00:07 =  217.2/s Avg:     9 Min:     2 Max:   480 Err:     0 (0.00%) Active: 3 Started: 3 Finished: 0
Generate Summary Results +    321 in 00:00:07 =   46.8/s Avg:    20 Min:    10 Max:   480 Err:     0 (0.00%) Active: 3 Started: 3 Finished: 0
summary +   8599 in 00:00:28 =  304.8/s Avg:     8 Min:     2 Max:   161 Err:     0 (0.00%) Active: 0 Started: 3 Finished: 3
summary =  10087 in 00:00:35 =  287.7/s Avg:     8 Min:     2 Max:   480 Err:     0 (0.00%)
Generate Summary Results +   1355 in 00:00:28 =   48.0/s Avg:    17 Min:     7 Max:   139 Err:     0 (0.00%) Active: 0 Started: 3 Finished: 3
Generate Summary Results =   1676 in 00:00:35 =   47.8/s Avg:    17 Min:     7 Max:   480 Err:     0 (0.00%)
Tidying up ...    @ Tue Jan 18 12:02:58 GMT 2022 (1642507378217)
... end of run
+ cat /source/report.jtl
1642507343336,480,POST send-json,200,,Thread Group 1-1,text,true,,400,378,1,2,,475,0,265
1642507343336,480,GET send-hello-world,200,,Thread Group 2-1,text,true,,355,281,1,2,,479,0,266
1642507343818,11,GET send-hello-world,200,,Thread Group 2-1,text,true,,355,281,1,2,,11,0,0
1642507343830,10,GET send-hello-world,200,,Thread Group 2-1,text,true,,355,281,1,2,,10,0,0
1642507343818,25,POST send-json,200,,Thread Group 1-1,text,true,,400,378,1,2,,25,0,14
1642507378177,4,GET receive,200,,Thread Group 3-1,text,true,,361,269,1,1,,4,0,0
1642507378181,5,GET receive,200,,Thread Group 3-1,text,true,,361,269,1,1,,5,0,0
1642507378186,4,GET receive,200,,Thread Group 3-1,text,true,,361,269,1,1,,4,0,0
1642507378190,6,GET receive,200,,Thread Group 3-1,text,true,,361,269,1,1,,6,0,0
1642507378196,5,GET receive,200,,Thread Group 3-1,text,true,,411,269,1,1,,5,0,0
1642507378201,6,GET receive,200,,Thread Group 3-1,text,true,,361,269,1,1,,6,0,0
1642507378207,4,GET receive,200,,Thread Group 3-1,text,true,,361,269,1,1,,4,0,0
1642507378211,4,GET receive,200,,Thread Group 3-1,text,true,,411,269,1,1,,4,0,0
+ echo 'checking for the existence of errors ..'
checking for the existence of errors ..
++ grep false /source/report.jtl
++ wc -l
++ cat /source/report.jtl
++ wc -l
+ '[' 0 -eq 0 ']'
no errors in jmeter test -> ready to promote the code
+ echo 'no errors in jmeter test -> ready to promote the code'
+ exit 0
Back to top