When we need to apply patches on Exadata Cloud Service (ExaCS) or Exadata Cloud at Customer (ExaCC), we have great facilities provided by cloud automation, with different methods like Web Console, Command Line and REST API. But, when something get wrong, the ways to resume a failed operation is more restricted, and we usually will need to use a Command Line provided by Cloud Tooling, like dbaascli.

Is important to notice that any Grid or DB Home patch operation on Exadata Cloud is always executed by dbaascli. When you use the Web Console, behind of scene, the cloud automation will initiate a dbaascli execution on target DB Node (virtual machine).

In this blog post I will demonstrate how we can use dbaascli to resume a failed GRID or DB Home patch operation in situations where you are able to identifiy the root cause of the problem. But is important to say that you don’t need to throbleshooting by your self, you can always open a SR to Oracle Cloud Support (I recommend). But in the end of day, after identifying and fix the issue that caused the patch operation to fail, the way to resume this operation will be the same.

Problem example

In this example, I was applying Database Home patching using OCI Console, and the opration failed with the below error (finded in the work request details):

Below the error presented in the Work request log:

TIP: The hostname listed in the log should be the node where we will start any throubleshooting and resume the failed operation.

[FATAL] [DBAAS-60022] Command '/u02/app/oracle/product/19.0.0.0/dbhome_1/OPatch/datapatch -verbose' has failed on nodes [exa01db01].

Investigation example (not the focus of this post)

The goal of this post is not show how to throubleshoting the specific or any other error in ExaCC/ExaCC patching operation, but how we can resume the failed operation after fix the issue.

In this case, the issue was an error in Datapatch execution caused by an custom Event Trigger created by the customer to catch any DDL or DML errors in the database, which probably will always be triggered during an Datapatch session.

Looking at $ORACLE_BASE/cfgtoollogs:

cd /u02/app/oracle/cfgtoollogs/sqlpatch
cd sqlpatch_350814_2025_07_06_01_59_40
vi sqlpatch_invocation.log

        'error_message' => [
          'ORA-04088: error during execution of trigger \'OWNER_NAME.TRIGGER_NAME\'',
          'ORA-00604: error occurred at recursive SQL level 1',
          'ORA-06502: PL/SQL: numeric or value error: character string buffer too small',
          'ORA-06512: at line 12',
          'ORA-00955: name is already used by an existing object'
        ],
        'error_stack' => 'ORA-04088|ORA-00604|ORA-06502|ORA-06512|ORA-00955',
        'line_num' => 282534,
        'script' => 'rdbms/admin/catgwmcat.sql'

How to Resume a Failed Patch Operation

The procedure is the same for Grid or DB Home patching, so I will show two examples.

When a Grid or DB Home operation is started by dbaascli, the cloud tooling write a suggested command to resume the current operation in the log file. We can inspect the latest log file updated by the cloud tooling and find the command to resume the operation.

GRID Example

Go to directory where OCI write the log for grid patching operations in ExaCC and ExaCS:

cd /var/opt/oracle/log/grid/patch/

Find the latest “dbaastools*.log” file and execute this grep command to find the resume command:

grep -A3 "Resume command" dbaastools_2025-07-05_08-30-48-PM_346830.log

The result should be similar the below:

******************************* Resume command *******************************
To resume this failed session, run the following command:
dbaascli grid patch --waitForCompletion false --targetVersion 19.27.0.0.0 --sessionID 49173 --resume
******************************************************************************

So you can copy and execute the dbaascli command as root user in the node that appears in the failed operation on OCI console.

DB Home example

Go to log directory where cloud tooling write logs for dbHome patch operations:

cd /var/opt/oracle/log/dbHome/patch

Search for the resume command in the log file:

grep -A3 "Resume command" dbaastools_2025-07-06_12-56-11-AM_286584.log

The result for DB Home is like this:

******************************* Resume command *******************************
To resume this failed session, run the following command:
dbaascli dbHome patch --oracleHomeName b1db1ffa-9a70-479b-8a45-1c2ac625f393 --waitForCompletion false --targetVersion 19.27.0.0.0 --sessionID 49187 --resume
******************************************************************************

Then you can copy and execute this command as root user on db node listed in the log error in OCI console.

When you execute this commando, the dbaascli will create a background job for run this operation:

[root@exa01db01 ~]# dbaascli dbHome patch --oracleHomeName b1db1ffa-9a70-479b-8a45-1c2ac625f393 --waitForCompletion false --targetVersion 19.27.0.0.0 --sessionID 49187 --resume
DBAAS CLI version 25.2.1.0.0
Executing command dbHome patch --oracleHomeName b1db1ffa-9a70-479b-8a45-1c2ac625f393 --waitForCompletion false --targetVersion 19.27.0.0.0 --sessionID 49187 --resume
Job id: 85dcd211-6e83-4b40-844d-be07070ca6f7
Session log: /var/opt/oracle/log/dbHome/patch/dbaastools_2025-07-06_02-39-49-AM_228330.log
pwd
Job accepted. Use "dbaascli job getStatus --jobID 85dcd211-6e83-4b40-844d-be07070ca6f7" to check the job status.

So we can get the JOB status:

[root@exa01db01 ~]# dbaascli job getStatus --jobID 85dcd211-6e83-4b40-844d-be07070ca6f7
DBAAS CLI version 25.2.1.0.0
Executing command job getStatus --jobID 85dcd211-6e83-4b40-844d-be07070ca6f7
{
  "jobId" : "85dcd211-6e83-4b40-844d-be07070ca6f7",
  "dbaascliParameters" : "dbHome patch --resume --oracleHomeName b1db1ffa-9a70-479b-8a45-1c2ac625f393 --waitForCompletion false --sessionID 49187 --targetVersion 19.27.0.0.0",
  "status" : "Running",
  "message" : "dbHome patch job: Running",
  "logFile" : "/var/opt/oracle/log/dbHome/patch/dbaastools_2025-07-06_02-39-49-AM_228330.log",
  "createTimestamp" : 1751780393804,
  "updatedTime" : 1751780424261,
  "description" : "Service job report for operation dbHome patch",
  "appMessages" : {
    "messages" : [ ],
    "localeMessages" : [ ],
    "errorAction" : null
  },
  "resourceList" : [ ],
  "sessionID" : 49189,
  "jobSpecificDetailsJson" : null,
  "pct_complete" : "100"
}

But the best way to monitor the job progress is using a tail command in the previous log file:

tail -100f /var/opt/oracle/log/dbHome/patch/dbaastools_2025-07-06_02-39-49-AM_228330.log

If the operation finish successfully, then after 10 to 20 minutes, the OCI console should recognize that the patch was completed.