Introduction

This blog post describes an issue I encountered while applying the latest GIRU 19.30 to two Exadata X9M systems running Oracle Linux 7.9, using the Out-Of-Place approach for Grid Infrastructure and Oracle Database homes. In summary, the issue was related to Secure Boot being enabled with obsolete public keys, which were blocking the new ACFS and ADVM modules from being loaded into the Kernel.

For my specific use case, the ACFS is not used in that cluster, so my workaround was just edit the shell script responsible for install the ACFS modules to skip the installation, and always return the exit code 0 (success), so the rootcrs.sh -postpatch could complete with no errors.

Problem

So I initiated the switchGiHome with the Grid owner (in this case, the oracle user itself):

[oracle@exadb01 ~]$  $ORACLE_HOME/gridSetup.sh -silent -switchGridHome
Launching Oracle Grid Infrastructure Setup Wizard...
As a root user, execute the following script(s):
1. /u01/app/19.0.0.0/grid_1/root.sh
Execute /u01/app/19.0.0.0/grid_1/root.sh on the following nodes: 
[exadb01, exadb04, exadb03, exadb02, exadb05]
Run the scripts on the local node first. After successful completion, run the scripts in sequence on all other nodes.
You can find the log of this install session at:
/u01/app/oraInventory/logs/UpdateNodeList2026-04-19_03-33-23PM.log
You can find the log of this install session at:
/u01/app/oraInventory/logs/UpdateNodeList2026-04-19_03-33-23PM.log
Successfully Setup Software.

And then executed the root.sh in the local node first:

[oracle@exadb01 ~]$  sudo su - 
Last login: Sun Apr 19 15:17:28 CDT 2026
[root@exadb01 ~]# 
[root@exadb01 ~]# nohup /u01/app/19.0.0.0/grid_1/root.sh  > /tmp/switchGiHome.out 2>&1 &

So I started to monitoring the log file and I got the unexpected behavior:

[root@exadb01 ~]# cat /tmp/switchGiHome.out
nohup: ignoring input
Check /u01/app/19.0.0.0/grid_1/install/root_exadb01_2026-04-19_15-37-44-498178418.log for the output of root script
[root@exadb01 ~]# tail -100f /u01/app/19.0.0.0/grid_1/install/root_exadb01_2026-04-19_15-37-44-498178418.log
Performing root user operation.
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/19.0.0.0/grid_1
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
LD_LIBRARY_PATH='/u01/app/19.0.0.0/grid/lib:/u01/app/19.0.0.0/grid_1/lib:'
Using configuration parameter file: /u01/app/19.0.0.0/grid_1/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/exadb01/crsconfig/rootcrs_exadb01_2026-04-19_03-37-52PM.log
Using configuration parameter file: /u01/app/19.0.0.0/grid_1/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/exadb01/crsconfig/crs_prepatch_apply_oop_exadb01_2026-04-19_03-37-52PM.log
2026/04/19 15:38:02 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.
Using configuration parameter file: /u01/app/19.0.0.0/grid_1/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/exadb01/crsconfig/crs_postpatch_apply_oop_exadb01_2026-04-19_03-38-02PM.log
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [689998692].
2026/04/19 15:39:08 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2026/04/19 15:39:27 CLSRSC-400: A system reboot is required to continue installing.
Died at /u01/app/19.0.0.0/grid_1/crs/install/crspatch.pm line 1114.
The command '/u01/app/19.0.0.0/grid_1/perl/bin/perl -I/u01/app/19.0.0.0/grid_1/perl/lib -I/u01/app/19.0.0.0/grid_1/crs/install /u01/app/19.0.0.0/grid_1/crs/install/rootcrs.pl  -dstcrshome /u01/app/19.0.0.0/grid_1 -postpatch' execution failed
^C

The main error above is:

2026/04/19 15:39:27 CLSRSC-400: A system reboot is required to continue installing.

Investigation

So I opened the crs_postpatch_apply_oop_exadb01_2026-04-19_03-38-02PM.log log file and found the actual error:

>  ACFS-9504: Copying file '/u01/app/19.0.0.0/grid_1/lib/libacfs19.so' to the path '/opt/oracle/extapi/64/acfs/orcl/1/'
>  No advmtunables file found. Creating new one.
>  Setting asm_acfsr_mode=0
>  ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf
>  ACFS-9308: Loading installed ADVM/ACFS drivers.
>  ACFS-9298: Installing SELinux policy for ACFS.
>  ACFS-9297: Creating ACFS SELinux policy file /usr/share/oracleacfs/acfs.cil.
>  Creating routine 'INSTALL_SELINUX_POLICY'.
>  ACFS-9321: Creating udev for ADVM/ACFS.
>  ACFS-9323: Creating module dependencies - this may take some time.
>  ACFS-9176: Entering 'depmod install kver 4.14.35-2047.528.2.4.el7uek.x86_64'
>  Depmod running in the background.
>  ACFS-9177: Return from 'depmod install'
>  ACFS-9176: Entering 'ld usm drvs'
>  ACFS-9154: Loading 'oracleoks.ko' driver.
>  Modprobe output 'insmod: ERROR: could not insert module /lib/modules/4.14.35-2047.528.2.4.el7uek.x86_64/extra/usm/oracleoks.ko: Operation not permitted
>  insmod: ERROR: could not insert module /lib/modules/4.14.35-2047.528.2.4.el7uek.x86_64/extra/usm/oracleadvm.ko: Operation not permitted
>  insmod: ERROR: could not insert module /lib/modules/4.14.35-2047.528.2.4.el7uek.x86_64/extra/usm/oracleacfs.ko: Operation not permitted
>  modprobe: ERROR: Error running install command for oracleoks
>  modprobe: ERROR: could not insert 'oracleoks': Operation not permitted
>  '
>  ACFS-9177: Return from 'mpr'
>  ACFS-9109: oracleoks.ko driver failed to load.
>  ACFS-9178: Return code = USM_FAIL
>  ACFS-9177: Return from 'ld usm drvs'
>  ACFS-9428: Failed to load ADVM/ACFS drivers. A system reboot is recommended.
>  ACFS-9310: ADVM/ACFS installation failed.
>  ACFS-9178: Return code = USM_REBOOT_RECOMMENDED
>  ACFS-9177: Return from 'install'
>  ACFS-9176: Entering 'acroot ex'
>  ACFS-9178: Return code = 3
>  ACFS-9177: Return from 'acroot ex'

The main error here is:

modprobe: ERROR: could not insert ‘oracleoks’: Operation not permitted

When you have some permission error to load a kernel module, a common reason is the Secure Boot blocking it.

So I checked if secure boot was enabled:

[root@exadb01]# mokutil --sb-state
SecureBoot enabled

So now I know the secure boot is blocking the installation of new kernel modules. At this point in the troubleshooting, I was not aware about the root cause, I figured out about this later.

Since in this Exadata the ACFS is not used, we decided to skip the ACFS modules installation, and it is exactly what I want to share in this blog post.

Workaround if ACFS is not used

When you execute the root.sh for the switchGiHome operation, the rootcrs.sh -postpatch is executed behind of scene, and the -postpatch operations executes another script called acfstoolsdriver.sh to install the ACFS modules.

The correct location for this script is <TARGET_GRID_HOME>/lib/acfstoolsdriver.sh

I used the TARGET_GRID_HOME here, because if we are doing an Out-Of-Place patching, we need to look for this script in the target new Grid Home, that one we are executing the switchGiHome. In my example, the exact location is /u01/app/19.0.0.0/grid_1/lib/acfstoolsdriver.sh

What we need to do is edit this script, and comment the line with “exec ${RUNTHIS} $@“, and also the line with “exit -1“.

For example, last lines in the original script are:

# Now run command with all arguments!
exec ${RUNTHIS} $@
# Should never get here.
exit -1

And to properly skip this script, that actually install the ACFS, the last lines should be like this:

# Now run command with all arguments!
#exec ${RUNTHIS} $@
# Should never get here.
#exit -1
exit 0

Note I have commented the two original lines, and added a explicit “exit 0“ to make sure the main script will interpret this one as successfully executed.

Resume the failed root.sh

To resume the failed root.sh execution, we just need execute it again:

[root@exadb01 ~]#  nohup /u01/app/19.0.0.0/grid_1/root.sh  > /tmp/switchGiHome.out 2>&1 &
[1] 171849
[root@exadb01 ~]# 
[root@exadb01 ~]# cat /tmp/switchGiHome.out
nohup: ignoring input
Check /u01/app/19.0.0.0/grid_1/install/root_exadb01_2026-04-19_16-20-52-126910188.log for the output of root script
[root@exadb01 ~]# 

And now checking the log file in real time, and the -postpatch operation was completed:

[root@exadb01 ~]# tail -100f /u01/app/19.0.0.0/grid_1/install/root_exadb01_2026-04-19_16-20-52-126910188.log
Performing root user operation.
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/19.0.0.0/grid_1
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
LD_LIBRARY_PATH='/u01/app/19.0.0.0/grid_1/lib:/u01/app/19.0.0.0/grid_1/lib:'
Using configuration parameter file: /u01/app/19.0.0.0/grid_1/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/exadb01/crsconfig/rootcrs_exadb01_2026-04-19_04-20-52PM.log
Using configuration parameter file: /u01/app/19.0.0.0/grid_1/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/exadb01/crsconfig/crs_prepatch_apply_oop_exadb01_2026-04-19_04-20-52PM.log
2026/04/19 16:20:54 CLSRSC-901: The destination GI home specified for out-of-place patching is the same as the configured GI home.
Using configuration parameter file: /u01/app/19.0.0.0/grid_1/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/exadb01/crsconfig/crs_postpatch_apply_oop_exadb01_2026-04-19_04-20-54PM.log
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [689998692].
2026/04/19 16:22:26 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2026/04/19 16:22:26 CLSRSC-4005: Failed to patch Oracle Trace File Analyzer (TFA) Collector. Grid Infrastructure operations will continue.
2026/04/19 16:22:29 CLSRSC-672: Post-patch steps for patching GI home successfully completed.

The remote nodes

Since I have modified this script only in the local node, I need to copy it to all cluster nodes before continue with the rolling patching. I mean, before to execute the root.sh in the remote nodes.

Since I was working in a Exadata, I can use the dcli utility for it:

cd /u01/app/19.0.0.0/grid_1/lib/
dcli -g ~/dbs_group -l oracle -f acfstoolsdriver.sh -d /u01/app/19.0.0.0/grid_1/lib/

If you are executing it in a non-Exadata, you can copy the script with scp. Or if you like to work hard, you can just connect in all the nodes and edit the script manuallay.

So I after update the script in all nodes, I just followed with the root.sh and it has successfully completed with no errors.

What is Secure Boot ?

Here are is just a briefly introduction to understand the root cause analysis:

The Secure Boot it is a feature that allows only known modules to be loaded in the kernel. It is, only the modules from the Linux kernel itself, and third party modules that are assigned with a private key, and the corresponding public key is imported in the server BIOS as an “trusted” vendor.

The ACFS and ADVM are kernel modules installed and loaded during the Grid Infrastructure installation or patching. From the OS perspective, they are third party modules, where Oracle is the vendor (even if the OS is an Oracle Linux). So in order to those modules work properly in a server with Secure Boot enabled, the valid Oracle Public Keys needs to be present in the Secure Boot.

Root cause analysis

After the patch has been completed, I started to review the logs and search for any known issue in MOS or other blog posts in the internet, and I found that we can face this issue starting with GIRU 19.29 being applied in Linux servers running old versions of Linux kernel, that is installed with old and expired Oracle Public Keys used by the Secure Boot.

Oracle has been using a key generated in 2015 with expiration date in 2025 to sign the ACFS kernel modules included 19c release updates until RU 19.28. Starting with GI RU 19.29, a new key with expiration date to 2036 is used to sign the modules. If the database server has Secure Boot enabled, the database server need to have these new keys properly imported in the Secure Boot in order to install new versions of ACFS modules.

Recommended Solution if ACFS is Required

The recommended solution for this issue it is download the new Oracle Public Keys from the My Oracle Support, import that keys using the mokutil utility in the operating system, reboot the server and enroll the new keys in the Secure Boot. I will share an example for this in another blog post.

If you disable the Secure Boot in the database server, the ACFS modules can be installed with no errors. However, if something it is enabled, usually it is because should be enabled. And in this case, Secure Boot is a security feature and can be required by your company security policies.

References

You can find the Oracle reference here:

ACFS/AFD Driver Install Fails Post-Applying 19.29 GIRU with Secure Boot Enabled” (KB829120)

ACFS / AFD Secure Boot Configuration (KB146591)