Introduction

In my last post GIRU 19.30 Patch Fails with “modprobe: ERROR: could not insert ‘oracleoks’: Operation not permitted” Followed by”CLSRSC-400: A system reboot is required to continue installing” I showed a scenario where the Grid patch has failed due expired keys in the Secure Boot blocking the ACFS/ADVM modules in the kernel. In that post I shared a workaround if you don’t use ACFS and wanted to skip it during the patch.

However, if you need to use ACFS, you should update the Oracle Public Keys in the Secure boot before patch the Grid Infrastructure with RU 19.29 or higher. In this blog post I’m showing a demo in how to achieve this, based on the ACFS / AFD Secure Boot Configuration (KB146591)

Checking the Existing Keys

Checking if secure boot is enabled:

[root@lab05 ~]#  mokutil --sb-state
SecureBoot enabled

Listing the existing Oracle keys configured in the secure boot:

[root@lab05 ~]# keyctl show %:.secondary_trusted_keys | grep Oracle
  56429526 --alswrv      0     0       \_ asymmetric: Oracle CA Server: 7db9dc26003a76978974b2cb9561ccc1f205cdcb
  69239516 --alswrv      0     0       \_ asymmetric: Oracle IMA signing CA: a2f28976a05984028f7d1a4904ae14e8e468e551
 867366577 --alswrv      0     0       \_ asymmetric: Oracle America, Inc.: Ksplice Kernel Module Signing Key: 09010ebef5545fa7c54b626ef518e077b5b1ee4c
  20082337 --alswrv      0     0       \_ asymmetric: Oracle Linux Kernel Module Signing Key: 2bb352412969a3653f0eb6021763408ebb9bb5ab

Let’s create a folder to export these keys and check what is the end date for each one:

[root@lab05 ~]# mkdir mok
[root@lab05 ~]# cd mok/

Now we can export the existing keys:

[root@lab05 mok]# mokutil --export
[root@lab05 mok]# 
[root@lab05 mok]# ls -lrt
total 12
-rw-r--r-- 1 root root 1927 May 16 16:21 MOK-0003.der
-rw-r--r-- 1 root root 1927 May 16 16:21 MOK-0002.der
-rw-r--r-- 1 root root 1927 May 16 16:21 MOK-0001.der

We can use the openssl utility to get the attributes from .der files.
In the following command I’m looking for the -subject and -enddate attribute, the last one is the key expiration date:

[root@lab05 mok]# openssl x509 -inform DER -in MOK-0003.der -noout -subject -enddate
subject= /jurisdictionC=US/jurisdictionST=Delaware/businessCategory=Private Organization/serialNumber=2101822/C=US/ST=California/L=Redwood City/O=Oracle America, Inc./CN=Oracle America, Inc.
notAfter=Mar 15 23:59:59 2025 GMT

Note that the end date is Mar 15 23:59:59 2025 GMT.
To get this information from all exported keys, we can use a FOR loop:

for f in MOK*.der; do 
    echo "--- $f ---"
    openssl x509 -inform DER -in "$f" -noout -subject -enddate
done

Example:

[root@lab05 mok]# for f in MOK*.der; do 
>     echo "--- $f ---"
>     openssl x509 -inform DER -in "$f" -noout -subject -enddate
> done
--- MOK-0001.der ---
subject= /jurisdictionC=US/jurisdictionST=Delaware/businessCategory=Private Organization/serialNumber=2101822/C=US/ST=California/L=Redwood City/O=Oracle America, Inc./CN=Oracle America, Inc.
notAfter=Apr  2 23:59:59 2025 GMT
--- MOK-0002.der ---
subject= /jurisdictionC=US/jurisdictionST=Delaware/businessCategory=Private Organization/serialNumber=2101822/C=US/ST=California/L=Redwood City/O=Oracle America, Inc./CN=Oracle America, Inc.
notAfter=Apr  2 23:59:59 2025 GMT
--- MOK-0003.der ---
subject= /jurisdictionC=US/jurisdictionST=Delaware/businessCategory=Private Organization/serialNumber=2101822/C=US/ST=California/L=Redwood City/O=Oracle America, Inc./CN=Oracle America, Inc.
notAfter=Mar 15 23:59:59 2025 GMT

Getting the New Keys

Oracle provides the new keys trough the note ACFS / AFD Secure Boot Configuration (KB146591)
At the end of the page, you should fine these attachments:

Click in the file name and copy the downloaded zip file to the database server.

Importing New Keys

Now let’s unzip the keys downloaded from My Oracle Support

[root@lab05 mok]# unzip /tmp/KB146591_oracle-linux-driver-signing-key1.zip
Archive:  /tmp/KB146591_oracle-linux-driver-signing-key1.zip
  inflating: oracle-linux-driver-signing-key1.cer 

You should notice a new cerfificate in the current folder:

[root@lab05 mok]# ls -lrt
total 16
-rw-r--r-- 1 root root 1546 Feb  5 15:52 oracle-linux-driver-signing-key1.cer
-rw-r--r-- 1 root root 1927 May 16 16:21 MOK-0003.der
-rw-r--r-- 1 root root 1927 May 16 16:21 MOK-0002.der
-rw-r--r-- 1 root root 1927 May 16 16:21 MOK-0001.der

We can now use mokutil to import the .cer file.
The mokutil will ask to create a password.
We need take note of the password, this password will be used later during the ENROL proccess in the next boot.

I used just ‘oracle’ as the password here:

[root@lab05 mok]# mokutil --import oracle-linux-driver-signing-key1.cer 
input password: 
input password again: 
[root@lab05 mok]#

Rebooting the OS & Server Management Console

The next step is to reboot the server.
But before restart, we need to make sure we are connected to the server console, in order to interact with the server during the boot process.

Be aware that during the boot, you have only 10 seconds to confirm if you want to ENROL the new key. If you don’t provide a answer in 10 seconds, the OS will be automatically started. That is why it is important to be connected in the server console before rebooting the OS.

Oracle Server with ILOM

If you you are working in a Oracle server like ODA or Exadata, you should be connected to the ILOM before proceeding with the reboot.

You don’t need access the web console, it is possible to complete the task connected to the command line.
Example:

ssh root@server-ilom
-> start /SP/console

Other Server Vendors

If you are working with others servers based on different IPMI solutions, like iDRAC, ILO and so on, it work in same way, but you need to check how to use each of these solutions.

VM Console

If you you are working with a VM, you can easily use the VM console for it.
In my test case, I’m using a VM on OLVM:

Please note you can do anything while connected to the server using SSH, but you should have a secondory connection to the ILOM or whetever console solution your server require in order to interact with the boot process,

Now since I’m ready to monitor the reboot process in the VM console, I can safely initiate the reboot:

[root@lab05 mok]#
[root@lab05 mok]# reboot

When the server is rebooted, we are presented to the Shim UEFI key management with a timeout of 10 seconds:

Select Enrol MOK

So select Continue and press ENTER

Select YES to confirm:

Then you are asked to provide the password you used in the import command:

Now we can select Reboot

Checking the New Key

After the server is rebooted, we can export the keys again and check we have a new one:

[root@lab05 ~]# cd mok/
[root@lab05 mok]# 
[root@lab05 mok]# ls -lrt
total 16
-rw-r--r-- 1 root root 1546 Feb  5 15:52 oracle-linux-driver-signing-key1.cer
-rw-r--r-- 1 root root 1927 May 16 16:21 MOK-0003.der
-rw-r--r-- 1 root root 1927 May 16 16:21 MOK-0002.der
-rw-r--r-- 1 root root 1927 May 16 16:21 MOK-0001.der
[root@lab05 mok]# 
[root@lab05 mok]# mokutil --export
[root@lab05 mok]# 
[root@lab05 mok]# ls -lrt
total 20
-rw-r--r-- 1 root root 1546 Feb  5 15:52 oracle-linux-driver-signing-key1.cer
-rw-r--r-- 1 root root 1927 May 16 16:47 MOK-0001.der
-rw-r--r-- 1 root root 1546 May 16 16:47 MOK-0004.der
-rw-r--r-- 1 root root 1927 May 16 16:47 MOK-0003.der
-rw-r--r-- 1 root root 1927 May 16 16:47 MOK-0002.der
[root@lab05 mok]# 

And if we check the enddate for the key MOK-0004.der, the expiration date points to April 2036:

[root@lab05 mok]# 
[root@lab05 mok]# openssl x509 -inform DER -in MOK-0004.der -noout -subject -enddate
subject= /CN=Oracle Linux Driver Signing (key 1)/C=us/L=Redwood City/ST=California
notAfter=Apr 12 21:10:29 2036 GMT

At this point we are good to follow with the Grid patching without issues with ACFS modules.

Conclusion

Please note the recommended solution from Oracle to their appliances like Exadata is to patch the image to latest releases, which will include the new keys as part of OS update. But if the Exadata patch isn’t feasible for now, you can safely follow this manual process as mentioned by Oracle in the ACFS / AFD Secure Boot Configuration (KB146591) note.

While this example was based on a VM, I had performed a similar work on Exadata in the past for different purpose, and the process is exactly the same. The only trick you need to pay attention is to connect in the ILOM before initiate the reboot, and do not miss the 10 seconds of timeout during the next boot.