Neste post apresento um cenário de erro que pode ocorrer ao tentar criar um Database no ODA via ODACLI utilizando o mesmo nome de um Database que tenha sido excluído anteriormente. Conforme demonstração abaixo, durante o processo de exclusão de um Database, o ODACLI não apresenta nenhum erro, mas deixa de limpar uma configuração importante no ASM, causando erro na próxima tentativa de criar um Database com o mesmo nome.
Removendo o database BIXZ2 com ODACLI:
# odacli delete-database --dbName bixz2
{
"jobId" : "17e96217-c1c9-4a43-abfa-96a13e74a17a",
"status" : "Running",
"message" : null,
"reports" : [ ],
"createTimestamp" : "August 13, 2021 10:19:04 AM BRT",
"resourceList" : [ ],
"description" : "Database service deletion with db name: bixz2 with id : 9f23eb31-2860-4c0a-b32c-cc379b9a1c42",
"updatedTime" : "August 13, 2021 10:19:07 AM BRT"
}
O job completou com sucesso:
# odacli describe-job -i 17e96217-c1c9-4a43-abfa-96a13e74a17a
Job details
----------------------------------------------------------------
ID: 17e96217-c1c9-4a43-abfa-96a13e74a17a
Description: Database service deletion with db name: bixz2 with id : 9f23eb31-2860-4c0a-b32c-cc379b9a1c42
Status: Success
Created: August 13, 2021 10:19:04 AM BRT
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate db 9f23eb31-2860-4c0a-b32c-cc379b9a1c42 for deletion August 13, 2021 10:19:07 AM BRT August 13, 2021 10:19:07 AM BRT Success
Database Deletion August 13, 2021 10:19:08 AM BRT August 13, 2021 10:20:28 AM BRT Success
Unregister Db From Cluster August 13, 2021 10:20:28 AM BRT August 13, 2021 10:20:28 AM BRT Success
Kill Pmon Process August 13, 2021 10:20:28 AM BRT August 13, 2021 10:20:28 AM BRT Success
Database Files Deletion August 13, 2021 10:20:28 AM BRT August 13, 2021 10:20:28 AM BRT Success
Deleting Volume August 13, 2021 10:20:40 AM BRT August 13, 2021 10:20:47 AM BRT Success
Deleting Volume August 13, 2021 10:20:59 AM BRT August 13, 2021 10:21:07 AM BRT Success
Delete File Groups of Database bixz2 August 13, 2021 10:21:08 AM BRT August 13, 2021 10:21:08 AM BRT Success
Criando um novo database com o mesmo nome:
# odacli create-database --dbname bixz2 --dbhomeid 68861293-e81b-4ca1-9490-ec26f4b7a5b7 --dbstorage ACFS --characterset WE8ISO8859P1
Password for SYS,SYSTEM and PDB Admin:
Job details
----------------------------------------------------------------
ID: cf0797d4-beba-458e-9f1c-fbc3120ffb44
Description: Database service creation with db name: bixz2
Status: Created
Created: August 13, 2021 10:39:22 AM BRT
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
E o job apresenta falha na criação do DB Storage:
# odacli describe-job -i cf0797d4-beba-458e-9f1c-fbc3120ffb44
Job details
----------------------------------------------------------------
ID: cf0797d4-beba-458e-9f1c-fbc3120ffb44
Description: Database service creation with db name: bixz2
Status: Failure
Created: August 13, 2021 10:39:22 AM BRT
Message: DCS-10001:Internal error encountered: Fail to run command Failed to create FileGroupDATBIXZ2 on DiskGroupDATA.
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Database Service creation August 13, 2021 10:39:30 AM BRT August 13, 2021 10:39:33 AM BRT Failure
Database Service creation August 13, 2021 10:39:30 AM BRT August 13, 2021 10:39:33 AM BRT Failure
Setting up ssh equivalance August 13, 2021 10:39:31 AM BRT August 13, 2021 10:39:31 AM BRT Success
Create DATA FileGroup DATBIXZ2 August 13, 2021 10:39:32 AM BRT August 13, 2021 10:39:33 AM BRT Failure
Database Service creation August 13, 2021 10:39:32 AM BRT August 13, 2021 10:39:33 AM BRT Failure
Creating FileGroup August 13, 2021 10:39:32 AM BRT August 13, 2021 10:39:33 AM BRT Failure
O processo falhou porque os File Group já existiam no ASM.
Tentativa de listar os DB Storages apresenta erro:
# odacli list-dbstorages
DCS-10001:Internal error encountered: For input string: "acfsutil info fs: ACFS-03037: not an ACFS file system".
Investigando o arquivo de log do DCS Agent:
# cd /opt/oracle/dcs/log
# grep -B5 -A5 ACFS-03037 dcs-agent.log
fs,
-o,
freespace,
/u02/app/oracle/oradata/bixz2]'
2021-08-13 10:40:08,267 DEBUG [Thread-16286] [] c.o.d.c.u.CommonsUtils: Output :
acfsutil info fs: ACFS-03037: not an ACFS file system
2021-08-13 10:40:08,270 DEBUG [dw-18339 - GET /dbstorage] [] c.o.d.c.u.c.DCSProcessBuilder: Return code: 1
2021-08-13 10:40:08,270 DEBUG [dw-18339 - GET /dbstorage] [] c.o.d.c.u.c.CommandExecutor: Return code: 1
2021-08-13 10:40:08,270 DEBUG [dw-18339 - GET /dbstorage] [] c.o.d.c.u.CommonsUtils: Output :
acfsutil info fs: ACFS-03037: not an ACFS file system
2021-08-13 10:40:08,270 WARN [dw-18339 - GET /dbstorage] [] c.o.d.a.r.s.s.FileSystemUtils: Failed to get the fs size mounted at : /u02/app/oracle/oradata/bixz2
2021-08-13 10:40:08,274 DEBUG [dw-18339 - GET /dbstorage] [] c.o.d.a.r.s.s.FileSystemUtils: fs size -> acfsutil info fs: ACFS-03037: not an ACFS file system
2021-08-13 10:40:08,274 DEBUG [dw-18339 - GET /dbstorage] [] c.o.d.c.e.CatchRuntimeExpMapper: got RuntimeException java.lang.NumberFormatException: For input string: "acfsutil info fs: ACFS-03037: not an ACFS file system", msg=For input string: "acfsutil info fs: ACFS-03037: not an ACFS file system"
2021-08-13 10:40:08,274 ERROR [dw-18339 - GET /dbstorage] [] c.o.d.c.e.CatchRuntimeExpMapper: java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
java.lang.Long.parseLong(Long.java:589)
java.lang.Long.valueOf(Long.java:803)
com.oracle.dcs.agent.rpc.service.storage.FileSystemUtils.getAcfsFsFreeSize(FileSystemUtils.java:303)
com.oracle.dcs.agent.rpc.service.storage.VolumeUtils.dbAcfsFillInfo(VolumeUtils.java:218)
--
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
java.lang.Thread.run(Thread.java:748)
2021-08-13 10:40:08,274 DEBUG [dw-18339 - GET /dbstorage] [] c.o.d.c.e.DcsExceptionMapper: Got DcsException DcsException{errorHttpCode=InternalError, msg=Internal error encountered: For input string: "acfsutil info fs: ACFS-03037: not an ACFS file system"., msgId=10001, causedBy=java.lang.NumberFormatException: For input string: "acfsutil info fs: ACFS-03037: not an ACFS file system"}
Um indicativo do problema:
Failed to get the fs size mounted at : /u02/app/oracle/oradata/bixz2
O que o agente está tentando fazer é algo similar a isto:
$ acfsutil info fs /u02/app/oracle/oradata/bixz2
Mas retorna erro, porque esse File System não existe:
acfsutil info fs: ACFS-03037: not an ACFS file system
Consultando os File Groups no ASM com asmcmd:
$ asmcmd
ASMCMD> lsfg
File Group Disk Group Quota Group Used Quota MB Client Name Client Type
DEFAULT_FILEGROUP DATA GENERIC 0
DEFAULT_FILEGROUP RECO GENERIC 0
SRV-ODAX8-C DATA GENERIC 376 SRV-ODAX8-C CLUSTER
RDOPRDXZ RECO GENERIC 184380 RDOPRDXZ VOLUME
COMMONSTORE DATA GENERIC 15420 COMMONSTORE VOLUME
RECO RECO GENERIC 3944508 RECO VOLUME
DATPRDXZ DATA GENERIC 9123900 DATPRDXZ VOLUME
RDOABCD RECO GENERIC 116796 RDOABCD VOLUME
DATABCD DATA GENERIC 307260 DATABCD VOLUME
RDOTRNXZ RECO GENERIC 116796 RDOTRNXZ VOLUME
DATTRNXZ DATA GENERIC 8570940 DATTRNXZ VOLUME
RDOABCDHML RECO GENERIC 116796 RDOABCDHML VOLUME
DATABCDHML DATA GENERIC 307260 DATABCDHML VOLUME
RDOBIQLIK RECO GENERIC 153660 RDOBIQLIK VOLUME
DATBIQLIK DATA GENERIC 1331240 DATBIQLIK VOLUME
RDOHMLXZ RECO GENERIC 73788 RDOHMLXZ VOLUME
DATHMLXZ DATA GENERIC 5836840 DATHMLXZ VOLUME
RDOBIXZ2 RECO GENERIC 0 RDOBIXZ2 VOLUME
DATBIXZ2 DATA GENERIC 0 DATBIXZ2 VOLUME
Causa
O procedimento de deletar o DB Storage não remove os File Groups no ASM.
Efeito Colateral
Ao tentar criar um novo DB Storage com mesmo nome, o agente tentar criar um novo File Group com o mesmo nome do existente, apresentado erro. Nesse ponto, a informação sobre o novo File System já foi incluída no metadados do ODA, mesmo que o File System ainda não tenha sido criado. Quando rodamos o comando odacli list-dbstorages, ele vai tentar rodar um acfsutil fs info em todos os File System ACFS catalogados em seu metadados.
Solução (Workaround)
Já passei por esse problema por pelo menos umas três vezes e o procedimento a seguir foi efetivo na resolução desse impasse, além não oferecer riscos de danificar o metadados do ODA editando arquivos de configuração manualmente. A estratégia é criar os File Systems ACFS que o odacli está tentando consultar, remover o DB Storage de forma limpa com o odacli, remover o File System criado manualmente e remover os File Groups que causaram o problema.
Passos a serem realizados
1) Identificar quais File Systems o odacli está tentando consultar e apresentado erro
2) Criar os volumes ADVM
3) Formtar os volumes com ACFS
4) Montar o File System ACFS no node local (se for um ODA-HA)
5) Identificar o ID do DB Storage com status de falha
6) Remover o DB Storage com odacli
7) Desmontar os File Systems ACFS criados manualmente
8) Remover os volumes ADVM
9) Remover os File Groups que originaram o problema
Passo a Passo
1) Identificar quais File Systems o odacli está tentando consultar e apresentado erro
Investigue o arquivo de log /opt/oracle/dcs/log/dcs-agent.log e procure por erros relacionados a ACFS, como ” Failed to get the fs size mounted at “, conforme exemplificado anteriormente. Note que pode haver dois pontos de montagem, um para oradata e outro de redo.
2) Via asmcmd com usuário grid, crie 2 volumes ADVM nos seus respectivos diskgroups, os tamanhos não precisam ser iguais aos originais:
ASMCMD> volcreate -G DATA -s 100g datbixz2
ASMCMD> volcreate -G RECO -s 20g recobixz2
ASMCMD>
ASMCMD> volinfo -G DATA datbixz2
Diskgroup Name: DATA
Volume Name: DATBIXZ2
Volume Device: /dev/asm/datbixz2-153
State: ENABLED
Size (MB): 102400
Resize Unit (MB): 64
Redundancy: MIRROR
Stripe Columns: 8
Stripe Width (K): 4096
Usage:
Mountpath:
ASMCMD> volinfo -G RECO recobixz2
Diskgroup Name: RECO
Volume Name: RECOBIXZ2
Volume Device: /dev/asm/recobixz2-46
State: ENABLED
Size (MB): 20480
Resize Unit (MB): 64
Redundancy: HIGH
Stripe Columns: 8
Stripe Width (K): 4096
Usage:
Mountpath:
ASMCMD> exit
3) Formatando os volumes ADVM com ACFS:
$ mkfs.acfs /dev/asm/datbixz2-153
mkfs.acfs: version = 19.0.0.0.0
mkfs.acfs: on-disk version = 46.0
mkfs.acfs: volume = /dev/asm/datbixz2-153
mkfs.acfs: volume size = 107374182400 ( 100.00 GB )
mkfs.acfs: Format complete.
$ mkfs.acfs /dev/asm/recobixz2-46
mkfs.acfs: version = 19.0.0.0.0
mkfs.acfs: on-disk version = 46.0
mkfs.acfs: volume = /dev/asm/recobixz2-46
mkfs.acfs: volume size = 21474836480 ( 20.00 GB )
mkfs.acfs: Format complete.
4) Montando os volumes manualmente no node local:
# mount -t acfs /dev/asm/datbixz2-153 /u02/app/oracle/oradata/bixz2
# mount -t acfs /dev/asm/recobixz2-46 /u04/app/oracle/redo/bixz2
5) Listandos todos os DB Storages para identificar o ID do DB Storage com falha:
# odacli list-dbstorages
ID Type DBUnique Name DiskGroup Location Total Used Available Status
---------------------------------------- ------ -------------------- ---------- ------------------------------------------------------------ ---------- ---------- ---------- ----------
36ee228c-876c-4152-950f-4a5b2858ec1c ACFS prdxz_prd Configured
DATA /u02/app/oracle/oradata/prdxz_prd 2.89 TB 2.49 TB 413.16 GB
REDO /u04/app/oracle/redo/prdxz/ 59.99 GB 50.83 GB 9.16 GB
RECO /u03/app/oracle/fast_recovery_area/ 1283.99 GB 328.05 GB 955.94 GB
c783d9cc-8574-4936-b79c-4c11c50b92c6 ACFS abcd_prd Configured
DATA /u02/app/oracle/oradata/abcd_prd 99.99 GB 46.41 GB 53.58 GB
REDO /u04/app/oracle/redo/abcd/ 37.99 GB 2.49 GB 35.50 GB
RECO /u03/app/oracle/fast_recovery_area/ 1283.99 GB 328.05 GB 955.94 GB
07d81563-9382-45d1-bffe-08e969f9d917 ACFS trnxz Configured
DATA /u02/app/oracle/oradata/trnxz 2.71 TB 2.42 TB 301.84 GB
REDO /u04/app/oracle/redo/trnxz/ 37.98 GB 3.64 GB 34.35 GB
RECO /u03/app/oracle/fast_recovery_area/ 1283.99 GB 328.05 GB 955.94 GB
9999777f-fc9b-4a25-9db4-c84a7f4e4627 ACFS abcdhml Configured
DATA /u02/app/oracle/oradata/abcdhml 99.99 GB 9.62 GB 90.37 GB
REDO /u04/app/oracle/redo/abcdhml/ 37.98 GB 1.35 GB 36.64 GB
RECO /u03/app/oracle/fast_recovery_area/ 1283.99 GB 328.05 GB 955.94 GB
9746b884-6887-4022-9847-70b45c88a23e ACFS hmlxz Configured
DATA /u02/app/oracle/oradata/hmlxz 2.77 TB 2.49 TB 293.02 GB
REDO /u04/app/oracle/redo/hmlxz/ 23.99 GB 20.75 GB 3.24 GB
RECO /u03/app/oracle/fast_recovery_area/ 1283.99 GB 328.05 GB 955.94 GB
32a68fdb-ea35-45ec-a614-719bace366d5 ACFS bixz2 Failed
DATA /u02/app/oracle/oradata/bixz2 99.99 GB 576.0 MB 99.43 GB
REDO /u04/app/oracle/redo/bixz2/ 19.99 GB 350.0 MB 19.65 GB
RECO /u03/app/oracle/fast_recovery_area/ 1283.99 GB 328.05 GB 955.94 GB
6) Deletando o DB Storage com falha:
# odacli delete-dbstorage -i 32a68fdb-ea35-45ec-a614-719bace366d5
{
"jobId" : "bc244967-dade-44f2-b1c3-9b3addb6ffdd",
"status" : "Running",
"message" : null,
"reports" : [ ],
"createTimestamp" : "August 13, 2021 12:29:51 PM BRT",
"resourceList" : [ ],
"description" : "Database storage service deletion with id: 32a68fdb-ea35-45ec-a614-719bace366d5",
"updatedTime" : "August 13, 2021 12:29:52 PM BRT"
}
# odacli describe-job -i bc244967-dade-44f2-b1c3-9b3addb6ffdd
Job details
----------------------------------------------------------------
ID: bc244967-dade-44f2-b1c3-9b3addb6ffdd
Description: Database storage service deletion with id: 32a68fdb-ea35-45ec-a614-719bace366d5
Status: Success
Created: August 13, 2021 12:29:51 PM BRT
Message:
7) Desmontando os File Systems criados manualmente:
# umount /u04/app/oracle/redo/bixz2
# umount /u02/app/oracle/oradata/bixz2
8) Removendo os volumes ADVM criados manualmente (usuário grid)
$ asmcmd
ASMCMD> voldelete -G DATA datbixz2
ASMCMD> voldelete -G RECO recobixz2
9) Removendo os File Groups que originaram o problema.
Primeiro, listamos os File Groups para confirmar os nomes corretos. Dica: Um File Group que não está sendo usado, deverá apresentar o valor 0 na coluna “Used Quota MB”:
ASMCMD> lsfg
File Group Disk Group Quota Group Used Quota MB Client Name Client Type
DEFAULT_FILEGROUP DATA GENERIC 0
DEFAULT_FILEGROUP RECO GENERIC 0
SRV-ODAX8-C DATA GENERIC 376 SRV-ODAX8-C CLUSTER
RDOPRDXZ RECO GENERIC 184380 RDOPRDXZ VOLUME
COMMONSTORE DATA GENERIC 15420 COMMONSTORE VOLUME
RECO RECO GENERIC 3944508 RECO VOLUME
DATPRDXZ DATA GENERIC 9123900 DATPRDXZ VOLUME
RDOABCD RECO GENERIC 116796 RDOABCD VOLUME
DATABCD DATA GENERIC 307260 DATABCD VOLUME
RDOTRNXZ RECO GENERIC 116796 RDOTRNXZ VOLUME
DATTRNXZ DATA GENERIC 8570940 DATTRNXZ VOLUME
RDOABCDHML RECO GENERIC 116796 RDOABCDHML VOLUME
DATABCDHML DATA GENERIC 307260 DATABCDHML VOLUME
RDOBIQLIK RECO GENERIC 153660 RDOBIQLIK VOLUME
DATBIQLIK DATA GENERIC 1331240 DATBIQLIK VOLUME
RDOHMLXZ RECO GENERIC 73788 RDOHMLXZ VOLUME
DATHMLXZ DATA GENERIC 5836840 DATHMLXZ VOLUME
RDOBIXZ2 RECO GENERIC 0 RDOBIXZ2 VOLUME
DATBIXZ2 DATA GENERIC 0 DATBIXZ2 VOLUME
Queremos remover os dois últimos, referentes ao database BIXZ2:
ASMCMD> rmfg -r RECO RDOBIXZ2
ASMCMD> rmfg -r DATA DATBIXZ2
A sintanxe do comando é: rmfg -r <Nome do Diskgroup> <Nome do File Group)
10) Nova tentativa de criar o database com mesmo nome não apresenta mais erros:
# odacli create-database --dbname bixz2 --dbhomeid 68861293-e81b-4ca1-9490-ec26f4b7a5b7 --dbstorage ACFS --characterset WE8ISO8859P1
Password for SYS,SYSTEM and PDB Admin:
Job details
----------------------------------------------------------------
ID: f083880c-709d-47d4-992c-0940bfd38932
Description: Database service creation with db name: bixz2
Status: Created
Created: August 13, 2021 12:34:42 PM BRT
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
O processo passou do ponto que apresentava erro:
# odacli describe-job -i f083880c-709d-47d4-992c-0940bfd38932
Job details
----------------------------------------------------------------
ID: f083880c-709d-47d4-992c-0940bfd38932
Description: Database service creation with db name: bixz2
Status: Running
Created: August 13, 2021 12:34:42 PM BRT
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Setting up ssh equivalance August 13, 2021 12:34:51 PM BRT August 13, 2021 12:34:51 PM BRT Success
Create DATA FileGroup DATBIXZ2 August 13, 2021 12:34:51 PM BRT August 13, 2021 12:34:53 PM BRT Success
Create RECO FileGroup RDOBIXZ2 August 13, 2021 12:34:53 PM BRT August 13, 2021 12:34:54 PM BRT Success
Creating volume datbixz2 August 13, 2021 12:34:54 PM BRT August 13, 2021 12:35:24 PM BRT Success
Creating volume rdobixz2 August 13, 2021 12:35:24 PM BRT August 13, 2021 12:35:50 PM BRT Success
Creating ACFS filesystem for DATA August 13, 2021 12:35:50 PM BRT August 13, 2021 12:36:11 PM BRT Success
Creating ACFS filesystem for RECO August 13, 2021 12:36:11 PM BRT August 13, 2021 12:36:34 PM BRT Success
Database Service creation August 13, 2021 12:36:35 PM BRT August 13, 2021 12:36:35 PM BRT Running
Database Creation August 13, 2021 12:36:35 PM BRT August 13, 2021 12:36:35 PM BRT Running