This morning, the Cloud Control and other applications are down.
Hard for the customer :( but as an Oracle DBA I love problems in production so it’s a great day today!
First, I tried to edit a file with vi but a swap error raised: “E297: Write error in swap file“
Another read-only error with SQL*Plus and access to the audit file trail:
sqlplus / as sysdba
Copyright (c) 1982, 2018, Oracle. All rights reserved.
ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 30: Read-only file system
Additional information: 9925
It seems some of my file systems are read-only!
How to check if the file system are Read/Write or read-only?
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=8068884k,nr_inodes=2017221,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
/dev/sda2 / ext4 ro,relatime,data=ordered 0 0
/dev/sda1 /boot ext4 rw,relatime,data=ordered 0 0
/dev/sda7 /u01 ext4 ro,relatime,data=ordered 0 0
/dev/sda3 /tmp ext4 ro,relatime,data=ordered 0 0
/dev/sda5 /var ext4 ro,relatime,data=ordered 0 0
==> KO: Because we can see “ext4 ro” ro = Read only!
Infrastructure team informed us that the NAS was in trouble.
The reboot of the server solved the problem.
We can also verify other things, like free space and inodes space.
I can also check free space
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 9.5G 5.6G 3.4G 63% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 477M 158M 291M 36% /boot
/dev/sda7 91G 78G 8.8G 90% /u01
/dev/sda3 5.7G 26M 5.4G 1% /tmp
/dev/sda5 5.7G 267M 5.1G 5% /var
==> Everything is OK
In case you have a space problem you can list big files with this command:
find . -type f -size +50M
(ex: file > 50Mb)
'c' for bytes 'w' for two-byte words 'k' for Kilobytes (units of 1024 bytes) 'M' for Megabytes (units of 1048576 bytes) 'G' for Gigabytes (units of 1073741824 bytes)
Finally, I can check free inodes space
An inode is used for each file on the filesystem. So running out of inodes generally means you’ve got a lot of small files laying around.
If you are very unlucky you have used about 100% of all inodes. This bash command may help you:
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 640848 73501 567347 12% /
tmpfs 2021073 2 2021071 1% /dev/shm
/dev/sda1 128016 60 127956 1% /boot
/dev/sda7 6045696 244708 5800988 5% /u01
/dev/sda3 384272 3009 381263 1% /tmp
/dev/sda5 384272 2088 382184 1% /var
We don’t have more than 12% inodes used (for /). I don’t have any problems with the number of inodes used.
In case you have a lot of inodes used, you can list the directories sorted with the number of files with this command:
find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n
Author: Vincent Fenoll – Oracle DBA