Unable to create audit trail file or Read-only file system

This morning, the Cloud Control and other applications are down.

Hard for the customer :(  but as an Oracle DBA I love problems in production so it’s a great day  today!

First, I tried to edit a file with vi but a swap error raised:  « E297: Write error in swap file« 

Another read-only error with SQL*Plus and access to the audit file trail:

sqlplus / as sysdba
Version 18.3.0.0.0
Copyright (c) 1982, 2018, Oracle. All rights reserved.

ERROR:
ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 30: Read-only file system
Additional information: 9925

 

It seems some of my file systems are read-only!

 

How to check if the file system are Read/Write or read-only?

cat /proc/mounts
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=8068884k,nr_inodes=2017221,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
/dev/sda2 / ext4 ro,relatime,data=ordered 0 0
/dev/sda1 /boot ext4 rw,relatime,data=ordered 0 0
/dev/sda7 /u01 ext4 ro,relatime,data=ordered 0 0
/dev/sda3 /tmp ext4 ro,relatime,data=ordered 0 0
/dev/sda5 /var ext4 ro,relatime,data=ordered 0 0

==> KO: Because we can see « ext4 ro » ro = Read only!

Infrastructure team informed us that the NAS was in trouble.
The reboot of the server solved the problem.

 

We can also verify other things, like free space and inodes space.

 

I can also check free space

 

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 9.5G 5.6G 3.4G 63% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 477M 158M 291M 36% /boot
/dev/sda7 91G 78G 8.8G 90% /u01
/dev/sda3 5.7G 26M 5.4G 1% /tmp
/dev/sda5 5.7G 267M 5.1G 5% /var

==> Everything is OK

 

In case you have a space problem you can list big files with this command:

find . -type f -size +50M
(ex: file > 50Mb)

'c' for bytes 'w' for two-byte words 'k' for Kilobytes (units of 1024 bytes) 'M' for Megabytes (units of 1048576 bytes) 'G' for Gigabytes (units of 1073741824 bytes)

 

Finally, I can check free inodes space 

 

An inode is used for each file on the filesystem. So running out of inodes generally means you’ve got a lot of small files laying around. 

 If you are very unlucky you have used about 100% of all inodes.  This bash command may help you:

df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 640848 73501 567347 12% /
tmpfs 2021073 2 2021071 1% /dev/shm
/dev/sda1 128016 60 127956 1% /boot
/dev/sda7 6045696 244708 5800988 5% /u01
/dev/sda3 384272 3009 381263 1% /tmp
/dev/sda5 384272 2088 382184 1% /var

We don’t have more than 12% inodes used (for /). I don’t have any problems with the number of inodes used.

In case you have a lot of inodes used, you can list the directories sorted with the number of files with this command:

find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n

 

Author: Vincent Fenoll – Oracle DBA

Summary
Article Name
Unable to create audit trail file or Read-only file system
Description
How to check if the file system are Read/Write or read-only? Check free inodes / space. Unable to create audit trail file.
Author