2

We are using BeagleBoneBlack based custom board,

During firmware upgrade I got following errors

[  194.912834] EXT4-fs error (device mmcblk0p15): ext4_journal_check_start:56: Detected aborted journal
[  194.922558] EXT4-fs (mmcblk0p15): Remounting filesystem read-only

Rebooting the board recovered this error but for testing I want to simulate this error and then in my upgrade scripts I want to handle the error case.

is there a way to simulate above error ?

EDIT:

Putting more details,
I saw above error while testing my firmware upgrade script, and it got resolved after reboot,

After some search i found that e2fsck could fix some file system errors.
So I updated my firmware upgrade script with following snippet. Now My problem is I need to test my script for below snippet, For that I need a situation where e2fsck correctable errors are there in the backup_partition, So I was looking for a options to create such error situation.

E2FSCK_EXITCODE_NO_ERROR=0
E2FSCK_EXITCODE_FS_ERROR_CORRECTED_REBOOT_REQUIRED=2
backup_partition="/dev/mmcblk0p16"

e2fsck -n "${backup_partition}" || status="${?}"
if [ "${status}" -ne "${E2FSCK_EXITCODE_NO_ERROR}" ];then
   echo "filesystem errors are observed in ${backup_partition}. e2fsck EXIT_CODE=${status}"
   status=0
   e2fsck -y "${backup_partition}" || status="${?}"
   if [ "${status}" -ge "${E2FSCK_EXITCODE_FS_ERROR_CORRECTED_REBOOT_REQUIRED}" ]; then
      return_code=$(( status & E2FSCK_EXITCODE_FS_ERROR_CORRECTED_REBOOT_REQUIRED ))
      if [ "${return_code}" -ne 0 ]; then
         echo "Exiting Filesystem errors corrected for ${backup_partition}, However reboot required, e2fsck EXIT_CODE=${status}."
      else
         echo "Exiting .. Filesystem errors are not corrected for ${backup_partition}, e2fsck EXIT_CODE=${status}."
      fi
      exit
   fi
fi

EDIT2

I could reproduce error, using following scenario
1. Read /dev/mmcblk0p15 using dd in one window,
2. Simultaneously mount /dev/mmcblk0p15 and remove files and copy files.

# e2fsck -n /dev/mmcblk0p15  > /home/e2fscklog
e2fsck 1.42.8 (20-Jun-2013)
    Warning: skipping journal recovery because doing a read-only filesystem check.
rootfs contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry '..' in .../??? (1224) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (1318) has deleted/unused inode 99310.  Clear? no
Entry 'jerq' in .../??? (1318) has deleted/unused inode 1321.  Clear? no
Entry '..' in .../??? (1344) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (1423) has deleted/unused inode 99310.  Clear? no
Entry '730MTGr-24' in .../??? (1423) has deleted/unused inode 1002.  Clear? no
Entry '730MTGr' in .../??? (1423) has deleted/unused inode 1035.  Clear? no
Entry '730MTG-41r' in .../??? (1423) has deleted/unused inode 906.  Clear? no
Entry '..' in .../??? (1424) has deleted/unused inode 99310.  Clear? no
Entry 'f1720a' in .../??? (1424) has deleted/unused inode 1425.  Clear? no
Entry 'fenix' in .../??? (1424) has deleted/unused inode 445.  Clear? no
Entry 'f1720' in .../??? (1424) has deleted/unused inode 1425.  Clear? no
Entry 'freedom110' in .../??? (1424) has deleted/unused inode 1426.  Clear? no
Entry 'fenixw' in .../??? (1424) has deleted/unused inode 440.  Clear? no
Entry 'f110-14w' in .../??? (1424) has deleted/unused inode 1427.  Clear? no
Entry 'fos' in .../??? (1424) has deleted/unused inode 1428.  Clear? no
Entry 'f200' in .../??? (1424) has deleted/unused inode 1429.  Clear? no
Entry 'fox' in .../??? (1424) has deleted/unused inode 498.  Clear? no
Entry 'fortune' in .../??? (1424) has deleted/unused inode 1428.  Clear? no
Entry 'f110-14' in .../??? (1424) has deleted/unused inode 1430.  Clear? no
Entry 'freedom100' in .../??? (1424) has deleted/unused inode 1431.  Clear? no
Entry 'falco-p' in .../??? (1424) has deleted/unused inode 1432.  Clear? no
Entry 'f200-w' in .../??? (1424) has deleted/unused inode 1433.  Clear? no
Entry 'freedom200' in .../??? (1424) has deleted/unused inode 1429.  Clear? no
Entry 'f200vi' in .../??? (1424) has deleted/unused inode 1434.  Clear? no
Entry 'falco' in .../??? (1424) has deleted/unused inode 1435.  Clear? no
Entry 'f110-w' in .../??? (1424) has deleted/unused inode 1436.  Clear? no
Entry 'f100-rv' in .../??? (1424) has deleted/unused inode 1437.  Clear? no
Entry 'f100-rv' in .../??? (1424) has an incorrect filetype (was 1, should be 2).
Fix? no
Entry 'f100' in .../??? (1424) has deleted/unused inode 1431.  Clear? no
Entry 'f200vi-w' in .../??? (1424) has deleted/unused inode 1438.  Clear? no
Entry 'f110' in .../??? (1424) has deleted/unused inode 1426.  Clear? no
Entry 'freedom' in .../??? (1424) has deleted/unused inode 1431.  Clear? no
Entry 'freedom-rv' in .../??? (1424) has deleted/unused inode 1437.  Clear? no
Entry 'freedom-rv' in .../??? (1424) has an incorrect filetype (was 1, should be 2).
Fix? no
Entry '..' in .../??? (1472) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (1499) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (1500) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (1553) has deleted/unused inode 99310.  Clear? no
Entry '630MTG-24' in .../??? (1553) has deleted/unused inode 981.  Clear? no
Entry '..' in .../??? (1554) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (1610) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (1614) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (511) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (608) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (610) has deleted/unused inode 99310.  Clear? no
Entry 'gs5430-22' in .../??? (610) has deleted/unused inode 444.  Clear? no
Entry 'gs5430' in .../??? (610) has deleted/unused inode 484.  Clear? no
Entry '..' in .../??? (652) has deleted/unused inode 99310.  Clear? no
Entry 'unixpc' in .../??? (652) has deleted/unused inode 431.  Clear? no
Entry '..' in .../??? (658) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (661) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (669) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (745) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (389) has deleted/unused inode 99308.  Clear? no
Entry '..' in .../??? (393) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (424) has deleted/unused inode 99310.  Clear? no
Entry '..' in .../??? (819) has deleted/unused inode 99310.  Clear? no
Entry 'liswb' in .../??? (819) has deleted/unused inode 833.  Clear? no
Entry 'layer' in .../??? (819) has deleted/unused inode 834.  Clear? no
Entry 'lisaterm' in .../??? (819) has deleted/unused inode 835.  Clear? no
Entry 'ln03-w' in .../??? (819) has deleted/unused inode 836.  Clear? no
Entry 'linux-vt' in .../??? (819) has deleted/unused inode 837.  Clear? no
Entry 'linux-c' in .../??? (819) has deleted/unused inode 838.  Clear? no
Entry 'linux-m' in .../??? (819) has deleted/unused inode 839.  Clear? no
Entry 'linux2.6.26' in .../??? (819) has deleted/unused inode 840.  Clear? no
Entry 'lisa' in .../??? (819) has deleted/unused inode 841.  Clear? no
Entry '..' in .../??? (1059) has deleted/unused inode 99310.  Clear? no
Entry 'share' in /usr (65572) has deleted/unused inode 99306.  Clear? no
Entry 'dg211' in .../??? (1059) has deleted/unused inode 1099.  Clear? no
Entry 'd430-unix-w' in .../??? (1059) has deleted/unused inode 1103.  Clear? no
Entry 'dku7102' in .../??? (1059) has deleted/unused inode 1101.  Clear? no
Entry 'dmd-24' in .../??? (1059) has deleted/unused inode 936.  Clear? no
Entry 'darwin-80x25' in .../??? (1059) has deleted/unused inode 1209.  Clear? no
Entry 'd217-unix-25' in .../??? (1059) has deleted/unused inode 1211.  Clear? no
Entry 'ddr3180' in .../??? (1059) has deleted/unused inode 1212.  Clear? no
Entry 'd414-unix-sr' in .../??? (1059) has deleted/unused inode 1215.  Clear? no
Entry 'ddr' in .../??? (1059) has deleted/unused inode 1212.  Clear? no
Entry 'darwin-m-f' in .../??? (1059) has deleted/unused inode 1216.  Clear? no
Entry 'dmd1' in .../??? (1059) has deleted/unused inode 907.  Clear? no
Entry 'd464-unix-sr' in .../??? (1059) has deleted/unused inode 1215.  Clear? no
Entry 'd578-dg' in .../??? (1059) has deleted/unused inode 1098.  Clear? no
Entry 'd430c-unix-w' in .../??? (1059) has deleted/unused inode 1103.  Clear? no
Entry 'd214' in .../??? (1059) has deleted/unused inode 1104.  Clear? no
Entry 'd210' in .../??? (1059) has deleted/unused inode 1104.  Clear? no
Entry 'd410-7b-w' in .../??? (1059) has deleted/unused inode 1100.  Clear? no
Entry 'd210-dg' in .../??? (1059) has deleted/unused inode 1155.  Clear? no
Entry 'd414-unix' in .../??? (1059) has deleted/unused inode 1097.  Clear? no
Entry 'darwin-80x25-m' in .../??? (1059) has deleted/unused inode 1160.  Clear? no
Entry 'dmd-34' in .../??? (1059) has deleted/unused inode 1032.  Clear? no
Entry 'dku7103-sna' in .../??? (1059) has deleted/unused inode 1213.  Clear? no
Entry 'd577-dg' in .../??? (1059) has deleted/unused inode 1098.  Clear? no
Entry 'd411-7b-w' in .../??? (1059) has deleted/unused inode 1100.  Clear? no
Entry 'ds40-2' in .../??? (1059) has deleted/unused inode 1157.  Clear? no
Entry 'vapple' in .../??? (1224) has deleted/unused inode 983.  Clear? no
Entry 'viewpoint60' in .../??? (1224) has deleted/unused inode 946.  Clear? no
Entry 'vp60' in .../??? (1224) has deleted/unused inode 946.  Clear? no
Entry 'vitty' in .../??? (1224) has deleted/unused inode 834.  Clear? no
Entry 'viewpoint' in .../??? (1224) has deleted/unused inode 988.  Clear? no
Entry 'xterm-r5' in .../??? (1344) has deleted/unused inode 1386.  Clear? no
Entry 'xterm-xf86-v333' in .../??? (1344) has deleted/unused inode 1388.  Clear? no
Entry 'xterm-xf86-v333' in .../??? (1344) has an incorrect filetype (was 1, should be 2).
Fix? no
Entry 'xterm-8bit' in .../??? (1344) has deleted/unused inode 1389.  Clear? no
Entry 'xterm-8bit' in .../??? (1344) has an incorrect filetype (was 1, should be 2).
Fix? no
Entry 'xnuppc-m-f' in .../??? (1344) has deleted/unused inode 1216.  Clear? no
Entry 'xnuppc+80x25' in .../??? (1344) has deleted/unused inode 1385.  Clear? no
Entry 'xnuppc+100x37' in .../??? (1344) has deleted/unused inode 1392.  Clear? no
Entry 'xnuppc+f2' in .../??? (1344) has deleted/unused inode 1390.  Clear? no
Entry 'xdku' in .../??? (1344) has deleted/unused inode 1391.  Clear? no
Entry 'xl83' in .../??? (1344) has deleted/unused inode 787.  Clear? no
Entry 'xnuppc-80x25' in .../??? (1344) has deleted/unused inode 1209.  Clear? no
Entry 'xterm+88color' in .../??? (1344) has deleted/unused inode 1387.  Clear? no
Entry 'xnuppc-80x25-m' in .../??? (1344) has deleted/unused inode 1160.  Clear? no
Entry 'xnuppc-m-b' in .../??? (1344) has deleted/unused inode 1156.  Clear? no
Entry 'microterm' in .../??? (1500) has deleted/unused inode 952.  Clear? no
Entry 'mai' in .../??? (1500) has deleted/unused inode 1546.  Clear? no
Entry 'mime2a-v' in .../??? (1500) has deleted/unused inode 1547.  Clear? no
Entry 'masscomp2' in .../??? (1500) has deleted/unused inode 1548.  Clear? no
Entry 'microterm5' in .../??? (1500) has deleted/unused inode 1034.  Clear? no
Entry 'mlterm-256color' in .../??? (1500) has deleted/unused inode 1551.  Clear? no
Entry 'modgraph2' in .../??? (1500) has deleted/unused inode 1545.  Clear? no
Entry 'megatek' in .../??? (1500) has deleted/unused inode 1549.  Clear? no
Entry 'mime2a' in .../??? (1500) has deleted/unused inode 1547.  Clear? no
Entry 'mach-color' in .../??? (1500) has deleted/unused inode 1550.  Clear? no
Entry 'mime-hb' in .../??? (1500) has deleted/unused inode 1552.  Clear? no
Entry 'ibmapa8c-c' in .../??? (1554) has deleted/unused inode 1563.  Clear? no
Entry 'ibm5151' in .../??? (1554) has deleted/unused inode 1564.  Clear? no
Entry 'ibm6154' in .../??? (1554) has deleted/unused inode 1569.  Clear? no
Entry 'ibm6154' in .../??? (1554) has an incorrect filetype (was 1, should be 2).
Fix? no
Entry 'iris40' in .../??? (1554) has deleted/unused inode 1570.  Clear? no
Entry 'iris40' in .../??? (1554) has an incorrect filetype (was 1, should be 2).
Fix? no
Entry 'ibm5154-c' in .../??? (1554) has deleted/unused inode 1575.  Clear? no
Entry 'ibm5154-c' in .../??? (1554) has an incorrect filetype (was 1, should be 2).
Fix? no
Entry 'ibmpc' in .../??? (1554) has deleted/unused inode 1576.  Clear? no
Entry 'ibmmono' in .../??? (1554) has deleted/unused inode 1577.  Clear? no
Entry 'ibm3164' in .../??? (1554) has deleted/unused inode 1578.  Clear? no
Entry 'ibm3161' in .../??? (1554) has deleted/unused inode 1579.  Clear? no
Entry 'ibm3161' in .../??? (1554) has an incorrect filetype (was 1, should be 2).
Fix? no
Entry 'ibm8512' in .../??? (1554) has deleted/unused inode 1572.  Clear? no
Entry 'ibm6153-40' in .../??? (1554) has deleted/unused inode 1581.  Clear? no
Entry 'ibm-apl' in .../??? (1554) has deleted/unused inode 1030.  Clear? no
Entry 'interix' in .../??? (1554) has deleted/unused inode 1582.  Clear? no
Entry 'ibm3101' in .../??? (1554) has deleted/unused inode 1584.  Clear? no
Entry 'ibm3163' in .../??? (1554) has deleted/unused inode 1579.  Clear? no
Entry 'ibm3163' in .../??? (1554) has an incorrect filetype (was 1, should be 2).
Fix? no

............................CUT.............


Unattached zero-length inode 944.  Clear? no

Unattached inode 944
Connect to /lost+found? no

Unattached zero-length inode 953.  Clear? no

Unattached inode 953
Connect to /lost+found? no

Unattached zero-length inode 954.  Clear? no

Unattached inode 954
Connect to /lost+found? no

Unattached zero-length inode 955.  Clear? no

Unattached inode 955
Connect to /lost+found? no

Unattached zero-length inode 956.  Clear? no

Unattached inode 956
Connect to /lost+found? no

Unattached zero-length inode 957.  Clear? no

Unattached inode 957
Connect to /lost+found? no

Unattached zero-length inode 958.  Clear? no

Unattached inode 958
Connect to /lost+found? no

Unattached zero-length inode 959.  Clear? no

Unattached inode 959
Connect to /lost+found? no

Unattached zero-length inode 960.  Clear? no

Unattached inode 960
Connect to /lost+found? no

Unattached zero-length inode 961.  Clear? no

Unattached inode 961
Connect to /lost+found? no

Unattached zero-length inode 962.  Clear? no

Unattached inode 962
Connect to /lost+found? no

Unattached zero-length inode 963.  Clear? no

Unattached inode 963
Connect to /lost+found? no

Inode 964 ref count is 2, should be 1.  Fix? no

Unattached zero-length inode 965.  Clear? no

Unattached inode 965
Connect to /lost+found? no

Unattached zero-length inode 966.  Clear? no

Unattached inode 966
Connect to /lost+found? no

Unattached zero-length inode 967.  Clear? no

Unattached inode 967
Connect to /lost+found? no

Unattached zero-length inode 968.  Clear? no

Unattached inode 968
Connect to /lost+found? no

Unattached zero-length inode 993.  Clear? no

Unattached inode 993
Connect to /lost+found? no

Unattached zero-length inode 994.  Clear? no

Unattached inode 994
Connect to /lost+found? no

Unattached zero-length inode 995.  Clear? no

Unattached inode 995
Connect to /lost+found? no

Unattached zero-length inode 996.  Clear? no

Unattached inode 996
Connect to /lost+found? no

Unattached zero-length inode 997.  Clear? no

Unattached inode 997
Connect to /lost+found? no

Unattached zero-length inode 998.  Clear? no

Unattached inode 998
Connect to /lost+found? no

Unattached zero-length inode 999.  Clear? no

Unattached inode 999
Connect to /lost+found? no

Unattached zero-length inode 1000.  Clear? no

Unattached inode 1000
Connect to /lost+found? no

Unattached zero-length inode 1009.  Clear? no

Unattached inode 1009
Connect to /lost+found? no

Unattached zero-length inode 1010.  Clear? no

Unattached inode 1010
Connect to /lost+found? no

Unattached zero-length inode 1011.  Clear? no

Unattached inode 1011
Connect to /lost+found? no

Unattached zero-length inode 1012.  Clear? no

Unattached inode 1012
Connect to /lost+found? no

Unattached zero-length inode 1013.  Clear? no

Unattached inode 1013
Connect to /lost+found? no

Unattached zero-length inode 1014.  Clear? no

Unattached inode 1014
Connect to /lost+found? no

Unattached zero-length inode 1015.  Clear? no

Unattached inode 1015
Connect to /lost+found? no

Unattached zero-length inode 1016.  Clear? no

Unattached inode 1016
Connect to /lost+found? no

Inode 1017 ref count is 2, should be 1.  Fix? no

Unattached zero-length inode 1018.  Clear? no

Unattached inode 1018
Connect to /lost+found? no

Unattached zero-length inode 1019.  Clear? no

Unattached inode 1019
Connect to /lost+found? no

Unattached zero-length inode 1020.  Clear? no

Unattached inode 1020
Connect to /lost+found? no

Unattached zero-length inode 1021.  Clear? no

Unattached inode 1021
Connect to /lost+found? no

Unattached zero-length inode 1022.  Clear? no

Unattached inode 1022
Connect to /lost+found? no

Unattached zero-length inode 1023.  Clear? no

Unattached inode 1023
Connect to /lost+found? no

Unattached zero-length inode 1024.  Clear? no

Unattached inode 1024
Connect to /lost+found? no

Inode 1041 ref count is 5, should be 3.  Fix? no

Unattached zero-length inode 1042.  Clear? no

Unattached inode 1042
Connect to /lost+found? no

Unattached zero-length inode 1043.  Clear? no

Unattached inode 1043
Connect to /lost+found? no

Unattached zero-length inode 1044.  Clear? no

Unattached inode 1044
Connect to /lost+found? no

Unattached zero-length inode 1045.  Clear? no

Unattached inode 1045
Connect to /lost+found? no

Unattached zero-length inode 1046.  Clear? no

Unattached inode 1046
Connect to /lost+found? no

Unattached zero-length inode 1047.  Clear? no

Unattached inode 1047
Connect to /lost+found? no

Unattached zero-length inode 1048.  Clear? no

Unattached inode 1048
Connect to /lost+found? no

Unattached zero-length inode 1057.  Clear? no

Unattached inode 1057
Connect to /lost+found? no

Unattached zero-length inode 1058.  Clear? no

Unattached inode 1058
Connect to /lost+found? no

Inode 1059 ref count is 2, should be 1.  Fix? no

Inode 1224 ref count is 2, should be 1.  Fix? no

Inode 1237 ref count is 3, should be 2.  Fix? no

Inode 1259 ref count is 4, should be 2.  Fix? no

Inode 1268 ref count is 4, should be 2.  Fix? no

Inode 1318 ref count is 2, should be 1.  Fix? no

Unattached zero-length inode 1329.  Clear? no

Unattached inode 1329
Connect to /lost+found? no

Unattached zero-length inode 1330.  Clear? no

Unattached inode 1330
Connect to /lost+found? no

Unattached zero-length inode 1331.  Clear? no

Unattached inode 1331
Connect to /lost+found? no

Unattached zero-length inode 1332.  Clear? no

Unattached inode 1332
Connect to /lost+found? no

Unattached zero-length inode 1333.  Clear? no

Unattached inode 1333
Connect to /lost+found? no

Unattached zero-length inode 1334.  Clear? no

Unattached inode 1334
Connect to /lost+found? no

Unattached zero-length inode 1335.  Clear? no

Unattached inode 1335
Connect to /lost+found? no

Unattached zero-length inode 1336.  Clear? no

Unattached inode 1336
Connect to /lost+found? no

Unattached zero-length inode 1337.  Clear? no

Unattached inode 1337
Connect to /lost+found? no

Unattached zero-length inode 1338.  Clear? no

Unattached inode 1338
Connect to /lost+found? no

Unattached zero-length inode 1339.  Clear? no

Unattached inode 1339
Connect to /lost+found? no

Unattached zero-length inode 1340.  Clear? no

Unattached inode 1340
Connect to /lost+found? no

Unattached zero-length inode 1341.  Clear? no

Unattached inode 1341
Connect to /lost+found? no

Unattached zero-length inode 1342.  Clear? no

Unattached inode 1342
Connect to /lost+found? no

Unattached zero-length inode 1343.  Clear? no

Unattached inode 1343
Connect to /lost+found? no

Inode 1344 ref count is 2, should be 1.  Fix? no

Inode 1423 ref count is 2, should be 1.  Fix? no

Inode 1424 ref count is 2, should be 1.  Fix? no

Unattached zero-length inode 1449.  Clear? no

Unattached inode 1449
Connect to /lost+found? no

Inode 1450 ref count is 2, should be 1.  Fix? no

Unattached zero-length inode 1451.  Clear? no

Unattached inode 1451
Connect to /lost+found? no

Unattached zero-length inode 1452.  Clear? no

Unattached inode 1452
Connect to /lost+found? no

Unattached zero-length inode 1453.  Clear? no

Unattached inode 1453
Connect to /lost+found? no

Unattached zero-length inode 1454.  Clear? no

Unattached inode 1454
Connect to /lost+found? no

Unattached zero-length inode 1455.  Clear? no

Unattached inode 1455
Connect to /lost+found? no

Unattached zero-length inode 1456.  Clear? no

Unattached inode 1456
Connect to /lost+found? no

Unattached zero-length inode 1457.  Clear? no

Unattached inode 1457
Connect to /lost+found? no

Unattached zero-length inode 1458.  Clear? no

Unattached inode 1458
Connect to /lost+found? no

Unattached zero-length inode 1459.  Clear? no

Unattached inode 1459
Connect to /lost+found? no

Unattached zero-length inode 1460.  Clear? no

Unattached inode 1460
Connect to /lost+found? no

Unattached zero-length inode 1461.  Clear? no

Unattached inode 1461
Connect to /lost+found? no

Unattached zero-length inode 1462.  Clear? no

Unattached inode 1462
Connect to /lost+found? no

Unattached zero-length inode 1463.  Clear? no

Unattached inode 1463
Connect to /lost+found? no

Unattached zero-length inode 1464.  Clear? no

Unattached inode 1464
Connect to /lost+found? no

Unattached zero-length inode 1465.  Clear? no

Unattached inode 1465
Connect to /lost+found? no

Unattached zero-length inode 1466.  Clear? no

Unattached inode 1466
Connect to /lost+found? no

Unattached zero-length inode 1467.  Clear? no

Unattached inode 1467
Connect to /lost+found? no

Unattached zero-length inode 1468.  Clear? no

Unattached inode 1468
Connect to /lost+found? no

Unattached zero-length inode 1469.  Clear? no

Unattached inode 1469
Connect to /lost+found? no

Unattached zero-length inode 1470.  Clear? no

Unattached inode 1470
Connect to /lost+found? no

Unattached zero-length inode 1471.  Clear? no

Unattached inode 1471
Connect to /lost+found? no

Inode 1472 ref count is 2, should be 1.  Fix? no

Inode 1499 ref count is 2, should be 1.  Fix? no

Inode 1500 ref count is 2, should be 1.  Fix? no

Inode 1553 ref count is 2, should be 1.  Fix? no

Inode 1554 ref count is 2, should be 1.  Fix? no

Unattached zero-length inode 1609.  Clear? no

Unattached inode 1609
Connect to /lost+found? no

Inode 1610 ref count is 2, should be 1.  Fix? no

Inode 1614 ref count is 2, should be 1.  Fix? no

Inode 1621 ref count is 2, should be 1.  Fix? no

Inode 65572 ref count is 9, should be 8.  Fix? no

Pass 5: Checking group summary information
Block bitmap differences:  +(4361--4368) +(4376--4394) +(4396--4402) +(4406--4414) +(4419--4425) +(4609--4632) +(4657--4658) +(4669--4672) -(68504--68514) -(68545--68560) -(71625--71647) -(71661--71680) -(397321--397324) -(413522--413590)
Fix? no

Free blocks count wrong for group #1 (6435, counted=0).
Fix? no

Free blocks count wrong for group #2 (8192, counted=0).
Fix? no

Free blocks count wrong for group #3 (7933, counted=0).
Fix? no

Free blocks count wrong for group #4 (8192, counted=0).
Fix? no

Free blocks count wrong for group #5 (7933, counted=0).
Fix? no

Free blocks count wrong for group #6 (8192, counted=0).
Fix? no

Free blocks count wrong for group #7 (7933, counted=0).
Fix? no

Free blocks count wrong for group #8 (8192, counted=3007).
Fix? no

Free blocks count wrong for group #50 (460, counted=107).
Fix? no

Free blocks count wrong for group #51 (6686, counted=5471).
Fix? no

Free blocks count wrong (346495, counted=346694).
Fix? no

Inode bitmap differences:  +(194--424) +(449--480) +(505--672) +(681--784) +(801--832) +(865--872) +(937--944) +(953--968) +(993--1000) +(1009--1024) +(1041--1048) +(1057--1096) +(1105--1152) +(1161--1208) +(1217--1320) +(1329--1384) +(1393--1424) +(1449--1544) +(1553--1560) +(1585--1600) +(1609--1624) +(1633--1640) +(1649--1664) +(1681--1688) +(1705--1720) -(99305--99310)
Fix? no

Directories count wrong for group #48 (252, counted=248).
Fix? no

Free inodes count wrong (119307, counted=122509).
Fix? no


rootfs: ********** WARNING: Filesystem still has errors **********

rootfs: 8709/128016 files (0.1% non-contiguous), 165505/512000 blocks

Error in dmesg

# [177307.226736] EXT4-fs error (device mmcblk0p15): ext4_journal_check_start:56: Detected aborted journal
[177307.236546] EXT4-fs (mmcblk0p15): Remounting filesystem read-only
# [177308.804317] EXT4-fs error (device mmcblk0p15): ext4_put_super:772: Couldn't clean up the journal
ART
  • 1,509
  • 3
  • 29
  • 47

3 Answers3

4

To answer the new, modified question, here's an example of how to create a file system with a corruption which e2fsck can fix:

<tytso@closure> {/u1/xfstests-bld/build-64}   (master)
521% debugfs -w /tmp/foo.img
debugfs 1.43.1 (08-Jun-2016)
debugfs:  write /dev/null file-to-clri
Allocated inode: 12
debugfs:  clri file-to-clri
debugfs:  q
<tytso@closure> {/u1/xfstests-bld/build-64}   (master)
522% sudo mount /tmp/foo.img /mnt
<tytso@closure> {/u1/xfstests-bld/build-64}   (master)
523% ls -sF /mnt
/bin/ls: cannot access '/mnt/file-to-clri': Structure needs cleaning
total 12
 ? file-to-clri  12 lost+found/
<tytso@closure> {/u1/xfstests-bld/build-64}   (master)
524% dmesg | tail -3
[35156.062886] EXT4-fs (loop0): mounting ext2 file system using the ext4 subsystem
[35156.065760] EXT4-fs (loop0): mounted filesystem without journal. Opts: (null)
[35161.963603] EXT4-fs error (device loop0): ext4_lookup:1608: inode #2: comm ls: deleted inode referenced: 12
<tytso@closure> {/u1/xfstests-bld/build-64}   (master)
525% sudo umount /mnt
<tytso@closure> {/u1/xfstests-bld/build-64}   (master)
526% e2fsck -y /tmp/foo.img
e2fsck 1.43.1 (08-Jun-2016)
/tmp/foo.img contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'file-to-clri' in / (2) has deleted/unused inode 12.  Clear? yes

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Inode bitmap differences:  -12
Fix? yes

Free inodes count wrong for group #0 (4, counted=5).
Fix? yes

Free inodes count wrong (4, counted=5).
Fix? yes


/tmp/foo.img: ***** FILE SYSTEM WAS MODIFIED *****
/tmp/foo.img: 11/16 files (0.0% non-contiguous), 21/100 blocks
Theodore Ts'o
  • 556
  • 2
  • 8
  • One more thing I missed is what if e2fsck takes much more times( trying to correct errors), do I need to think about timeout or e2fsck handles timeout? Do you have any suggestion for timeout ? How can I test worst possible timeout for e2fsck. We have 500MB of ext4 partition to be checked for file system errors. – ART Aug 19 '16 at 15:22
  • I really think you're going about this the wrong way. It's far more important for you to understand why the file system is getting corrupted in the first place. If it's because your supply chain folks are selecting crappy flash devices, then tell them to get some quality product in. After all, what are you going to do if e2fsck times out? Tell the customer to ship the box back for a warantee replacement? And if the file system is getting corrupted due to crappy hardware, things could get so corrupted e2fsck can't even run at all, and then what will you do? – Theodore Ts'o Aug 20 '16 at 19:48
  • I know what you are pointing at. Even we intend to find root cause. It started with error mentioned in [this](https://groups.google.com/forum/#!topic/beagleboard/L7piqfHiyO8) we might not have this scenario all the time And changing `/proc/sys/vm/min_free_kbytes` didn't solve issue. So we need to find the root cause, In above post Mr.Nelson says it looks to mmc driver issue as it was not stable in 3.12 kernel. However we can't kernel at this stage of project, in next phase we might be able to. We want to find the root cause, however we need stop-gap solution. – ART Aug 22 '16 at 11:37
  • While debugging I found another case(mentioned in above post) where we end up with readonly partition. So at-least to recover from mentioned situation next time we run script we use `e2fsck`. One of the question during review was what is worst case time `e2fsck` might take (in script snippet mentioned in above post). If `e2fsck` couldn't correct error within X seconds/minutes we intend to error out in our script. – ART Aug 22 '16 at 11:45
  • Can we simulate error which is *uncorrectable* by `e2fsck -p` command ? – ART Dec 29 '16 at 14:33
  • There are plenty of errors which are uncorrectable by "e2fsck -p". Can you be more specific about what it is you want to test? Do you want something which is correctable by "e2fsck -y"? Does any data have to be left to be recoverable afterwards? You're just being incredibly vague..... – Theodore Ts'o Dec 30 '16 at 16:36
  • Thank you for replying, actually I want both of them. And If I may ask I wanted to test with different types of errors which are possible incase of power outage and kernel bugs. – ART Dec 30 '16 at 17:23
  • I tried deleting inode and super blocks for a partition using debugfs but none of them worked :( , I guess might be journal is helping it to recover. Any suggestion/pointers how can I create such an error ? – ART Jan 05 '17 at 12:05
  • any hint or suggestion for me ? – ART Jan 09 '17 at 14:39
1

The simplest way to simulate an aborted journal is to mount the file system with the mount option errors=remount-ro, and then trigger a simulated file system error using /sys/fs//trigger_fs_error. For example:

% kvm-xfstests shell
...
Debian GNU/Linux 8 kvm-xfstests ttyS0

kvm-xfstests login: root (automatic login)

Last login: Wed Jul 27 09:45:36 EDT 2016 on ttyS2
Linux kvm-xfstests 4.7.0-rc1-00021-g5431928 #647 SMP Fri Jul 15 00:09:57 EDT 2016 i686

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@kvm-xfstests:~# mke2fs -t ext4 -Fq /dev/vdc
/dev/vdc contains a ext4 file system
    last mounted on Tue Jul 26 22:07:38 2016
root@kvm-xfstests:~# dmesg -n 7
root@kvm-xfstests:~# mount -o errors=remount-ro /dev/vdc /vdc
[  212.945779] EXT4-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro
root@kvm-xfstests:~# echo test error > /sys/fs/ext4/vdc/trigger_fs_error 
[  218.987724] EXT4-fs error (device vdc): trigger_test_error:123: comm bash: test error
[  218.993043] Aborting journal on device vdc-8.
[  218.997476] EXT4-fs (vdc): Remounting filesystem read-only
root@kvm-xfstests:~# umount /vdc
[  312.335032] EXT4-fs error (device vdc): ext4_put_super:837: Couldn't clean up the journal
root@kvm-xfstests:~# e2fsck -fy /dev/vdc
e2fsck 1.43.1 (08-Jun-2016)
/dev/vdc: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vdc: 11/327680 files (0.0% non-contiguous), 58462/1310720 blocks
root@kvm-xfstests:~#

For more information about kvm-xfstests, please see: http://thunk.org/gce-xfstests or https://github.com/tytso/xfstests-bld/blob/master/README.md

Theodore Ts'o
  • 556
  • 2
  • 8
  • This simulates the error, however I was wondering how to catch this errors ? because `$?` still indicates its success. – ART Jul 27 '16 at 14:11
  • Basically I want to mount the partition `rw` however because mentioned error it gets mounted as `ro` and it doesn't give error and my scripts proceeds with other stuffs and eventually fails. I want to detect error in advanced and then exit or try `e2fsck`. One way i thought was to after mount try writing/touching file to test if its `rw` but I am not sure if there is better way to do that. – ART Jul 27 '16 at 14:31
  • I think one more way is just check errors using `e2fsck -n /dev/mmcblk0p15` but wondering why mount doesn't return error ? – ART Jul 27 '16 at 14:41
  • `e2fsck -n /dev/mmcblk0p15` doesn't return error when simulated file system error as you suggested :( Any suggestion ? – ART Aug 05 '16 at 16:18
  • e2fsck -n doesn't return an error because we didn't actually corrupt the file system. You asked how to simulate an aborted journal. The command sequence I gave you did that. It seems now you are asking a different question, which is how to simulate a corrupted file system. (Note that in real life you can have kernel-detected fs errors caused by bad memory, or a hiccup in the data xfer, and after you reboot e2fsck finds no problems.) You may want to be a lot more explicit about what you are trying to test, and why. – Theodore Ts'o Aug 08 '16 at 04:24
  • Thank you for reply, I didnt know that e2fsck author himself is answering my question, this the reason I love open source, where I get guidance from experts :) Now, I have written a shell script which does firmware upgrade for the board. I had seen error i mentioned in the question while testing. It gets resolved after reboot however fw upgrade will fail. And reboot is not nice. I made a fix i.e. check fs error in system backup partition before mounting and correct if it exist. 4 testing i need to simulate fs error and see if 'e2fsck -n' detects the error and if 'e2fsck -y' corrects it or not. – ART Aug 08 '16 at 05:04
  • There is no such thing as "a" fs error. It's like asking the question, how many differrent ways can DNA get mutated? Or how many ways can a human being get injured? So if I rephrased your request to, "I want to test to see whether or not a doctor can fix a human being's problem," does that help you understand how open-ended the problem is? – Theodore Ts'o Aug 09 '16 at 12:29
  • One issue is that in the original question, the "journal aborted" error simply tells you that there was a previous error that caused the journal to abort. That is a file system corruption, but rather a declaration of a condition. To use the medical analogy, that's like saying, "The patient is in shock." OK, was he in shock because of a gunshot wound? Or because of internal bleeding? If you had posted the output of e2fsck I could tell you how to corrupt the file system in the same way. – Theodore Ts'o Aug 09 '16 at 12:31
  • Also please note there are plenty of ways of corrupting the file system which e2fsck can't fix --- or which e2fsck can fix, but the resulting file system won't be all that useful without human intervention. For example: "debugfs -w -R 'clri <2>' /dev/hdXX". This will zap the root directory. e2fsck -fy will fix it, but you will see a new root directory and all of your top-level directories will end up in the lost+found directory with their inode number as the name of the directory. So e2fsck is not magic; there is no guarantee the result is something that your upgrade script can deal with – Theodore Ts'o Aug 09 '16 at 12:33
  • Thank you for reply Theodore, I get your point, I updated my question with more details. Basically I updated my script to correct file system errors using `e2fsck` and I need to test it for different errors possible in normal scenario. I want to verify if my changes works. Let me know if you suggest better approach for doing it. – ART Aug 09 '16 at 12:53
0

This is not a answer to this question, however adding more information

1) These steps show how to simulate file system error which can't be corrected by fsck -p command.

    #Create a test directory and a testfile
    mkdir /mnt/mntpt/testdir

    #create a test file
    echo "This is just a Test file" > "/mnt/mntpt/testfile"

    #Create a hard link
    ln "/mnt/mntpt/testfile" "/mnt/mntpt/testdir/hardlink"

    #Crea a soft link
    ln -sf "/mnt/mntpt/testfile" "/mnt/mntpt/testdir/softlink"

    #Now Run debugfs command to get inode number of the directory
    test_dir_inode="$(debugfs -R "ls -l" -c "/dev/mmcblk0p16" 2> /dev/null | grep "testdir" | awk '{print $1}')"

    #clear test directory inode
    debugfs -R "clri <${test_dir_inode}>" -w "/dev/mmcblk0p16"

    #umount partition
    umount "/mnt/mntpt"

    #mount partition
    mount /dev/mmcblk0p16 /mnt/mntpt

    #List the test directory as that will generate the file system error.
    ls -sF "${test_dir_path}"

    #unmount partition
    umount "/mnt/mntpt"

    #This error won't possible to correct using fsck -p command 
    fsck -p /dev/mmcblk0p16 

It will display below error

fsck from util-linux 2.21.2
firmware contains a file system with errors, check forced.
firmware: Entry 'testdir' in / (2) has deleted/unused inode 43.  CLEARED.
firmware: Inode 2 ref count is 4, should be 3.  FIXED.
firmware: Inode 20 ref count is 2, should be 1.  FIXED.
firmware: Unattached inode 24


firmware: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)


    #that error will be corrected by using fsck -y command as mentioned below
    fsck -y /dev/mmcblk0p16

2) I am looking for simulating error for which fsck -p command will correct errors but reboot is required to complete recovery. I think this happens only when partition under test is mounted read-only, I will check and update details as soon as I get more information

Okay it seems "Reboot required to complete recovery error i.e. for a command fsck -p message of will be printed like ***** REBOOT LINUX *****" is meant for current root file-system partition only. If current root-file-system partition is mounted ro fsck can correct file system errors but for that REBOOT is required. For other partitions it is not true.[This all I got to know by testing and some code reading of e2fsckpro code.]

Now I am
3) Looking for an error reproduction for which fsck -y would also fail to correct an error. (Will update details here, once I get that).

Here is an update

Obviously this operation is destructive and you will loose all your data. 

easiest way to create error which can't be corrected by `fsck -y` is as follows,  
dd if=/dev/urandom of=<partition for which error to be simulated> bs=<block size> count=<Partition size / block size>

so e.g. if I am performing test on /dev/mmcblk0p16 partition with 50MB size, command would be as follows, 

dd if=/dev/urandom of=/dev/mmcblk0p16 bs=1M count=50


To recover that partition we can use below command

mkfs.ext4 -j -L <Partition name> /dev/mmcblk0pN so sample command would be as follows,

mkfs.ext4 -j -L testpartition /dev/mmcblk0pN.
ART
  • 1,509
  • 3
  • 29
  • 47
  • @Theodore Ts'o : I did some homework :) do you have any suggestion/correction on this ? – ART Jan 18 '17 at 11:55