问题发现
突然发现机器重启后很多服务都出现了问题,不仅如此,还无法对文件进行操作:
>mkdir test mkdir: cannot create directory ‘test’: Read-only file system
排查问题
查看mount发现根目录被挂载为了read only:
>mount ....... /dev/mapper/vg0-root on / type ext4 (ro,relatime,stripe=256,data=ordered) .......
dmesg日志记录
筛选dmesg的结果可以看到,EXT4-FS出现了问题(dmesg | grep "EXT4-fs error"
):
[ 124.369135] systemd-readahead[724]: Failed to open pack file: Read-only file system [ 303.068201] EXT4-fs (dm-1): error count since last fsck: 9 [ 303.068274] EXT4-fs (dm-1): initial error at time 1591123830: ext4_find_entry:1318: inode 25430455 [ 303.068412] EXT4-fs (dm-1): last error at time 1591123878: ext4_wait_block_bitmap:516: inode 25430531 [ 549.490580] EXT4-fs (dm-1): Couldn't remount RDWR because of unprocessed orphan inode list. Please umount/remount instead [ 1050.216980] EXT4-fs (dm-1): Couldn't remount RDWR because of unprocessed orphan inode list. Please umount/remount instead [ 1138.023493] EXT4-fs (dm-1): Couldn't remount RDWR because of unprocessed orphan inode list. Please umount/remount instead [ 1179.341551] EXT4-fs (dm-1): Couldn't remount RDWR because of unprocessed orphan inode list. Please umount/remount instead
看来是磁盘出现了问题,使用smartctl查看是否有报错,但是没问题:
>smartctl --all /dev/sda ....... === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED ....... >smartctl --all /dev/sdb ....... === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED .......
那么很可能是最近服务器IO高负载导致崩溃,然后文件系统损坏了,需要进行修复。
开始修复
先尝试使用fsck来修复,报错版本太低:
>fsck.ext4 /dev/mapper/vg0-root e2fsck 1.42.9 (28-Dec-2013) /dev/vg0/root has unsupported feature(s): metadata_csum e2fsck: Get a newer version of e2fsck!
但是尝试升级却会因为read only报错,因此直接接入rescue模式
Rescue模式下挂载lvm
>ls /dev/mapper/* #查看lvm列表 mount /dev/mapper/vg0-root /mnt #挂载
修复磁盘
root@rescue / # fsck /dev/mapper/vg0-root fsck from util-linux 2.33.1 e2fsck 1.44.5 (15-Dec-2018) /dev/mapper/vg0-root contains a file system with errors, check forced. Pass 1: Checking inodes, blocks, and sizes Inode 4980739 extent tree (at level 2) could be narrower. Optimize? yes Inode 4980740 extent tree (at level 1) could be narrower. Optimize ? yes Inode 4980741 extent tree (at level 2) could be narrower. Optimize ? yes Inode 4980743 extent tree (at level 2) could be narrower. Optimize ? yes Inode 4980744 extent tree (at level 2) could be narrower. Optimize ? yes Inodes that were part of a corrupted orphan linked list found. Fix ? yes Inode 25756179 was part of the orphaned inode list. FIXED. Inode 25756180 was part of the orphaned inode list. FIXED. Inode 25756181 was part of the orphaned inode list. FIXED. Inode 25756182 was part of the orphaned inode list. FIXED. Inode 25756183 was part of the orphaned inode list. FIXED. Inode 25756184 was part of the orphaned inode list. FIXED. Inode 25756211 was part of the orphaned inode list. FIXED. Inode 25756233 was part of the orphaned inode list. FIXED. Inode 25756648 was part of the orphaned inode list. FIXED. Inode 25758032 was part of the orphaned inode list. FIXED. Inode 25758033 was part of the orphaned inode list. FIXED. Inode 25758034 was part of the orphaned inode list. FIXED. Inode 25758035 was part of the orphaned inode list. FIXED. Inode 26607626 was part of the orphaned inode list. FIXED. Inode 26607637 was part of the orphaned inode list. FIXED. Inode 26607638 was part of the orphaned inode list. FIXED. Inode 26607639 was part of the orphaned inode list. FIXED. Inode 26607640 was part of the orphaned inode list. FIXED. Deleted inode 177733645 has zero dtime. Fix ? yes Inode 177733646 was part of the orphaned inode list. FIXED. Inode 177733647 was part of the orphaned inode list. FIXED. Inode 177733651 was part of the orphaned inode list. FIXED. Inode 177733656 was part of the orphaned inode list. FIXED. Deleted inode 177733664 has zero dtime. Fix ? yes Inode 177733665 was part of the orphaned inode list. FIXED. Inode 177733666 was part of the orphaned inode list. FIXED. Inode 177733667 was part of the orphaned inode list. FIXED. Inode 177733668 was part of the orphaned inode list. FIXED. Pass 1E: Optimizing extent trees Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Inode bitmap differences: -(25756179--25756184) -25756211 -25756233 -25756648 -(25758032--25758035) -26607626 -(26607637--26607640) -(177733645--177733647) -177733651 -177733656 -(177733664--177733668) Fix ? yes Free inodes count wrong for group #6288 (1541, counted=1554). Fix ? yes Free inodes count wrong for group #6496 (4070, counted=4075). Fix ? yes Free inodes count wrong for group #43392 (4059, counted=4069). Fix ? yes Free inodes count wrong (182861554, counted=182861582). Fix ? yes /dev/mapper/vg0-root: ***** FILE SYSTEM WAS MODIFIED ***** /dev/mapper/vg0-root: 110834/182972416 files (0.3% non-contiguous), 860247893/1463754752 blocks
确认修复
root@rescue / # fsck /dev/mapper/vg0-root fsck from util-linux 2.33.1 e2fsck 1.44.5 (15-Dec-2018) /dev/mapper/vg0-root: clean, 110834/182972416 files, 860247893/1463754752 blocks
文章来源于互联网:使用e2fsck修复损坏的Linux磁盘|Read only|LVM|EXT4-FS
相关推荐: 小蜜蜂微信公众号文章助手(BeePress)2.2.1
小蜜蜂微信公众号文章助手(BeePress)2.2.1 来源:小蜜蜂微信公众号日期:2020/01/14 16:11 浏览: 支持系统(当前版本2.2.1) 支持双平台:macOS、Windows 最近微信手机安卓客户端改版了(7.0.5), 没有以往的历…
© 版权声明
文章版权归作者所有,未经允许请勿转载。
THE END
喜欢就支持一下吧
请登录后发表评论
注册