serious xfs (file ystem) issues

Fri Feb 20 00:50:32 UTC 2009

Can it be that the xfs module is corrupt? The other possibility is that the file system is corrupt which the filesystem check does not detect or fix.

I would try the following
1.   shut down mysql and try to read and write large directory/files on the xfs file system (assuming you got ample disk space) to make sure the filesystem is all fine. If xfs give you the similar problem then either you got to recreate the filesystem or reinstall the machine.

2. remove the mysql files (they may be in /var/mysql) and restoring them from a backup. (this might work if the problem is with mysql files which might probably force mysql to request file reading outside of xfs file allocations).

Last resort is, of course, to start from a clean slate and restore the database from backups, upgrading the system on the way.

Hope that helps.

EK

--- On Thu, 2/19/09, William Muriithi <william.muriithi-Re5JQEeQqe8AvxtiuMwx3w at public.gmane.org> wrote:

> From: William Muriithi <william.muriithi-Re5JQEeQqe8AvxtiuMwx3w at public.gmane.org>
> Subject: [TLUG]: serious xfs (file ystem) issues
> To: "TLUG mailing list" <tlug-lxSQFCZeNF4 at public.gmane.org>
> Received: Thursday, February 19, 2009, 1:37 PM
> Hi pals
> 
> Got this problem and I am just seeking help here in case
> someone has a quick
> solution that can get us out of the problem.
> 
> We have a mysql database that run on xfs. About noon today,
> the database
> died and a quick digging pointed to an issue with the file
> system. We have
> tried file system check, but the restarting the database
> causes the problem
> to reappear again. Now, I am not sure whether this is a bug
> in xfs,  dmks or
> data, and thats where I am seeking some assistance.
> 
> Here is the tech details
> - Linux database 2.6.18-92.1.10.el5 #1 SMP Tue Aug 5
> 07:42:41 EDT 2008
> x86_64 x86_64 x86_64 GNU/Linux
> - kmod-xfs-0.4-1.2.6.18_92.1.10.el5
> - xfsprogs-2.9.4-1.el5.centos
> 
> Below are the logs. What would the most advisable way to go
> about it if it
> were in you in facing this problem:
> 
> Thannks for any assistance:
> 
> Regards,
> 
> William
> 
> Feb 19 12:08:22 database kernel: Filesystem
> "dm-0": XFS internal error
> xfs_alloc_read_agf at line 2190 of file
> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_alloc.c.
>  Caller
> 0xffffffff8858d0f4
> Feb 19 12:08:22 database kernel: 
> [<ffffffff885af0ec>]
> :xfs:xfs_corruption_error+0xe4/0xf6
> Feb 19 12:08:22 database kernel: 
> [<ffffffff885d1ef5>]
> :xfs:kmem_zone_zalloc+0x1e/0x2f
> Feb 19 12:08:22 database kernel: 
> [<ffffffff885a3287>]
> :xfs:xfs_buf_item_init+0x56/0xc8
> Feb 19 12:08:22 database kernel: 
> [<ffffffff885c8cd4>]
> :xfs:xfs_trans_read_buf+0x27a/0x2c7
> Feb 19 12:08:22 database kernel: 
> [<ffffffff8858b675>]
> :xfs:xfs_alloc_read_agf+0x110/0x189
> Feb 19 12:08:22 database kernel: 
> [<ffffffff8858d0f4>]
> :xfs:xfs_alloc_fix_freelist+0x45/0x413
> Feb 19 12:08:22 database kernel: 
> [<ffffffff8858d0f4>]
> :xfs:xfs_alloc_fix_freelist+0x45/0x413
> Feb 19 12:08:22 database kernel: 
> [<ffffffff8858d54a>]
> :xfs:xfs_free_extent+0x88/0xc9
> Feb 19 12:08:22 database kernel: 
> [<ffffffff885c20e5>]
> :xfs:xlog_recover_finish+0x15a/0x244
> Feb 19 12:08:22 database kernel: 
> [<ffffffff885c5b00>]
> :xfs:xfs_mountfs+0xa24/0xc30
> Feb 19 12:08:22 database kernel: 
> [<ffffffff885cb748>]
> :xfs:xfs_mount+0x762/0x83b
> Feb 19 12:08:22 database kernel: 
> [<ffffffff885dac79>]
> :xfs:xfs_fs_fill_super+0x0/0x1e3
> Feb 19 12:08:22 database kernel: 
> [<ffffffff885dacf7>]
> :xfs:xfs_fs_fill_super+0x7e/0x1e3
> Feb 19 12:24:35 database kernel: Filesystem
> "dm-0": XFS internal error
> xfs_alloc_read_agf at line 2190 of file
> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_alloc.c.
>  Caller
> 0xffffffff8858d0f4
> Feb 19 12:24:35 database kernel: 
> [<ffffffff885af0ec>]
> :xfs:xfs_corruption_error+0xe4/0xf6
> Feb 19 12:24:35 database kernel: 
> [<ffffffff885d1ef5>]
> :xfs:kmem_zone_zalloc+0x1e/0x2f
> Feb 19 12:24:35 database kernel: 
> [<ffffffff885a3287>]
> :xfs:xfs_buf_item_init+0x56/0xc8
> Feb 19 12:24:35 database kernel: 
> [<ffffffff885c8cd4>]
> :xfs:xfs_trans_read_buf+0x27a/0x2c7
> Feb 19 12:24:35 database kernel: 
> [<ffffffff8858b675>]
> :xfs:xfs_alloc_read_agf+0x110/0x189
> Feb 19 12:24:35 database kernel: 
> [<ffffffff8858d0f4>]
> :xfs:xfs_alloc_fix_freelist+0x45/0x413
> Feb 19 12:24:35 database kernel: 
> [<ffffffff8858d0f4>]
> :xfs:xfs_alloc_fix_freelist+0x45/0x413
> Feb 19 12:24:35 database kernel: 
> [<ffffffff8858d54a>]
> :xfs:xfs_free_extent+0x88/0xc9
> Feb 19 12:24:35 database kernel: 
> [<ffffffff8859a2ad>]
> :xfs:xfs_bmap_finish+0xf0/0x169
> Feb 19 12:24:35 database kernel: 
> [<ffffffff885b7e46>]
> :xfs:xfs_itruncate_finish+0x172/0x2b3
> Feb 19 12:24:35 database kernel: 
> [<ffffffff885cc479>]
> :xfs:xfs_inactive_free_eofblocks+0x190/0x1d6
> Feb 19 12:24:35 database kernel: 
> [<ffffffff885cf5b3>]
> :xfs:xfs_release+0x94/0xc5
> Feb 19 12:24:35 database kernel: 
> [<ffffffff885d5f3d>]
> :xfs:xfs_file_release+0x1a/0x1e
> Feb 19 12:24:35 database kernel:
> xfs_force_shutdown(dm-0,0x8) called from
> line 4267 of file
> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_bmap.c.
>  Return
> address = 0xffffffff8859a2ea
> Feb 19 12:26:55 database kernel:
> xfs_force_shutdown(dm-0,0x1) called from
> line 424 of file
> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_rw.c.
>  Return
> address = 0xffffffff885d17f1
> Feb 19 12:26:55 database kernel:
> xfs_force_shutdown(dm-0,0x1) called from
> line 424 of file
> /home/buildsvn/rpmbuild/BUILD/xfs-kmod-0.4/_kmod_build_/xfs_rw.c.
>  Return
> address = 0xffffffff885d17f1
> 
> 
> -- 
> For a list of all the ways technology has failed to improve
> the quality of
> life, please press three.
> —Alice Kahn

      __________________________________________________________________
Instant Messaging, free SMS, sharing photos and more... Try the new Yahoo! Canada Messenger at http://ca.beta.messenger.yahoo.com/
--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists