Kernel panic reading - Can you tell what triggered it?

William Muriithi william.muriithi-Re5JQEeQqe8AvxtiuMwx3w at public.gmane.org
Sat Feb 11 23:07:18 UTC 2012


Hello,

I am sharing this as I am not sure what triggered it and therefore
have not figured how to fix it.  This is the third time the system has
crashed and the system is just just running vertica.  Vertica claim
they do not have any kernel level module - I feel like they are not
truthful but have no evidence to prove them wrong.  If they are
telling the truth, then RHEL6 does have a bug would like to post on
RedHat bugzilla

The most interesting part:-

BUG: Bad page state in process vertica  pfn:6db6d9adbf6d47af
page:ffff8801dfe7ae48 flags:000000000000001f count:36561488
mapcount:-5631 mapping:ffff8801dfe7ae60 index:ffff8801dfe7ae60
(Tainted: G        W  ----------------  )
Pid: 682, comm: vertica Tainted: G        W  ----------------
2.6.32-220.el6.x86_64 #1
Call Trace:
 [<ffffffff811212f7>] ? bad_page+0x107/0x160


I read this and concluded that vertica is messing around with the
kernel.  Vertica pushed back and said its a xen problem, something I
am finding hard to figure out how they arrived to that decision.  Am I
wrong to assume they are on drugs?  What else can taint a kernel other
than a kernel module?

The odd part though is "cat /proc/sys/kernel/tainted" return zero, yet
the system think its tainted when going down.

Look like the iptables module also have issues, but I doubt RedHat
would would look at it as the kernel claim to be tainted.



Full logs below.



Microcode Update Driver: v2.00 <tigran-ppwZ4lME3+KI6QP4U9MhSdBc4/FLrbF6 at public.gmane.org>, Peter Oruba
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
ip6_tables: (C) 2000-2006 Netfilter Core Team
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
ip_tables: (C) 2000-2006 Netfilter Core Team

Red Hat Enterprise Linux Server release 6.2 (Santiago)
Kernel 2.6.32-220.el6.x86_64 on an x86_64

dev.bigdatalabs login: ip_tables: (C) 2000-2006 Netfilter Core Team
------------[ cut here ]------------
WARNING: at lib/list_debug.c:48 list_del+0x6e/0xa0() (Not tainted)
list_del corruption. prev->next should be ffffea00027543d0, but was
ffffea0002724588
Modules linked in: nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter
ip_tables autofs4 ipt_REJECT ip6t_REJECT nf_conntrack_ipv6
nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6
microcode xen_netfront ext4 mbcache jbd2 xen_blkfront dm_mirror
dm_region_hash dm_log dm_mod [last unloaded: nf_defrag_ipv4]
Pid: 768, comm: vertica Not tainted 2.6.32-220.el6.x86_64 #1
Call Trace:
 [<ffffffff81069b77>] ? warn_slowpath_common+0x87/0xc0
 [<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
 [<ffffffff81069c66>] ? warn_slowpath_fmt+0x46/0x50
 [<ffffffffa00481b4>] ? do_get_write_access+0x3b4/0x520 [jbd2]
 [<ffffffff8127b7ae>] ? list_del+0x6e/0xa0
 [<ffffffff8112343c>] ? free_pcppages_bulk+0x15c/0x390
 [<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
 [<ffffffff81007ca2>] ? check_events+0x12/0x20
 [<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81124378>] ? free_hot_cold_page+0x1b8/0x220
 [<ffffffff81124422>] ? __pagevec_free+0x42/0x90
 [<ffffffff81127afc>] ? release_pages+0x21c/0x250
 [<ffffffff81127e77>] ? ____pagevec_lru_add+0x167/0x180
 [<ffffffff81128026>] ? __pagevec_release+0x26/0x40
 [<ffffffff8112884b>] ? truncate_inode_pages_range+0x1fb/0x460
 [<ffffffff8113a5e2>] ? unmap_mapping_range+0x72/0x140
 [<ffffffff81128ac5>] ? truncate_inode_pages+0x15/0x20
 [<ffffffff81128b17>] ? truncate_pagecache+0x47/0x70
 [<ffffffff81128b59>] ? truncate_setsize+0x19/0x20
 [<ffffffff81128b9e>] ? vmtruncate+0x3e/0x70
 [<ffffffff811922b0>] ? inode_setattr+0x30/0x60
 [<ffffffffa007c82c>] ? ext4_setattr+0x10c/0x360 [ext4]
 [<ffffffff81192698>] ? notify_change+0x168/0x340
 [<ffffffff8118eab7>] ? __d_lookup+0xa7/0x150
 [<ffffffff81174de4>] ? do_truncate+0x64/0xa0
 [<ffffffff8120d52f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff811876e9>] ? do_filp_open+0x829/0xd60
 [<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
 [<ffffffff811935e2>] ? alloc_fd+0x92/0x160
 [<ffffffff81173ba9>] ? do_sys_open+0x69/0x140
 [<ffffffff81173cc0>] ? sys_open+0x20/0x30
 [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b
---[ end trace 2de2020846513d27 ]---
BUG: Bad page state in process vertica  pfn:6db6d9adbf6d47af
page:ffff8801dfe7ae48 flags:000000000000001f count:36561488
mapcount:-5631 mapping:ffff8801dfe7ae60 index:ffff8801dfe7ae60
(Tainted: G        W  ----------------  )
Pid: 682, comm: vertica Tainted: G        W  ----------------
2.6.32-220.el6.x86_64 #1
Call Trace:
 [<ffffffff811212f7>] ? bad_page+0x107/0x160
 [<ffffffff811226b4>] ? get_page_from_freelist+0x724/0x820
 [<ffffffff810046b6>] ? xen_mc_flush+0x106/0x250
 [<ffffffff811238a1>] ? __alloc_pages_nodemask+0x111/0x940
 [<ffffffff81010b4e>] ? __copy_from_user_inatomic+0xe/0x20
 [<ffffffff810a22bc>] ? get_futex_value_locked+0x2c/0x50
 [<ffffffff810a32d1>] ? futex_wait_setup+0x121/0x140
 [<ffffffff81007ca2>] ? check_events+0x12/0x20
 [<ffffffff81158c7a>] ? alloc_pages_vma+0x9a/0x150
 [<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8113beeb>] ? handle_pte_fault+0x76b/0xb50
 [<ffffffff810a2cce>] ? futex_wake+0x10e/0x120
 [<ffffffff81004a49>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
 [<ffffffff8113c4b4>] ? handle_mm_fault+0x1e4/0x2b0
 [<ffffffff810a4c10>] ? do_futex+0x100/0xb00
 [<ffffffff81042b39>] ? __do_page_fault+0x139/0x480
 [<ffffffff81007ca2>] ? check_events+0x12/0x20
 [<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff814ef38c>] ? _spin_unlock_irqrestore+0x1c/0x20
 [<ffffffff81038448>] ? pvclock_clocksource_read+0x58/0xd0
 [<ffffffff814f248e>] ? do_page_fault+0x3e/0xa0
 [<ffffffff814ef845>] ? page_fault+0x25/0x30
Disabling lock debugging due to kernel taint
general protection fault: 0000 [#1] SMP
last sysfs file: /sys/module/nf_conntrack/initstate
CPU 0
Modules linked in: nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter
ip_tables autofs4 ipt_REJECT ip6t_REJECT nf_conntrack_ipv6
nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6
microcode xen_netfront ext4 mbcache jbd2 xen_blkfront dm_mirror
dm_region_hash dm_log dm_mod [last unloaded: nf_defrag_ipv4]

Pid: 910, comm: vertica Tainted: G    B   W  ----------------
2.6.32-220.el6.x86_64 #1
RIP: e030:[<ffffffff8127b74c>]  [<ffffffff8127b74c>] list_del+0xc/0xa0
RSP: e02b:ffff88009f6b9a28  EFLAGS: 00010096
RAX: 0000000000000200 RBX: dead000000100100 RCX: 0000000000012b20
RDX: 0000000000000030 RSI: ffff8801dfe7ae70 RDI: dead000000100100
RBP: ffff88009f6b9a38 R08: 0000000000000002 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801dfe7ae40
R13: ffff8800000106c0 R14: 000000000000073a R15: dead0000001000d8
FS:  00007f5c92a65700(0000) GS:ffff88002804f000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f5d19d4c500 CR3: 00000001d74ea000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vertica (pid: 910, threadinfo ffff88009f6b8000, task ffff8801d9b9e040)
Stack:
 ffff88009f6b9a38 0000000000000001 ffff88009f6b9b58 ffffffff81122218
<0> 0000000000000180 0000000000000180 ffffffff00000002 ffffffff810046b6
<0> 000000000002be6e 00000040ffffffff 0000000000000000 ffff880000029b08
Call Trace:
 [<ffffffff81122218>] get_page_from_freelist+0x288/0x820
 [<ffffffff810046b6>] ? xen_mc_flush+0x106/0x250
 [<ffffffff811238a1>] __alloc_pages_nodemask+0x111/0x940
 [<ffffffff81062375>] ? enqueue_entity+0x125/0x410
------------[ cut here ]------------
WARNING: at lib/list_debug.c:51 list_del+0x8d/0xa0() (Tainted: G    B
 W  ----------------  )
list_del corruption. next->prev should be ffffea00005faac0, but was
ffffea000204b060
Modules linked in: nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter
ip_tables autofs4 ipt_REJECT ip6t_REJECT nf_conntrack_ipv6
nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6
microcode xen_netfront ext4 mbcache jbd2 xen_blkfront dm_mirror
dm_region_hash dm_log dm_mod [last unloaded: nf_defrag_ipv4]
Pid: 25265, comm: rs:main Q:Reg Tainted: G    B   W  ----------------
 2.6.32-220.el6.x86_64 #1
Call Trace:
 [<ffffffff81069b77>] ? warn_slowpath_common+0x87/0xc0
 [<ffffffff81069c66>] ? warn_slowpath_fmt+0x46/0x50
 [<ffffffff81007ca2>] ? check_events+0x12/0x20
 [<ffffffff8127b7cd>] ? list_del+0x8d/0xa0
 [<ffffffff81120823>] ? __rmqueue+0xc3/0x490
 [<ffffffff81007ca2>] ? check_events+0x12/0x20
 [<ffffffff81122528>] ? get_page_from_freelist+0x598/0x820
 [<ffffffff811238a1>] ? __alloc_pages_nodemask+0x111/0x940
 [<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8115fbf4>] ? kmem_cache_free+0xc4/0x2b0
 [<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff814ef38c>] ? _spin_unlock_irqrestore+0x1c/0x20
 [<ffffffff81051813>] ? __wake_up+0x53/0x70
 [<ffffffff81158b7a>] ? alloc_pages_current+0xaa/0x110
 [<ffffffff81110e57>] ? __page_cache_alloc+0x87/0x90
 [<ffffffff81126a7b>] ? __do_page_cache_readahead+0xdb/0x210
 [<ffffffff81126bd1>] ? ra_submit+0x21/0x30
 [<ffffffff81112123>] ? filemap_fault+0x4c3/0x500
 [<ffffffff8113b2c4>] ? __do_fault+0x54/0x510
 [<ffffffff81112d60>] ? __generic_file_aio_write+0x250/0x480
 [<ffffffff8113b877>] ? handle_pte_fault+0xf7/0xb50
 [<ffffffff8111304e>] ? generic_file_aio_write+0xbe/0xe0
 [<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
 [<ffffffff81007ca2>] ? check_events+0x12/0x20
 [<ffffffff81004a49>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
 [<ffffffff8113c4b4>] ? handle_mm_fault+0x1e4/0x2b0
 [<ffffffff81042b39>] ? __do_page_fault+0x139/0x480
 [<ffffffff811b50c9>] ? fsnotify_put_event+0x49/0x70
 [<ffffffff81038448>] ? pvclock_clocksource_read+0x58/0xd0
 [<ffffffff81007b21>] ? xen_clocksource_read+0x21/0x30
 [<ffffffff81007c09>] ? xen_clocksource_get_cycles+0x9/0x10
 [<ffffffff8109b470>] ? getnstimeofday+0x60/0xf0
 [<ffffffff814f248e>] ? do_page_fault+0x3e/0xa0
 [<ffffffff814ef845>] ? page_fault+0x25/0x30
---[ end trace 2de2020846513d28 ]---
 [<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
 [<ffffffff81007ca2>] ? check_events+0x12/0x20
 [<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81158c7a>] alloc_pages_vma+0x9a/0x150
 [<ffffffff8113beeb>] handle_pte_fault+0x76b/0xb50
 [<ffffffff810a2490>] ? wake_futex+0x40/0x60
 [<ffffffff810a4590>] ? futex_requeue+0x310/0x890
 [<ffffffff810a2cce>] ? futex_wake+0x10e/0x120
 [<ffffffff81004a49>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
 [<ffffffff8113c4b4>] handle_mm_fault+0x1e4/0x2b0
 [<ffffffff810a4c10>] ? do_futex+0x100/0xb00
 [<ffffffff81042b39>] __do_page_fault+0x139/0x480
 [<ffffffff81007ca2>] ? check_events+0x12/0x20
 [<ffffffff81090280>] ? register_posix_clock+0x50/0xa0
 [<ffffffff81090280>] ? register_posix_clock+0x50/0xa0
 [<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff814ef38c>] ? _spin_unlock_irqrestore+0x1c/0x20
 [<ffffffff8105ca46>] ? task_sched_runtime+0x46/0xa0
 [<ffffffff81090280>] ? register_posix_clock+0x50/0xa0
 [<ffffffff81092478>] ? sample_to_timespec+0x38/0x50
 [<ffffffff81092c0c>] ? cpu_clock_sample+0x4c/0x70
 [<ffffffff814f248e>] do_page_fault+0x3e/0xa0
 [<ffffffff814ef845>] page_fault+0x25/0x30
Code: 00 ff ff ff 89 95 fc fe ff ff e9 ab fd ff ff 4c 8b ad e8 fe ff
ff e9 db fd ff ff 90 90 90 90 55 48 89 e5 53 48 89 fb 48 83 ec 08 <48>
8b 47 08 4c 8b 00 4c 39 c7 75 39 48 8b 03 4c 8b 40 08 4c 39
RIP  [<ffffffff8127b74c>] list_del+0xc/0xa0
 RSP <ffff88009f6b9a28>
---[ end trace 2de2020846513d29 ]---
Kernel panic - not syncing: Fatal exception
Pid: 910, comm: vertica Tainted: G    B D W  ----------------
2.6.32-220.el6.x86_64 #1
Call Trace:
 [<ffffffff814ec341>] ? panic+0x78/0x143
 [<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff814ef38c>] ? _spin_unlock_irqrestore+0x1c/0x20
 [<ffffffff814f04d4>] ? oops_end+0xe4/0x100
 [<ffffffff8100f26b>] ? die+0x5b/0x90
 [<ffffffff814f0042>] ? do_general_protection+0x152/0x160
 [<ffffffff814ef815>] ? general_protection+0x25/0x30
 [<ffffffff8127b74c>] ? list_del+0xc/0xa0
 [<ffffffff81122218>] ? get_page_from_freelist+0x288/0x820
 [<ffffffff810046b6>] ? xen_mc_flush+0x106/0x250
 [<ffffffff811238a1>] ? __alloc_pages_nodemask+0x111/0x940
 [<ffffffff81062375>] ? enqueue_entity+0x125/0x410
 [<ffffffff810074fd>] ? xen_force_evtchn_callback+0xd/0x10
 [<ffffffff81007ca2>] ? check_events+0x12/0x20
 [<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81158c7a>] ? alloc_pages_vma+0x9a/0x150
 [<ffffffff8113beeb>] ? handle_pte_fault+0x76b/0xb50
 [<ffffffff810a2490>] ? wake_futex+0x40/0x60
 [<ffffffff810a4590>] ? futex_requeue+0x310/0x890
 [<ffffffff810a2cce>] ? futex_wake+0x10e/0x120
 [<ffffffff81004a49>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
 [<ffffffff8113c4b4>] ? handle_mm_fault+0x1e4/0x2b0
 [<ffffffff810a4c10>] ? do_futex+0x100/0xb00
 [<ffffffff81042b39>] ? __do_page_fault+0x139/0x480
 [<ffffffff81007ca2>] ? check_events+0x12/0x20
 [<ffffffff81090280>] ? register_posix_clock+0x50/0xa0
 [<ffffffff81090280>] ? register_posix_clock+0x50/0xa0
 [<ffffffff81007c8f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff814ef38c>] ? _spin_unlock_irqrestore+0x1c/0x20
 [<ffffffff8105ca46>] ? task_sched_runtime+0x46/0xa0
 [<ffffffff81090280>] ? register_posix_clock+0x50/0xa0
 [<ffffffff81092478>] ? sample_to_timespec+0x38/0x50
 [<ffffffff81092c0c>] ? cpu_clock_sample+0x4c/0x70
 [<ffffffff814f248e>] ? do_page_fault+0x3e/0xa0
 [<ffffffff814ef845>] ? page_fault+0x25/0x30
Xen Minimal OS!
  start_info: 0x1890000(VA)
    nr_pages: 0x1e0000
  shared_inf: 0xbf712000(MA)
     pt_base: 0x1893000(VA)
nr_pt_frames: 0x11
    mfn_list: 0x990000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: root=/dev/sda1 ro 4
  stack:      0x94f860-0x96f860
MM: Init
      _text: 0x0(VA)
     _etext: 0x5ff6d(VA)
   _erodata: 0x78000(VA)
     _edata: 0x80b00(VA)
stack start: 0x94f860(VA)
       _end: 0x98fe68(VA)
  start_pfn: 18a7
    max_pfn: 1e0000
Mapping memory range 0x1c00000 - 0x1e0000000
setting 0x0-0x78000 readonly
skipped 0x1000
MM: Initialise page allocator for 27a0000(27a0000)-1e0000000(1e0000000)
MM: done
Demand map pfns at 1e0001000-21e0001000.
Heap resides at 21e0002000-41e0002000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x1e0001000.
Initialising scheduler
Thread "Idle": pointer: 0x21e0002010, stack: 0x36f0000
Initialising xenbus
Thread "xenstore": pointer: 0x21e00027c0, stack: 0x3700000
Dummy main: start_info=0x96f960
Thread "main": pointer: 0x21e0002f70, stack: 0x3710000
"main" "root=/dev/sda1" "ro" "4"
vbd 2049 is hd0
--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists





More information about the Legacy mailing list