Xfs › Xfs - General

XFS hung on 2.6.33.3 kernel

classic Classic list List threaded Threaded
♦
♦
 20 messages Options
Ilia Mirkin-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

XFS hung on 2.6.33.3 kernel

Ilia Mirkin-2
Hi XFS developers,

I ended up getting this on a 2.6.33.3 kernel. I realize this is not
the latest, but I've only gotten this once, so it will not be possible
to easily determine whether this is gone in the latest release or not.

The details of the situation is that there is an XFS filesystem on a
hardware RAID10 array (directly on the device, no partition table),
and it exclusively houses a mysql (5.0) database. The database gets
backed up nightly, which is basically a glorified tar'ing of most of
the partition. The non-mysql process you see there are related to a
mysql backup program (which coincidentally triggers WARN's in xfs
code, as per http://marc.info/?l=linux-xfs&m=127632333825584&w=2;
however it sounded like those WARN's were not warning about an
in-kernel problems; curiously, the last WARN's in the kernel log
appear to have been from the prior day's backup run, not the one that
triggered the hang). There's also a sync that I ran as a last-ditch
effort at saving the system. On reboot, everything came back up OK.

[

For reference, the warns look like this:

[5444728.784257] ------------[ cut here ]------------
[5444728.784269] WARNING: at fs/xfs/linux-2.6/xfs_lrw.c:651
xfs_write+0x3dc/0x784()
[5444728.784272] Hardware name: PowerEdge R710
[5444728.784275] Modules linked in: nfsd cifs iTCO_wdt iTCO_vendor_support
[5444728.784286] Pid: 7765, comm: mysqld Tainted: G        W
2.6.33-gentoo-r2 #1
[5444728.784287] Call Trace:
[5444728.784291]  [<ffffffff8120da46>] ? xfs_write+0x3dc/0x784
[5444728.784295]  [<ffffffff81038733>] warn_slowpath_common+0x77/0xa4
[5444728.784297]  [<ffffffff8103876f>] warn_slowpath_null+0xf/0x11
[5444728.784299]  [<ffffffff8120da46>] xfs_write+0x3dc/0x784
[5444728.784303]  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
[5444728.784306]  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
[5444728.784310]  [<ffffffff81032879>] ? finish_task_switch+0x43/0xac
[5444728.784312]  [<ffffffff810d8624>] vfs_write+0xab/0x105
[5444728.784314]  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
[5444728.784319]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[5444728.784321] ---[ end trace 8b0c2a6e5e867a01 ]---

]

While the system was stuck, vmstat did not report any block-in/out on
the device. I have rrd's of those if necessary. Let me know if I might
be able to provide any other useful info, or if you think that this
sort of thing should be fixed in the latest linux kernel release. I
saw that there have been a lot of XFS updates since 2.6.33, but none
stood out as addressing this type of situation.

Here is the output of "echo w > /proc/sysrq-trigger":

 SysRq : Show Blocked State
   task                        PC stack   pid father
 xfsdatad/7    D ffff88123885d3e0  6792   772      2 0x00000000
  ffff88123887fd90 0000000000000046 0000000000000000 ffff88123887fd5c
  0000000000000000 000000000000de78 ffff88123887ffd8 0000000000013c00
  ffff88123887ffd8 0000000000013c00 ffff88123885d3e0 ffff881239d23d00
 Call Trace:
  [<ffffffff81045f9c>] ? __mod_timer+0x13d/0x14f
  [<ffffffff814eb81c>] schedule_timeout+0x98/0xbf
  [<ffffffff81045b27>] ? process_timeout+0x0/0xb
  [<ffffffff814eb85c>] schedule_timeout_uninterruptible+0x19/0x1b
  [<ffffffff81206512>] xfs_end_io+0x7d/0x8c
  [<ffffffff8104de65>] worker_thread+0x18d/0x222
  [<ffffffff81206495>] ? xfs_end_io+0x0/0x8c
  [<ffffffff810517ab>] ? autoremove_wake_function+0x0/0x38
  [<ffffffff8104dcd8>] ? worker_thread+0x0/0x222
  [<ffffffff81051336>] kthread+0x7d/0x85
  [<ffffffff81003814>] kernel_thread_helper+0x4/0x10
  [<ffffffff810512b9>] ? kthread+0x0/0x85
  [<ffffffff81003810>] ? kernel_thread_helper+0x0/0x10
 mysqld        D ffff880938821380  4696  7750      1 0x00000000
  ffff880650e01cb8 0000000000000082 00715f4f00000001 0000000000000000
  ffff88090a0bd3e0 000000000000de78 ffff880650e01fd8 0000000000013c00
  ffff880650e01fd8 0000000000013c00 ffff88090a0bd3e0 ffff88090a0bf260
 Call Trace:
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff8102832b>] ? pick_next_task+0x22/0x3e
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D ffff880938821380  4760  7751      1 0x00000000
  ffff880650e6bcb8 0000000000000082 007cd91100000001 ffff88090a0bcc78
  ffff88095fc13c00 000000000000de78 ffff880650e6bfd8 0000000000013c00
  ffff880650e6bfd8 0000000000013c00 ffff88090a0bf260 ffff88090a0b8000
 Call Trace:
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D ffff880938821380  4768  7752      1 0x00000000
  ffff880650e6dcb8 0000000000000082 0085722500000001 ffff88090a0b8038
  ffff880028273c00 000000000000de78 ffff880650e6dfd8 0000000000013c00
  ffff880650e6dfd8 0000000000013c00 ffff88090a0bcc40 ffff88090a0b96e0
 Call Trace:
  [<ffffffff81027beb>] ? activate_task+0x28/0x31
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff81032879>] ? finish_task_switch+0x43/0xac
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4768  7755      1 0x00000000
  ffff88029a60fcb8 0000000000000082 0000000000000000 ffff88090a0b9718
  ffff88095fc13c00 000000000000de78 ffff88029a60ffd8 0000000000013c00
  ffff88029a60ffd8 0000000000013c00 ffff88090a0b8000 ffff881239d23d00
 Call Trace:
  [<ffffffff81027beb>] ? activate_task+0x28/0x31
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff8102832b>] ? pick_next_task+0x22/0x3e
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D ffff880938821380  4696  7756      1 0x00000000
  ffff88029a633cb8 0000000000000082 0091c4f90a0b9718 ffff88090a0bf298
  ffff880028273c00 000000000000de78 ffff88029a633fd8 0000000000013c00
  ffff88029a633fd8 0000000000013c00 ffff88090a0be320 ffff8812384c8000
 Call Trace:
  [<ffffffff81027beb>] ? activate_task+0x28/0x31
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff8102832b>] ? pick_next_task+0x22/0x3e
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4696  7757      1 0x00000000
  ffff8801b4fdfcb8 0000000000000082 0000000000000000 ffff88090a0beaf8
  ffff88095fc33c00 000000000000de78 ffff8801b4fdffd8 0000000000013c00
  ffff8801b4fdffd8 0000000000013c00 ffff88090a0b9e80 ffff880939c9adc0
 Call Trace:
  [<ffffffff81027beb>] ? activate_task+0x28/0x31
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff8102832b>] ? pick_next_task+0x22/0x3e
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4648  7758      1 0x00000000
  ffff8801b4f1bc08 0000000000000082 0000000000000000 ffffffff8102cbe0
  ffff8801b4f1bb88 000000000000de78 ffff8801b4f1bfd8 0000000000013c00
  ffff8801b4f1bfd8 0000000000013c00 ffff88090a0bb560 ffff880939ce8000
 Call Trace:
  [<ffffffff8102cbe0>] ? check_preempt_wakeup+0xde/0x15d
  [<ffffffff814ec6ac>] ? rwsem_down_failed_common+0x96/0xc8
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120d922>] xfs_write+0x2b8/0x784
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D ffff880938821380  4648  7759      1 0x00000000
  ffff88029a627c08 0000000000000082 000000009a627b88 ffffffff8102cbe0
  ffff88029a627b88 000000000000de78 ffff88029a627fd8 0000000000013c00
  ffff88029a627fd8 0000000000013c00 ffff88090a0ba620 ffff8812116787a0
 Call Trace:
  [<ffffffff8102cbe0>] ? check_preempt_wakeup+0xde/0x15d
  [<ffffffff814ec6ac>] ? rwsem_down_failed_common+0x96/0xc8
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120d922>] xfs_write+0x2b8/0x784
  [<ffffffff81034825>] ? wake_up_state+0xb/0xd
  [<ffffffff814ecb8e>] ? common_interrupt+0xe/0x13
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff810ff862>] ? __fsnotify_parent+0xc7/0xd3
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4648  7760      1 0x00000000
  ffff88029a611c08 0000000000000082 0000000000000000 ffffffff8102cbe0
  ffff88029a611b88 000000000000de78 ffff88029a611fd8 0000000000013c00
  ffff88029a611fd8 0000000000013c00 ffff88090a0badc0 ffff880939ce8000
 Call Trace:
  [<ffffffff8102cbe0>] ? check_preempt_wakeup+0xde/0x15d
  [<ffffffff814ec6ac>] ? rwsem_down_failed_common+0x96/0xc8
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120d922>] xfs_write+0x2b8/0x784
  [<ffffffff81034825>] ? wake_up_state+0xb/0xd
  [<ffffffff810033ce>] ? apic_timer_interrupt+0xe/0x20
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4648  7761      1 0x00000000
  ffff88029a60bc08 0000000000000082 0000000000000000 ffff88002826de78
  0000000249c7788e 000000000000de78 ffff88029a60bfd8 0000000000013c00
  ffff88029a60bfd8 0000000000013c00 ffff88090a0bdb80 ffff881239d23d00
 Call Trace:
  [<ffffffff814ec6ac>] ? rwsem_down_failed_common+0x96/0xc8
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120d922>] xfs_write+0x2b8/0x784
  [<ffffffff81034825>] ? wake_up_state+0xb/0xd
  [<ffffffff814ecb8e>] ? common_interrupt+0xe/0x13
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff81101c00>] ? inotify_dentry_parent_queue_event+0x80/0x8c
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4648  7762      1 0x00000000
  ffff8806357d5c08 0000000000000082 0000000000000000 ffffffff8102cbe0
  ffff8806357d5b88 000000000000de78 ffff8806357d5fd8 0000000000013c00
  ffff8806357d5fd8 0000000000013c00 ffff88090a0bc4a0 ffff880939ce8000
 Call Trace:
  [<ffffffff8102cbe0>] ? check_preempt_wakeup+0xde/0x15d
  [<ffffffff814ec6ac>] ? rwsem_down_failed_common+0x96/0xc8
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120d922>] xfs_write+0x2b8/0x784
  [<ffffffff814ecb8e>] ? common_interrupt+0xe/0x13
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff81101c00>] ? inotify_dentry_parent_queue_event+0x80/0x8c
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D ffff880938821380  4648  7763      1 0x00000000
  ffff8801b4febbb8 0000000000000082 ffff8801b4febb58 ffff880937ea2df8
  000000000000de78 000000000000de78 ffff8801b4febfd8 0000000000013c00
  ffff8801b4febfd8 0000000000013c00 ffff880937ea6ac0 ffff8812116787a0
 Call Trace:
  [<ffffffff81027beb>] ? activate_task+0x28/0x31
  [<ffffffff814ec6ac>] rwsem_down_failed_common+0x96/0xc8
  [<ffffffff814ec6fe>] rwsem_down_write_failed+0x20/0x2a
  [<ffffffff812504b3>] call_rwsem_down_write_failed+0x13/0x20
  [<ffffffff814ec005>] ? down_write+0x2d/0x31
  [<ffffffff811ebd14>] xfs_ilock+0x23/0x94
  [<ffffffff8120d92f>] xfs_write+0x2c5/0x784
  [<ffffffff81034825>] ? wake_up_state+0xb/0xd
  [<ffffffff814ecb8e>] ? common_interrupt+0xe/0x13
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4648  7764      1 0x00000000
  ffff880287b4dc08 0000000000000082 0000000000000000 ffff880287b4dec8
  ffff880287b4db78 000000000000de78 ffff880287b4dfd8 0000000000013c00
  ffff880287b4dfd8 0000000000013c00 ffff880937ea4c40 ffff880939ce8000
 Call Trace:
  [<ffffffff814ec6ac>] ? rwsem_down_failed_common+0x96/0xc8
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120d922>] xfs_write+0x2b8/0x784
  [<ffffffff81034825>] ? wake_up_state+0xb/0xd
  [<ffffffff814ecb8e>] ? common_interrupt+0xe/0x13
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff814ecb8e>] ? common_interrupt+0xe/0x13
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000002  4648  7765      1 0x00000000
  ffff880287b4ba78 0000000000000082 0000000000000000 0000000000000001
  0000000000000096 000000000000de78 ffff880287b4bfd8 0000000000013c00
  ffff880287b4bfd8 0000000000013c00 ffff880937ea2dc0 ffff88123885d3e0
 Call Trace:
  [<ffffffff8104e062>] ? insert_work+0x92/0x9e
  [<ffffffff8105945a>] ? ktime_get_ts+0xb1/0xbe
  [<ffffffff814eb4b9>] io_schedule+0x7b/0xc1
  [<ffffffff8109fc02>] sync_page+0x41/0x45
  [<ffffffff814eb8d9>] __wait_on_bit_lock+0x45/0x8c
  [<ffffffff8109fbc1>] ? sync_page+0x0/0x45
  [<ffffffff8109fb84>] __lock_page+0x63/0x6a
  [<ffffffff810517e3>] ? wake_bit_function+0x0/0x2a
  [<ffffffff810a82de>] lock_page+0x24/0x28
  [<ffffffff810a8364>] invalidate_inode_pages2_range+0x82/0x2af
  [<ffffffff8120646a>] ? xfs_vm_direct_IO+0x90/0xbb
  [<ffffffff812066a8>] ? xfs_get_blocks_direct+0x0/0x17
  [<ffffffff812062c8>] ? xfs_end_io_direct+0x0/0x4e
  [<ffffffff810a085d>] generic_file_direct_write+0xf5/0x14b
  [<ffffffff8120db02>] xfs_write+0x498/0x784
  [<ffffffff81034825>] ? wake_up_state+0xb/0xd
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4240  7787      1 0x00000000
  ffff8801b4f67cb8 0000000000000082 0000000000000000 0000000000000296
  ffff8801b4f67c58 000000000000de78 ffff8801b4f67fd8 0000000000013c00
  ffff8801b4f67fd8 0000000000013c00 ffff8809388d6320 ffff880939ce8000
 Call Trace:
  [<ffffffff8105ee38>] ? wake_futex+0x3b/0x57
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff811f6f46>] ? xfs_log_reserve+0x101/0x10a
  [<ffffffff81204b5c>] ? xfs_fsync+0x135/0x16a
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  3776 22038      1 0x00000000
  ffff880da3973cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff880da3973c28 000000000000de78 ffff880da3973fd8 0000000000013c00
  ffff880da3973fd8 0000000000013c00 ffff88097f63c4a0 ffff881239d20f40
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  3776 29433      1 0x00000000
  ffff881236b49cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff881236b49c28 000000000000de78 ffff881236b49fd8 0000000000013c00
  ffff881236b49fd8 0000000000013c00 ffff880dc6e244a0 ffff881239d20f40
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4544  5673      1 0x00000000
  ffff880fa9189cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff880d751f0d00 000000000000de78 ffff880fa9189fd8 0000000000013c00
  ffff880fa9189fd8 0000000000013c00 ffff880bf12c2620 ffff880939c9c4a0
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff8102832b>] ? pick_next_task+0x22/0x3e
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  3632  6093      1 0x00000000
  ffff880e39e61cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff880e39e61c28 000000000000de78 ffff880e39e61fd8 0000000000013c00
  ffff880e39e61fd8 0000000000013c00 ffff8811f78d1e80 ffff880939c9c4a0
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  5336  6840      1 0x00000000
  ffff880e3d89fcb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff880e3d89fc28 000000000000de78 ffff880e3d89ffd8 0000000000013c00
  ffff880e3d89ffd8 0000000000013c00 ffff8809aba6f260 ffff880939c9c4a0
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  5336  6841      1 0x00000000
  ffff8803f0d73cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff8803f0d73c28 000000000000de78 ffff8803f0d73fd8 0000000000013c00
  ffff8803f0d73fd8 0000000000013c00 ffff8807c98444a0 ffff880939c9f260
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  5328  6883      1 0x00000000
  ffff880a3994fcb8 0000000000000082 0000000000000000 ffffffff813fc68a
  0000000000000005 000000000000de78 ffff880a3994ffd8 0000000000013c00
  ffff880a3994ffd8 0000000000013c00 ffff8811567cdb80 ffffffff81a0e020
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff810ed26a>] ? mntput_no_expire+0x24/0xe0
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  5312  6887      1 0x00000000
  ffff880fb56c9cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff881039c8a080 000000000000de78 ffff880fb56c9fd8 0000000000013c00
  ffff880fb56c9fd8 0000000000013c00 ffff880e3659cc40 ffffffff81a0e020
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  5336  6888      1 0x00000000
  ffff88030d555cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff880671892d80 000000000000de78 ffff88030d555fd8 0000000000013c00
  ffff88030d555fd8 0000000000013c00 ffff88071109d3e0 ffff880939c9adc0
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  5336  6890      1 0x00000000
  ffff880690c15cb8 0000000000000082 0000000000000000 ffff8812116787d8
  ffff88095fc93c00 000000000000de78 ffff880690c15fd8 0000000000013c00
  ffff880690c15fd8 0000000000013c00 ffff8808b9585b80 ffffffff81a0e020
 Call Trace:
  [<ffffffff81027beb>] ? activate_task+0x28/0x31
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  5312  6899      1 0x00000000
  ffff88090a091cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff88090a091c28 000000000000de78 ffff88090a091fd8 0000000000013c00
  ffff88090a091fd8 0000000000013c00 ffff88085196cc40 ffffffff81a0e020
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 tar4ibd       D 0000000000000000  4552  6721   6720 0x00000004
  ffff88016a7c3740 0000000000000086 0000000000000000 ffff88095fc13c00
  0000000000000001 000000000000de78 ffff88016a7c3fd8 0000000000013c00
  ffff88016a7c3fd8 0000000000013c00 ffff8806b39344a0 ffff881239d253e0
 Call Trace:
  [<ffffffff81034837>] ? wake_up_process+0x10/0x12
  [<ffffffff8124c328>] ? __rwsem_do_wake+0x80/0x152
  [<ffffffff814ec6ac>] rwsem_down_failed_common+0x96/0xc8
  [<ffffffff81244595>] ? cfq_close_cooperator+0xd9/0x184
  [<ffffffff814ec72e>] rwsem_down_read_failed+0x26/0x30
  [<ffffffff81250484>] call_rwsem_down_read_failed+0x14/0x30
  [<ffffffff814ec028>] ? down_read+0x1f/0x23
  [<ffffffff811ebd4e>] xfs_ilock+0x5d/0x94
  [<ffffffff811ebdae>] xfs_ilock_map_shared+0x29/0x2f
  [<ffffffff811f231b>] xfs_iomap+0x9f/0x39c
  [<ffffffff81239215>] ? __blk_run_queue+0x4a/0x70
  [<ffffffff81247a91>] ? cpumask_next_and+0x2b/0x3c
  [<ffffffff812065bd>] __xfs_get_blocks+0x71/0x15c
  [<ffffffff812066d1>] xfs_get_blocks+0x12/0x14
  [<ffffffff810fec23>] do_mpage_readpage+0x173/0x467
  [<ffffffff810b2234>] ? __inc_zone_page_state+0x25/0x27
  [<ffffffff8109fecc>] ? add_to_page_cache_locked+0x82/0xc6
  [<ffffffff810ff04a>] mpage_readpages+0xd6/0x11c
  [<ffffffff812066bf>] ? xfs_get_blocks+0x0/0x14
  [<ffffffff812066bf>] ? xfs_get_blocks+0x0/0x14
  [<ffffffff81207724>] xfs_vm_readpages+0x1a/0x1c
  [<ffffffff810a72c4>] __do_page_cache_readahead+0x10c/0x1a2
  [<ffffffff810a7376>] ra_submit+0x1c/0x20
  [<ffffffff810a7606>] ondemand_readahead+0x1b2/0x1c5
  [<ffffffff810a76f3>] page_cache_sync_readahead+0x38/0x3a
  [<ffffffff810a0e0c>] generic_file_aio_read+0x229/0x582
  [<ffffffff8120e262>] xfs_read+0x1a3/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff81032879>] ? finish_task_switch+0x43/0xac
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d8931>] sys_read+0x47/0x70
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 sync          D 0000000000000000  4496  8295   8248 0x00000000
  ffff8805b78b1cd8 0000000000000086 0000000000000000 0000000000013c00
  ffff8809388acac0 000000000000de78 ffff8805b78b1fd8 0000000000013c00
  ffff8805b78b1fd8 0000000000013c00 ffff8801002dadc0 ffff881239d23d00
 Call Trace:
  [<ffffffff810519e4>] ? prepare_to_wait+0x70/0x7b
  [<ffffffff81207928>] xfs_ioend_wait+0x83/0x9b
  [<ffffffff810517ab>] ? autoremove_wake_function+0x0/0x38
  [<ffffffff8121071b>] ? xfs_sync_inode_valid+0x68/0x9a
  [<ffffffff81210aaa>] xfs_sync_inode_data+0x98/0xbf
  [<ffffffff81210bf0>] xfs_inode_ag_walk+0x95/0xd4
  [<ffffffff81210a12>] ? xfs_sync_inode_data+0x0/0xbf
  [<ffffffff8109fdac>] ? wait_on_page_bit+0x6e/0x75
  [<ffffffff81210c82>] xfs_inode_ag_iterator+0x53/0x81
  [<ffffffff81210a12>] ? xfs_sync_inode_data+0x0/0xbf
  [<ffffffff81210e49>] xfs_sync_data+0x26/0x4d
  [<ffffffff81210ebf>] xfs_quiesce_data+0x1e/0x57
  [<ffffffff8120e841>] xfs_fs_sync_fs+0x26/0xd7
  [<ffffffff8111a159>] ? vfs_quota_sync+0x196/0x1a7
  [<ffffffff8111ba70>] sync_quota_sb+0x4e/0xfc
  [<ffffffff810f5e64>] __sync_filesystem+0x41/0x72
  [<ffffffff810f5f37>] sync_filesystems+0xa2/0xee
  [<ffffffff810f5fd4>] sys_sync+0x1c/0x2e
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b


--
Ilia Mirkin
[hidden email]

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Dave Chinner
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Dave Chinner
On Sat, Jul 17, 2010 at 12:01:11AM -0400, Ilia Mirkin wrote:

> Hi XFS developers,
>
> I ended up getting this on a 2.6.33.3 kernel. I realize this is not
> the latest, but I've only gotten this once, so it will not be possible
> to easily determine whether this is gone in the latest release or not.
>
> The details of the situation is that there is an XFS filesystem on a
> hardware RAID10 array (directly on the device, no partition table),
> and it exclusively houses a mysql (5.0) database. The database gets
> backed up nightly, which is basically a glorified tar'ing of most of
> the partition. The non-mysql process you see there are related to a
> mysql backup program (which coincidentally triggers WARN's in xfs
> code, as per http://marc.info/?l=linux-xfs&m=127632333825584&w=2;
> however it sounded like those WARN's were not warning about an
> in-kernel problems; curiously, the last WARN's in the kernel log
> appear to have been from the prior day's backup run, not the one that
> triggered the hang). There's also a sync that I ran as a last-ditch
> effort at saving the system. On reboot, everything came back up OK.

<sigh>

That WARN_ON() is not coincidental at all.  I´ll come back to this
later.

> While the system was stuck, vmstat did not report any block-in/out on
> the device. I have rrd's of those if necessary. Let me know if I might
> be able to provide any other useful info, or if you think that this
> sort of thing should be fixed in the latest linux kernel release. I
> saw that there have been a lot of XFS updates since 2.6.33, but none
> stood out as addressing this type of situation.
>
> Here is the output of "echo w > /proc/sysrq-trigger":
>
>  SysRq : Show Blocked State
>    task                        PC stack   pid father
>  xfsdatad/7    D ffff88123885d3e0  6792   772      2 0x00000000
>   ffff88123887fd90 0000000000000046 0000000000000000 ffff88123887fd5c
>   0000000000000000 000000000000de78 ffff88123887ffd8 0000000000013c00
>   ffff88123887ffd8 0000000000013c00 ffff88123885d3e0 ffff881239d23d00
>  Call Trace:
>   [<ffffffff81045f9c>] ? __mod_timer+0x13d/0x14f
>   [<ffffffff814eb81c>] schedule_timeout+0x98/0xbf
>   [<ffffffff81045b27>] ? process_timeout+0x0/0xb
>   [<ffffffff814eb85c>] schedule_timeout_uninterruptible+0x19/0x1b
>   [<ffffffff81206512>] xfs_end_io+0x7d/0x8c
>   [<ffffffff8104de65>] worker_thread+0x18d/0x222
>   [<ffffffff81206495>] ? xfs_end_io+0x0/0x8c
>   [<ffffffff810517ab>] ? autoremove_wake_function+0x0/0x38
>   [<ffffffff8104dcd8>] ? worker_thread+0x0/0x222
>   [<ffffffff81051336>] kthread+0x7d/0x85
>   [<ffffffff81003814>] kernel_thread_helper+0x4/0x10
>   [<ffffffff810512b9>] ? kthread+0x0/0x85
>   [<ffffffff81003810>] ? kernel_thread_helper+0x0/0x10

Waiting on IO completion - this is probably getting EAGAIN due
to not being able to take the inode ilock....

>  mysqld        D ffff880938821380  4696  7750      1 0x00000000
>   ffff880650e01cb8 0000000000000082 00715f4f00000001 0000000000000000
>   ffff88090a0bd3e0 000000000000de78 ffff880650e01fd8 0000000000013c00
>   ffff880650e01fd8 0000000000013c00 ffff88090a0bd3e0 ffff88090a0bf260
>  Call Trace:
>   [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
>   [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
>   [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
>   [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
>   [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
>   [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
>   [<ffffffff8102832b>] ? pick_next_task+0x22/0x3e
>   [<ffffffff810d8813>] vfs_read+0xa8/0x102
>   [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
>   [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b

Waiting on i_mutex, no other locks held.

....

>  mysqld        D ffff880938821380  4648  7763      1 0x00000000
>   ffff8801b4febbb8 0000000000000082 ffff8801b4febb58 ffff880937ea2df8
>   000000000000de78 000000000000de78 ffff8801b4febfd8 0000000000013c00
>   ffff8801b4febfd8 0000000000013c00 ffff880937ea6ac0 ffff8812116787a0
>  Call Trace:
>   [<ffffffff81027beb>] ? activate_task+0x28/0x31
>   [<ffffffff814ec6ac>] rwsem_down_failed_common+0x96/0xc8
>   [<ffffffff814ec6fe>] rwsem_down_write_failed+0x20/0x2a
>   [<ffffffff812504b3>] call_rwsem_down_write_failed+0x13/0x20
>   [<ffffffff814ec005>] ? down_write+0x2d/0x31
>   [<ffffffff811ebd14>] xfs_ilock+0x23/0x94
>   [<ffffffff8120d92f>] xfs_write+0x2c5/0x784
>   [<ffffffff81034825>] ? wake_up_state+0xb/0xd
>   [<ffffffff814ecb8e>] ? common_interrupt+0xe/0x13
>   [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
>   [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
>   [<ffffffff810d8624>] vfs_write+0xab/0x105
>   [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
>   [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b

waiting on ione of the xfs inode locks in exclusive mode here  -
probably holds the i_mutex too.  Because it's an exclusive lock we're
waiting on here that, IIRC, will hold out any new readers, too.
Its probably stuck because the lock is already held in shared mode.

.....

>  mysqld        D 0000000000000002  4648  7765      1 0x00000000
>   ffff880287b4ba78 0000000000000082 0000000000000000 0000000000000001
>   0000000000000096 000000000000de78 ffff880287b4bfd8 0000000000013c00
>   ffff880287b4bfd8 0000000000013c00 ffff880937ea2dc0 ffff88123885d3e0
>  Call Trace:
>   [<ffffffff8104e062>] ? insert_work+0x92/0x9e
>   [<ffffffff8105945a>] ? ktime_get_ts+0xb1/0xbe
>   [<ffffffff814eb4b9>] io_schedule+0x7b/0xc1
>   [<ffffffff8109fc02>] sync_page+0x41/0x45
>   [<ffffffff814eb8d9>] __wait_on_bit_lock+0x45/0x8c
>   [<ffffffff8109fbc1>] ? sync_page+0x0/0x45
>   [<ffffffff8109fb84>] __lock_page+0x63/0x6a
>   [<ffffffff810517e3>] ? wake_bit_function+0x0/0x2a
>   [<ffffffff810a82de>] lock_page+0x24/0x28
>   [<ffffffff810a8364>] invalidate_inode_pages2_range+0x82/0x2af
>   [<ffffffff8120646a>] ? xfs_vm_direct_IO+0x90/0xbb
>   [<ffffffff812066a8>] ? xfs_get_blocks_direct+0x0/0x17
>   [<ffffffff812062c8>] ? xfs_end_io_direct+0x0/0x4e
>   [<ffffffff810a085d>] generic_file_direct_write+0xf5/0x14b
>   [<ffffffff8120db02>] xfs_write+0x498/0x784
>   [<ffffffff81034825>] ? wake_up_state+0xb/0xd
>   [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
>   [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
>   [<ffffffff810d8624>] vfs_write+0xab/0x105
>   [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
>   [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b

Direct IO write waiting on page cache invalidation which is blocked
on a locked page. i.e. it's trying to flush pages cached during
buffered IO.  Holds inode iolock here shared.

Also, it does not hold the i_mutex but is invalidating the page cache.
Remember that WARN_ON() condition that I'd say I'd come back to?
Guess what is warns about:

        if ((ioflags & IO_ISDIRECT)) {
                if (mapping->nrpages) {
>>>>>>>                 WARN_ON(need_i_mutex == 0);
                        error = xfs_flushinval_pages(xip,
                                        (pos & PAGE_CACHE_MASK),
                                        -1, FI_REMAPF_LOCKED);
                        if (error)
                                goto out_unlock_internal;
                }

Right - it's warning about invalidating the page cache without the
i_mutex held. And FWIW the comment in generic_file_direct_write()
about this invalidation during direct IO:

        /*
         * Finally, try again to invalidate clean pages which might have been
         * cached by non-direct readahead, or faulted in by get_user_pages()
         * if the source of the write was an mmap'ed region of the file
         * we're writing.  Either one is a pretty crazy thing to do,
         * so we don't support it 100%.  If this invalidation
         * fails, tough, the write still worked...
         */
        if (mapping->nrpages) {
                invalidate_inode_pages2_range(mapping,
                                              pos >> PAGE_CACHE_SHIFT, end);
        }

So the generic page cache code does not handle mixed bufered/direct
IO to the same file concurrently with a 100% coherency guarantee.
IOWs, just because the WARN_ON() is not being triggered, it *does not
mean that there is no problem*.


>  tar4ibd       D 0000000000000000  4552  6721   6720 0x00000004
>   ffff88016a7c3740 0000000000000086 0000000000000000 ffff88095fc13c00
>   0000000000000001 000000000000de78 ffff88016a7c3fd8 0000000000013c00
>   ffff88016a7c3fd8 0000000000013c00 ffff8806b39344a0 ffff881239d253e0
>  Call Trace:
>   [<ffffffff81034837>] ? wake_up_process+0x10/0x12
>   [<ffffffff8124c328>] ? __rwsem_do_wake+0x80/0x152
>   [<ffffffff814ec6ac>] rwsem_down_failed_common+0x96/0xc8
>   [<ffffffff81244595>] ? cfq_close_cooperator+0xd9/0x184
>   [<ffffffff814ec72e>] rwsem_down_read_failed+0x26/0x30
>   [<ffffffff81250484>] call_rwsem_down_read_failed+0x14/0x30
>   [<ffffffff814ec028>] ? down_read+0x1f/0x23
>   [<ffffffff811ebd4e>] xfs_ilock+0x5d/0x94
>   [<ffffffff811ebdae>] xfs_ilock_map_shared+0x29/0x2f
>   [<ffffffff811f231b>] xfs_iomap+0x9f/0x39c
>   [<ffffffff81239215>] ? __blk_run_queue+0x4a/0x70
>   [<ffffffff81247a91>] ? cpumask_next_and+0x2b/0x3c
>   [<ffffffff812065bd>] __xfs_get_blocks+0x71/0x15c
>   [<ffffffff812066d1>] xfs_get_blocks+0x12/0x14
>   [<ffffffff810fec23>] do_mpage_readpage+0x173/0x467
>   [<ffffffff810b2234>] ? __inc_zone_page_state+0x25/0x27
>   [<ffffffff8109fecc>] ? add_to_page_cache_locked+0x82/0xc6
>   [<ffffffff810ff04a>] mpage_readpages+0xd6/0x11c
>   [<ffffffff812066bf>] ? xfs_get_blocks+0x0/0x14
>   [<ffffffff812066bf>] ? xfs_get_blocks+0x0/0x14
>   [<ffffffff81207724>] xfs_vm_readpages+0x1a/0x1c
>   [<ffffffff810a72c4>] __do_page_cache_readahead+0x10c/0x1a2
>   [<ffffffff810a7376>] ra_submit+0x1c/0x20
>   [<ffffffff810a7606>] ondemand_readahead+0x1b2/0x1c5
>   [<ffffffff810a76f3>] page_cache_sync_readahead+0x38/0x3a
>   [<ffffffff810a0e0c>] generic_file_aio_read+0x229/0x582
>   [<ffffffff8120e262>] xfs_read+0x1a3/0x1f9
>   [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
>   [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
>   [<ffffffff81032879>] ? finish_task_switch+0x43/0xac
>   [<ffffffff810d8813>] vfs_read+0xa8/0x102
>   [<ffffffff810d8931>] sys_read+0x47/0x70
>   [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b

And there's the backup program, doing buffered readahead, holding a
page locked and stuck trying to get the xfs inode ilock. i.e. this
trace shows your system has tripped over the exact case that the
above code says is "a pretty crazy thing to do". IOWs, this is not
an XFS specific problem.

I can't find a thread that holds the XFS inode lock that everything
is waiting on. I think it is the ILOCK, but none of the threads in
this trace should be holding it where they are blocked. IOWs, the
output does not give me enough information to get to the root cause.

So, back to the situation with the WARN_ON(). You're running
applications that are doing something that:

        a) is not supported;
        b) compromises data integrity guarantees;
        c) is not reliably reported; and
        d) might be causing hangs

Right now I'm not particularly inclined to dig into this further;
it's obvious the applications are doing something that is not
supported (by XFS or the generic page cache code), so this is the
first thing you really need to care about getting fixed if you value
your backups...

Cheers,

Dave.
--
Dave Chinner
[hidden email]

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Ilia Mirkin-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Ilia Mirkin-2
Hi Dave,

On Sat, Jul 17, 2010 at 9:20 PM, Dave Chinner <[hidden email]> wrote:

> On Sat, Jul 17, 2010 at 12:01:11AM -0400, Ilia Mirkin wrote:
>> Hi XFS developers,
>>
>> I ended up getting this on a 2.6.33.3 kernel. I realize this is not
>> the latest, but I've only gotten this once, so it will not be possible
>> to easily determine whether this is gone in the latest release or not.
>>
>> The details of the situation is that there is an XFS filesystem on a
>> hardware RAID10 array (directly on the device, no partition table),
>> and it exclusively houses a mysql (5.0) database. The database gets
>> backed up nightly, which is basically a glorified tar'ing of most of
>> the partition. The non-mysql process you see there are related to a
>> mysql backup program (which coincidentally triggers WARN's in xfs
>> code, as per http://marc.info/?l=linux-xfs&m=127632333825584&w=2;
>> however it sounded like those WARN's were not warning about an
>> in-kernel problems; curiously, the last WARN's in the kernel log
>> appear to have been from the prior day's backup run, not the one that
>> triggered the hang). There's also a sync that I ran as a last-ditch
>> effort at saving the system. On reboot, everything came back up OK.
>
> <sigh>
>
> That WARN_ON() is not coincidental at all.  I´ll come back to this
> later.
>
>> While the system was stuck, vmstat did not report any block-in/out on
>> the device. I have rrd's of those if necessary. Let me know if I might
>> be able to provide any other useful info, or if you think that this
>> sort of thing should be fixed in the latest linux kernel release. I
>> saw that there have been a lot of XFS updates since 2.6.33, but none
>> stood out as addressing this type of situation.
>>
>> Here is the output of "echo w > /proc/sysrq-trigger":
>>
>>  SysRq : Show Blocked State
>>    task                        PC stack   pid father
>>  xfsdatad/7    D ffff88123885d3e0  6792   772      2 0x00000000
>>   ffff88123887fd90 0000000000000046 0000000000000000 ffff88123887fd5c
>>   0000000000000000 000000000000de78 ffff88123887ffd8 0000000000013c00
>>   ffff88123887ffd8 0000000000013c00 ffff88123885d3e0 ffff881239d23d00
>>  Call Trace:
>>   [<ffffffff81045f9c>] ? __mod_timer+0x13d/0x14f
>>   [<ffffffff814eb81c>] schedule_timeout+0x98/0xbf
>>   [<ffffffff81045b27>] ? process_timeout+0x0/0xb
>>   [<ffffffff814eb85c>] schedule_timeout_uninterruptible+0x19/0x1b
>>   [<ffffffff81206512>] xfs_end_io+0x7d/0x8c
>>   [<ffffffff8104de65>] worker_thread+0x18d/0x222
>>   [<ffffffff81206495>] ? xfs_end_io+0x0/0x8c
>>   [<ffffffff810517ab>] ? autoremove_wake_function+0x0/0x38
>>   [<ffffffff8104dcd8>] ? worker_thread+0x0/0x222
>>   [<ffffffff81051336>] kthread+0x7d/0x85
>>   [<ffffffff81003814>] kernel_thread_helper+0x4/0x10
>>   [<ffffffff810512b9>] ? kthread+0x0/0x85
>>   [<ffffffff81003810>] ? kernel_thread_helper+0x0/0x10
>
> Waiting on IO completion - this is probably getting EAGAIN due
> to not being able to take the inode ilock....
>
>>  mysqld        D ffff880938821380  4696  7750      1 0x00000000
>>   ffff880650e01cb8 0000000000000082 00715f4f00000001 0000000000000000
>>   ffff88090a0bd3e0 000000000000de78 ffff880650e01fd8 0000000000013c00
>>   ffff880650e01fd8 0000000000013c00 ffff88090a0bd3e0 ffff88090a0bf260
>>  Call Trace:
>>   [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
>>   [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
>>   [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
>>   [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
>>   [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
>>   [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
>>   [<ffffffff8102832b>] ? pick_next_task+0x22/0x3e
>>   [<ffffffff810d8813>] vfs_read+0xa8/0x102
>>   [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
>>   [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
>
> Waiting on i_mutex, no other locks held.
>
> ....
>
>>  mysqld        D ffff880938821380  4648  7763      1 0x00000000
>>   ffff8801b4febbb8 0000000000000082 ffff8801b4febb58 ffff880937ea2df8
>>   000000000000de78 000000000000de78 ffff8801b4febfd8 0000000000013c00
>>   ffff8801b4febfd8 0000000000013c00 ffff880937ea6ac0 ffff8812116787a0
>>  Call Trace:
>>   [<ffffffff81027beb>] ? activate_task+0x28/0x31
>>   [<ffffffff814ec6ac>] rwsem_down_failed_common+0x96/0xc8
>>   [<ffffffff814ec6fe>] rwsem_down_write_failed+0x20/0x2a
>>   [<ffffffff812504b3>] call_rwsem_down_write_failed+0x13/0x20
>>   [<ffffffff814ec005>] ? down_write+0x2d/0x31
>>   [<ffffffff811ebd14>] xfs_ilock+0x23/0x94
>>   [<ffffffff8120d92f>] xfs_write+0x2c5/0x784
>>   [<ffffffff81034825>] ? wake_up_state+0xb/0xd
>>   [<ffffffff814ecb8e>] ? common_interrupt+0xe/0x13
>>   [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
>>   [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
>>   [<ffffffff810d8624>] vfs_write+0xab/0x105
>>   [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
>>   [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
>
> waiting on ione of the xfs inode locks in exclusive mode here  -
> probably holds the i_mutex too.  Because it's an exclusive lock we're
> waiting on here that, IIRC, will hold out any new readers, too.
> Its probably stuck because the lock is already held in shared mode.
>
> .....
>
>>  mysqld        D 0000000000000002  4648  7765      1 0x00000000
>>   ffff880287b4ba78 0000000000000082 0000000000000000 0000000000000001
>>   0000000000000096 000000000000de78 ffff880287b4bfd8 0000000000013c00
>>   ffff880287b4bfd8 0000000000013c00 ffff880937ea2dc0 ffff88123885d3e0
>>  Call Trace:
>>   [<ffffffff8104e062>] ? insert_work+0x92/0x9e
>>   [<ffffffff8105945a>] ? ktime_get_ts+0xb1/0xbe
>>   [<ffffffff814eb4b9>] io_schedule+0x7b/0xc1
>>   [<ffffffff8109fc02>] sync_page+0x41/0x45
>>   [<ffffffff814eb8d9>] __wait_on_bit_lock+0x45/0x8c
>>   [<ffffffff8109fbc1>] ? sync_page+0x0/0x45
>>   [<ffffffff8109fb84>] __lock_page+0x63/0x6a
>>   [<ffffffff810517e3>] ? wake_bit_function+0x0/0x2a
>>   [<ffffffff810a82de>] lock_page+0x24/0x28
>>   [<ffffffff810a8364>] invalidate_inode_pages2_range+0x82/0x2af
>>   [<ffffffff8120646a>] ? xfs_vm_direct_IO+0x90/0xbb
>>   [<ffffffff812066a8>] ? xfs_get_blocks_direct+0x0/0x17
>>   [<ffffffff812062c8>] ? xfs_end_io_direct+0x0/0x4e
>>   [<ffffffff810a085d>] generic_file_direct_write+0xf5/0x14b
>>   [<ffffffff8120db02>] xfs_write+0x498/0x784
>>   [<ffffffff81034825>] ? wake_up_state+0xb/0xd
>>   [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
>>   [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
>>   [<ffffffff810d8624>] vfs_write+0xab/0x105
>>   [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
>>   [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
>
> Direct IO write waiting on page cache invalidation which is blocked
> on a locked page. i.e. it's trying to flush pages cached during
> buffered IO.  Holds inode iolock here shared.
>
> Also, it does not hold the i_mutex but is invalidating the page cache.
> Remember that WARN_ON() condition that I'd say I'd come back to?
> Guess what is warns about:
>
>        if ((ioflags & IO_ISDIRECT)) {
>                if (mapping->nrpages) {
>>>>>>>>                 WARN_ON(need_i_mutex == 0);
>                        error = xfs_flushinval_pages(xip,
>                                        (pos & PAGE_CACHE_MASK),
>                                        -1, FI_REMAPF_LOCKED);
>                        if (error)
>                                goto out_unlock_internal;
>                }
>
> Right - it's warning about invalidating the page cache without the
> i_mutex held. And FWIW the comment in generic_file_direct_write()
> about this invalidation during direct IO:
>
>        /*
>         * Finally, try again to invalidate clean pages which might have been
>         * cached by non-direct readahead, or faulted in by get_user_pages()
>         * if the source of the write was an mmap'ed region of the file
>         * we're writing.  Either one is a pretty crazy thing to do,
>         * so we don't support it 100%.  If this invalidation
>         * fails, tough, the write still worked...
>         */
>        if (mapping->nrpages) {
>                invalidate_inode_pages2_range(mapping,
>                                              pos >> PAGE_CACHE_SHIFT, end);
>        }
>
> So the generic page cache code does not handle mixed bufered/direct
> IO to the same file concurrently with a 100% coherency guarantee.
> IOWs, just because the WARN_ON() is not being triggered, it *does not
> mean that there is no problem*.
>
>
>>  tar4ibd       D 0000000000000000  4552  6721   6720 0x00000004
>>   ffff88016a7c3740 0000000000000086 0000000000000000 ffff88095fc13c00
>>   0000000000000001 000000000000de78 ffff88016a7c3fd8 0000000000013c00
>>   ffff88016a7c3fd8 0000000000013c00 ffff8806b39344a0 ffff881239d253e0
>>  Call Trace:
>>   [<ffffffff81034837>] ? wake_up_process+0x10/0x12
>>   [<ffffffff8124c328>] ? __rwsem_do_wake+0x80/0x152
>>   [<ffffffff814ec6ac>] rwsem_down_failed_common+0x96/0xc8
>>   [<ffffffff81244595>] ? cfq_close_cooperator+0xd9/0x184
>>   [<ffffffff814ec72e>] rwsem_down_read_failed+0x26/0x30
>>   [<ffffffff81250484>] call_rwsem_down_read_failed+0x14/0x30
>>   [<ffffffff814ec028>] ? down_read+0x1f/0x23
>>   [<ffffffff811ebd4e>] xfs_ilock+0x5d/0x94
>>   [<ffffffff811ebdae>] xfs_ilock_map_shared+0x29/0x2f
>>   [<ffffffff811f231b>] xfs_iomap+0x9f/0x39c
>>   [<ffffffff81239215>] ? __blk_run_queue+0x4a/0x70
>>   [<ffffffff81247a91>] ? cpumask_next_and+0x2b/0x3c
>>   [<ffffffff812065bd>] __xfs_get_blocks+0x71/0x15c
>>   [<ffffffff812066d1>] xfs_get_blocks+0x12/0x14
>>   [<ffffffff810fec23>] do_mpage_readpage+0x173/0x467
>>   [<ffffffff810b2234>] ? __inc_zone_page_state+0x25/0x27
>>   [<ffffffff8109fecc>] ? add_to_page_cache_locked+0x82/0xc6
>>   [<ffffffff810ff04a>] mpage_readpages+0xd6/0x11c
>>   [<ffffffff812066bf>] ? xfs_get_blocks+0x0/0x14
>>   [<ffffffff812066bf>] ? xfs_get_blocks+0x0/0x14
>>   [<ffffffff81207724>] xfs_vm_readpages+0x1a/0x1c
>>   [<ffffffff810a72c4>] __do_page_cache_readahead+0x10c/0x1a2
>>   [<ffffffff810a7376>] ra_submit+0x1c/0x20
>>   [<ffffffff810a7606>] ondemand_readahead+0x1b2/0x1c5
>>   [<ffffffff810a76f3>] page_cache_sync_readahead+0x38/0x3a
>>   [<ffffffff810a0e0c>] generic_file_aio_read+0x229/0x582
>>   [<ffffffff8120e262>] xfs_read+0x1a3/0x1f9
>>   [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
>>   [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
>>   [<ffffffff81032879>] ? finish_task_switch+0x43/0xac
>>   [<ffffffff810d8813>] vfs_read+0xa8/0x102
>>   [<ffffffff810d8931>] sys_read+0x47/0x70
>>   [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
>
> And there's the backup program, doing buffered readahead, holding a
> page locked and stuck trying to get the xfs inode ilock. i.e. this
> trace shows your system has tripped over the exact case that the
> above code says is "a pretty crazy thing to do". IOWs, this is not
> an XFS specific problem.
>
> I can't find a thread that holds the XFS inode lock that everything
> is waiting on. I think it is the ILOCK, but none of the threads in
> this trace should be holding it where they are blocked. IOWs, the
> output does not give me enough information to get to the root cause.

In case this happens again, was there something more useful I could
have collected? Should I have grabbed all task states?

> So, back to the situation with the WARN_ON(). You're running
> applications that are doing something that:
>
>        a) is not supported;
>        b) compromises data integrity guarantees;
>        c) is not reliably reported; and
>        d) might be causing hangs
>
> Right now I'm not particularly inclined to dig into this further;
> it's obvious the applications are doing something that is not
> supported (by XFS or the generic page cache code), so this is the
> first thing you really need to care about getting fixed if you value
> your backups...

Thanks for the analysis. I will try to switch off O_DIRECT from mysql
-- it sounds like that should solve all of these problems. In the
earlier thread, it sounded like the WARN was harmless from an OS
standpoint, and the backups were checking out fine.

It's pretty obvious that allowing userspace to hang the FS is really
bad, but I appreciate that the app is doing something that the kernel
didn't expect.

Thanks again,

--
Ilia Mirkin
[hidden email]

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Christoph Hellwig
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Christoph Hellwig
In reply to this post by Dave Chinner
On Sun, Jul 18, 2010 at 11:20:33AM +1000, Dave Chinner wrote:

> So, back to the situation with the WARN_ON(). You're running
> applications that are doing something that:
>
> a) is not supported;
> b) compromises data integrity guarantees;
> c) is not reliably reported; and
> d) might be causing hangs
>
> Right now I'm not particularly inclined to dig into this further;
> it's obvious the applications are doing something that is not
> supported (by XFS or the generic page cache code), so this is the
> first thing you really need to care about getting fixed if you value
> your backups...

While it's slightly crazy it's also a pretty easy way for users to shoot
themselve in their feet.  Unlike the generic filesystems with their
simplistic i_mutex locking we have a way to assure this works properly
in XFS with the shared/exclusive iolock, so I'm willing to look into
this further.

Ilia, would you be willing to test patches for this?

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Ilia Mirkin-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Ilia Mirkin-2
On Sun, Jul 18, 2010 at 12:57 AM, Christoph Hellwig <[hidden email]> wrote:

> On Sun, Jul 18, 2010 at 11:20:33AM +1000, Dave Chinner wrote:
>> So, back to the situation with the WARN_ON(). You're running
>> applications that are doing something that:
>>
>>       a) is not supported;
>>       b) compromises data integrity guarantees;
>>       c) is not reliably reported; and
>>       d) might be causing hangs
>>
>> Right now I'm not particularly inclined to dig into this further;
>> it's obvious the applications are doing something that is not
>> supported (by XFS or the generic page cache code), so this is the
>> first thing you really need to care about getting fixed if you value
>> your backups...
>
> While it's slightly crazy it's also a pretty easy way for users to shoot
> themselve in their feet.  Unlike the generic filesystems with their
> simplistic i_mutex locking we have a way to assure this works properly
> in XFS with the shared/exclusive iolock, so I'm willing to look into
> this further.
>
> Ilia, would you be willing to test patches for this?

If by "this" you mean the WARN_ON's, no problem. It should be easy to
repro in a non-critical setup, although I haven't tried. If you mean
the hang, it will not be so easy to reproduce, as it has only happened
once so far.

I would also be happy to share the details of our setup, if you'd like
to be able to play with it directly yourself. With our setup, multiple
WARN_ON's happen every time we run a backup (last time I checked, it
was ~50, but I'm sure it varies).

As a last thought on the matter, I'm sure that this is all brought on
by our use of innodb_flush_method=O_DIRECT which tells mysql to use
direct io. Leaving that at its default which will just open the files
regularly will probably make all these issues go away. I do not have a
sufficient understanding of the details of how mysql uses direct io,
and how it interacts with xtrabackup (which claims to work fine with
direct io, but who knows) to be able to declare that it's safe, so I'm
happy to accept Dave's advice to Not Do That.

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Ilia Mirkin-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Ilia Mirkin-2
On Sun, Jul 18, 2010 at 1:28 AM, Ilia Mirkin <[hidden email]> wrote:

> On Sun, Jul 18, 2010 at 12:57 AM, Christoph Hellwig <[hidden email]> wrote:
>> On Sun, Jul 18, 2010 at 11:20:33AM +1000, Dave Chinner wrote:
>>> So, back to the situation with the WARN_ON(). You're running
>>> applications that are doing something that:
>>>
>>>       a) is not supported;
>>>       b) compromises data integrity guarantees;
>>>       c) is not reliably reported; and
>>>       d) might be causing hangs
>>>
>>> Right now I'm not particularly inclined to dig into this further;
>>> it's obvious the applications are doing something that is not
>>> supported (by XFS or the generic page cache code), so this is the
>>> first thing you really need to care about getting fixed if you value
>>> your backups...
>>
>> While it's slightly crazy it's also a pretty easy way for users to shoot
>> themselve in their feet.  Unlike the generic filesystems with their
>> simplistic i_mutex locking we have a way to assure this works properly
>> in XFS with the shared/exclusive iolock, so I'm willing to look into
>> this further.
>>
>> Ilia, would you be willing to test patches for this?
>
> If by "this" you mean the WARN_ON's, no problem. It should be easy to
> repro in a non-critical setup, although I haven't tried. If you mean
> the hang, it will not be so easy to reproduce, as it has only happened
> once so far.
>
> I would also be happy to share the details of our setup, if you'd like
> to be able to play with it directly yourself. With our setup, multiple
> WARN_ON's happen every time we run a backup (last time I checked, it
> was ~50, but I'm sure it varies).
>
> As a last thought on the matter, I'm sure that this is all brought on
> by our use of innodb_flush_method=O_DIRECT which tells mysql to use
> direct io. Leaving that at its default which will just open the files
> regularly will probably make all these issues go away. I do not have a
> sufficient understanding of the details of how mysql uses direct io,
> and how it interacts with xtrabackup (which claims to work fine with
> direct io, but who knows) to be able to declare that it's safe, so I'm
> happy to accept Dave's advice to Not Do That.
>

In case you guys are interested, I've also opened a bug at
https://bugs.launchpad.net/percona-xtrabackup/+bug/606981. I never
quite bothered to _really_ understand all of the details of O_DIRECT
vs mmap vs read, so if I've misrepresented reality in the bug, feel
free to correct it there, or let me know and I'll try to straighten
things out.

--
Ilia Mirkin
[hidden email]

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Dave Chinner
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Dave Chinner
In reply to this post by Christoph Hellwig
On Sun, Jul 18, 2010 at 12:57:02AM -0400, Christoph Hellwig wrote:

> On Sun, Jul 18, 2010 at 11:20:33AM +1000, Dave Chinner wrote:
> > So, back to the situation with the WARN_ON(). You're running
> > applications that are doing something that:
> >
> > a) is not supported;
> > b) compromises data integrity guarantees;
> > c) is not reliably reported; and
> > d) might be causing hangs
> >
> > Right now I'm not particularly inclined to dig into this further;
> > it's obvious the applications are doing something that is not
> > supported (by XFS or the generic page cache code), so this is the
> > first thing you really need to care about getting fixed if you value
> > your backups...
>
> While it's slightly crazy it's also a pretty easy way for users to shoot
> themselve in their feet.  Unlike the generic filesystems with their
> simplistic i_mutex locking we have a way to assure this works properly
> in XFS with the shared/exclusive iolock, so I'm willing to look into
> this further.

Sorry, that wasn't paticularly clear - What I was trying to say is
that I'm not really interested in solving all the generic
buffered/direct IO coherency issues. I agree that it should not
hang, so we do need to find out why it hung....

Cheers,

Dave.
--
Dave Chinner
[hidden email]

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Dave Chinner
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Dave Chinner
In reply to this post by Ilia Mirkin-2
On Sat, Jul 17, 2010 at 09:35:33PM -0400, Ilia Mirkin wrote:
> On Sat, Jul 17, 2010 at 9:20 PM, Dave Chinner <[hidden email]> wrote:
> > On Sat, Jul 17, 2010 at 12:01:11AM -0400, Ilia Mirkin wrote:
> > I can't find a thread that holds the XFS inode lock that everything
> > is waiting on. I think it is the ILOCK, but none of the threads in
> > this trace should be holding it where they are blocked. IOWs, the
> > output does not give me enough information to get to the root cause.
>
> In case this happens again, was there something more useful I could
> have collected? Should I have grabbed all task states?

All the task states, including the running tasks, is probably a good
start. Also, if the kernel you are running has tracing events
enabled and has the necessary XFS tracepoints (I can't remember off
the top of my head whether they are in 2.6.33), you might want to
enable tracing of:

        xfs_ilock
        xfs_ilock_nowait
        xfs_ilock_demote
        xfs_iunlock

via:

# echo 1 > /sys/kernel/debug/tracing/events/xfs/<trace_point>/enable

and when the problem is hit dumping the trace via:

# cat /sys/kernel/debug/tracing/trace > trace.log

You may also want to bump up the trace buffer size to capture more
events:

# echo 32768 > /sys/kernel/debug/tracing/buffer_size_kb

Though I suspect the only way to get to the bottom of it will
be to work out a reproducable test case....

> It's pretty obvious that allowing userspace to hang the FS is really
> bad, but I appreciate that the app is doing something that the kernel
> didn't expect.

Yeah, we need to fix the hang - it's the bigger issues of mixed
direct/buffered IO that I was refering to...

Cheers,

Dave.
--
Dave Chinner
[hidden email]

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Ilia Mirkin-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Ilia Mirkin-2
On Sun, Jul 18, 2010 at 7:50 PM, Dave Chinner <[hidden email]> wrote:

> On Sat, Jul 17, 2010 at 09:35:33PM -0400, Ilia Mirkin wrote:
>> On Sat, Jul 17, 2010 at 9:20 PM, Dave Chinner <[hidden email]> wrote:
>> > On Sat, Jul 17, 2010 at 12:01:11AM -0400, Ilia Mirkin wrote:
>> > I can't find a thread that holds the XFS inode lock that everything
>> > is waiting on. I think it is the ILOCK, but none of the threads in
>> > this trace should be holding it where they are blocked. IOWs, the
>> > output does not give me enough information to get to the root cause.
>>
>> In case this happens again, was there something more useful I could
>> have collected? Should I have grabbed all task states?
>
> All the task states, including the running tasks, is probably a good
> start. Also, if the kernel you are running has tracing events
> enabled and has the necessary XFS tracepoints (I can't remember off
> the top of my head whether they are in 2.6.33), you might want to
> enable tracing of:
>
>        xfs_ilock
>        xfs_ilock_nowait
>        xfs_ilock_demote
>        xfs_iunlock
>
> via:
>
> # echo 1 > /sys/kernel/debug/tracing/events/xfs/<trace_point>/enable
>
> and when the problem is hit dumping the trace via:
>
> # cat /sys/kernel/debug/tracing/trace > trace.log
>
> You may also want to bump up the trace buffer size to capture more
> events:
>
> # echo 32768 > /sys/kernel/debug/tracing/buffer_size_kb

Sad. Nothing in /sys/kernel/debug, I tried turning on some config
options at random before doing this run (including lockdep), but I
still don't have these. Guess it's time to read some docs :)

>
> Though I suspect the only way to get to the bottom of it will
> be to work out a reproducable test case....

I felt a little bothered by this issue, so I dug in a little further.
I basically created a sample mysql, tar, and it seemed that having
memory pressure helped, so I created an allocator as well.

The "mysql" program basically just has 10 threads, each with their own
O_DIRECT-opened fd, reading/writing to random locations in a large
file.

The "tar" program opens the file, reads it in sequentially with plain read().

The "malloc" program allocates a configured amount of memory in the
system and then exits. Ideally this should be the amount of RAM in the
system.

I ran all this in a VM (kvm) with 4 VCPUs (the machine itself is 4S,
8C/S), 512M ram, 512M swap, and a dedicated lvm-backed volume for the
test XFS partition (3G). The warnings were no problem to trigger,
coming off at around one per second on average, the hang took about
4-5h to appear [although in the previous 8+ hour run, nothing
happened, and then I added lockdep].

Basically I was running in three separate shells:

./test-mysqld /mnt/tmp/test
./test-tar /mnt/tmp/test
while true; do ./test-malloc; done

(oh, and /mnt/tmp/test is a 3G file on a newish XFS; the fs was
created/used in prior boots as well, so not completely brand new)

test-tar ended up eventually stuck in a D state, as well as one of the
mysql threads. I could still create/delete files on the filesystem
(although I didn't try writing anything substantial).

The system is still up and running in this state, let me know if you'd
like me to try doing something in particular to it.

Below I have the program source code, as well as output from echo t >
/proc/sysrq-trigger.

The programs [this is not the cleanest code I've written, but it
apparently does the trick... probably left over unnecessary stuff from
earlier debugging in there]:

test-malloc.c
-------------
// gcc -Wall -o test-malloc -pthread test-malloc.c

#include <stdlib.h>
#include <unistd.h>
#include <string.h>

#define BLKSIZE 4096

int main(int argc, char *argv[]) {
    int i;
    for (i = 0; i < 512 * 1024 * 1024 / BLKSIZE; i++) {
        void *x = malloc(BLKSIZE);
        memset(x, 0, BLKSIZE);
        usleep(1);
    }
    return 0;
}

-------------
test-mysqld.c
-------------
// gcc -Wall -o test-mysqld -pthread test-mysqld.c

#define _GNU_SOURCE

#include <pthread.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>

#define THREADS 10
#define BLKSIZE 4096

void *thread(void *arg) {
    struct stat s;
    int fd = *(int *)arg;
    fstat(fd, &s);

    struct random_data rnd;
    char state[128];
    initstate_r(fd, state, 128, &rnd);

    char *buf = mmap(NULL, BLKSIZE, PROT_READ | PROT_WRITE, MAP_SHARED
| MAP_ANONYMOUS, -1, 0);

    int t;

    while (1) {
        random_r(&rnd, &t);
        off_t block = (t % s.st_size) / BLKSIZE;
        lseek(fd, block * BLKSIZE, SEEK_SET);
        random_r(&rnd, &t);
        int r = t % 2;
        if (r) {
            read(fd, buf, BLKSIZE);
        } else {
            int i;
            for (i = 0; i < BLKSIZE; i++) {
                random_r(&rnd, &t);
                buf[i] = (char)t;
            }
            write(fd, buf, BLKSIZE);
        }
    }

    close(fd);
    return NULL;
}

void *datasync(void *arg) {
    int *iarg = arg;
    int i;
    while (1) {
        for (i = 0; i < THREADS; i++) {
            fsync(iarg[i]);
        }
        sleep(1);
    }
}

int main(int argc, char *argv[]) {
    pthread_t threads[THREADS];
    int fds[THREADS];
    int i;
    for (i = 0; i < THREADS; i++) {
        fds[i] = open(argv[1], O_RDWR | O_DIRECT);
        pthread_create(&threads[i], NULL, thread, &fds[i]);
    }

    pthread_t sync_thread;
    pthread_create(&sync_thread, NULL, datasync, fds);

    for (i = 0; i < THREADS; i++) {
        pthread_join(threads[i], NULL);
    }

    return 0;
}


-------------
test-tar.c
-------------
// gcc -Wall -o test-tar -pthread test-tar.c

#define _GNU_SOURCE

#include <pthread.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <errno.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    while (1) {
        int fd = open(argv[1], O_RDWR);
        char buf[4096];

        lseek(fd, 0, SEEK_SET);
        while (read(fd, buf, 4096) != 0);

        close(fd);
    }
}

-------------------

The output of echo t > /proc/sysrq-trigger [apologies for line-wrapping]

[16087.359042] SysRq : Show State
[16087.359872]   task                        PC stack   pid father
[16087.359872] init          S 0000000000000000  3296     1      0 0x00000000
[16087.359872]  ffff88001f8ff908 0000000000000046 ffff88001f8ff868
0000000000000286
[16087.359872]  0000000000000040 000000000000def8 ffff88001f8fffd8
0000000000014180
[16087.359872]  ffff88001f8fffd8 0000000000014180 ffff88001f910000
ffff88001fa542c0
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff8148f948>] schedule_hrtimeout_range+0xc2/0xf7
[16087.359872]  [<ffffffff8105a9e0>] ? hrtimer_wakeup+0x0/0x21
[16087.359872]  [<ffffffff8105b4eb>] ? hrtimer_start_range_ns+0xf/0x11
[16087.359872]  [<ffffffff810f8b92>] poll_schedule_timeout+0x46/0x69
[16087.359872]  [<ffffffff810f985a>] do_select+0x5c1/0x605
[16087.359872]  [<ffffffff810f9299>] ? do_select+0x0/0x605
[16087.359872]  [<ffffffff81066948>] ? register_lock_class+0x20/0x327
[16087.359872]  [<ffffffff810f9d51>] ? __pollwait+0x0/0xcf
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff810698a9>] ? lock_release_non_nested+0x97/0x20d
[16087.359872]  [<ffffffff810c885b>] ? might_fault+0x57/0xa4
[16087.359872]  [<ffffffff810c885b>] ? might_fault+0x57/0xa4
[16087.359872]  [<ffffffff810c88a4>] ? might_fault+0xa0/0xa4
[16087.359872]  [<ffffffff810c885b>] ? might_fault+0x57/0xa4
[16087.359872]  [<ffffffff810f9a57>] core_sys_select+0x1b9/0x255
[16087.359872]  [<ffffffff810f98df>] ? core_sys_select+0x41/0x255
[16087.359872]  [<ffffffff810698a9>] ? lock_release_non_nested+0x97/0x20d
[16087.359872]  [<ffffffff810c88a4>] ? might_fault+0xa0/0xa4
[16087.359872]  [<ffffffff810c885b>] ? might_fault+0x57/0xa4
[16087.359872]  [<ffffffff810c885b>] ? might_fault+0x57/0xa4
[16087.359872]  [<ffffffff81060016>] ? ktime_get_ts+0xb1/0xbe
[16087.359872]  [<ffffffff810f9d29>] sys_select+0x94/0xbc
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] kthreadd      S 0000000000000000  4096     2      0 0x00000000
[16087.359872]  ffff88001f91df00 0000000000000046 0000000000000000
ffff88001f91decc
[16087.359872]  0000000000000000 000000000000def8 ffff88001f91dfd8
0000000000014180
[16087.359872]  ffff88001f91dfd8 0000000000014180 ffff88001f9110b0
ffff88001f152160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff810578cd>] kthreadd+0x83/0x109
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff8105784a>] ? kthreadd+0x0/0x109
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] migration/0   S 0000000000000000  6496     3      2 0x00000000
[16087.359872]  ffff88001f91fe50 0000000000000046 0000000000000000
ffffffff81039a5e
[16087.359872]  ffff880000000000 000000000000def8 ffff88001f91ffd8
0000000000014180
[16087.359872]  ffff88001f91ffd8 0000000000014180 ffff88001f912160
ffff88001f0b5370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81039a5e>] ? migration_thread+0x76/0x297
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff81039b84>] migration_thread+0x19c/0x297
[16087.359872]  [<ffffffff810399e8>] ? migration_thread+0x0/0x297
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] ksoftirqd/0   S 0000000000000000  6600     4      2 0x00000000
[16087.359872]  ffff88001f931e80 0000000000000046 ffffffff8104ad59
0000000000000000
[16087.359872]  ffff88001f930000 000000000000def8 ffff88001f931fd8
0000000000014180
[16087.359872]  ffff88001f931fd8 0000000000014180 ffff88001f913210
ffff88001e622160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff8104ad59>] ? run_timer_softirq+0x28a/0x2d3
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff810430b4>] run_ksoftirqd+0x52/0x121
[16087.359872]  [<ffffffff81043062>] ? run_ksoftirqd+0x0/0x121
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] migration/1   S 0000000000000001  6552     5      2 0x00000000
[16087.359872]  ffff88001f933e50 0000000000000046 0000000000000000
ffffffff81039a5e
[16087.359872]  ffff880000000000 000000000000def8 ffff88001f933fd8
0000000000014180
[16087.359872]  ffff88001f933fd8 0000000000014180 ffff88001f9142c0
ffff88001f980000
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81039a5e>] ? migration_thread+0x76/0x297
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff81039b84>] migration_thread+0x19c/0x297
[16087.359872]  [<ffffffff810399e8>] ? migration_thread+0x0/0x297
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] ksoftirqd/1   S 0000000000000000  6600     6      2 0x00000000
[16087.359872]  ffff88001f939e80 0000000000000046 0000000000000000
0000000000000000
[16087.359872]  ffff88001f938000 000000000000def8 ffff88001f939fd8
0000000000014180
[16087.359872]  ffff88001f939fd8 0000000000014180 ffff88001f915370
ffff88001dcb5370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff810430b4>] run_ksoftirqd+0x52/0x121
[16087.359872]  [<ffffffff81043062>] ? run_ksoftirqd+0x0/0x121
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] migration/2   S 0000000000000002  6552     7      2 0x00000000
[16087.359872]  ffff88001f95be50 0000000000000046 0000000000000000
ffffffff81039a5e
[16087.359872]  ffff880000000000 000000000000def8 ffff88001f95bfd8
0000000000014180
[16087.359872]  ffff88001f95bfd8 0000000000014180 ffff88001f960000
ffff88001f152160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81039a5e>] ? migration_thread+0x76/0x297
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff81039b84>] migration_thread+0x19c/0x297
[16087.359872]  [<ffffffff810399e8>] ? migration_thread+0x0/0x297
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] ksoftirqd/2   S 0000000000000000  6600     8      2 0x00000000
[16087.359872]  ffff88001f969e80 0000000000000046 ffffffff814903d9
ffff88001e7f9ea8
[16087.359872]  ffff880002714b40 000000000000def8 ffff88001f969fd8
0000000000014180
[16087.359872]  ffff88001f969fd8 0000000000014180 ffff88001f9610b0
ffff88001fa542c0
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff814903d9>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff810430b4>] run_ksoftirqd+0x52/0x121
[16087.359872]  [<ffffffff81043062>] ? run_ksoftirqd+0x0/0x121
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] migration/3   S 0000000000000003  6552     9      2 0x00000000
[16087.359872]  ffff88001f96de50 0000000000000046 0000000000000000
0000000000000046
[16087.359872]  0000000000000000 000000000000def8 ffff88001f96dfd8
0000000000014180
[16087.359872]  ffff88001f96dfd8 0000000000014180 ffff88001f963210
ffff88001f965370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff81039b84>] migration_thread+0x19c/0x297
[16087.359872]  [<ffffffff810399e8>] ? migration_thread+0x0/0x297
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] ksoftirqd/3   S 0000000000000000  6600    10      2 0x00000000
[16087.359872]  ffff88001f96fe80 0000000000000046 0000000000000000
0000000000000000
[16087.359872]  ffff88001f96e000 000000000000def8 ffff88001f96ffd8
0000000000014180
[16087.359872]  ffff88001f96ffd8 0000000000014180 ffff88001f9642c0
ffff88001e622160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff810430b4>] run_ksoftirqd+0x52/0x121
[16087.359872]  [<ffffffff81043062>] ? run_ksoftirqd+0x0/0x121
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] events/0      S ffff88001f966420  6152    11      2 0x00000000
[16087.359872]  ffff88001f977dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f966420 000000000000def8 ffff88001f977fd8
0000000000014180
[16087.359872]  ffff88001f977fd8 0000000000014180 ffff88001f966420
ffff88001f0b5370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff810c6802>] ? vmstat_update+0x0/0x3a
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] events/1      S ffff88001f980000  5776    12      2 0x00000000
[16087.359872]  ffff88001f989dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f980000 000000000000def8 ffff88001f989fd8
0000000000014180
[16087.359872]  ffff88001f989fd8 0000000000014180 ffff88001f980000
ffff88001dcb5370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff810c6802>] ? vmstat_update+0x0/0x3a
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] events/2      S ffff88001f9810b0  5704    13      2 0x00000000
[16087.359872]  ffff88001f98bdc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f9810b0 000000000000def8 ffff88001f98bfd8
0000000000014180
[16087.359872]  ffff88001f98bfd8 0000000000014180 ffff88001f9810b0
ffff88001f152160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff812acbd4>] ? flush_to_ldisc+0x0/0x1a0
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] events/3      S ffff88001f982160  5328    14      2 0x00000000
[16087.359872]  ffff88001f98ddc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f982160 000000000000def8 ffff88001f98dfd8
0000000000014180
[16087.359872]  ffff88001f98dfd8 0000000000014180 ffff88001f982160
ffff88001f965370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff813ab4ea>] ? neigh_periodic_work+0x0/0x185
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] cpuset        S 0000000000000000  6712    15      2 0x00000000
[16087.359872]  ffff88001f98fdc0 0000000000000046 ffff88001f98fd90
ffff88001f98fd8c
[16087.359872]  ffff88001f983210 000000000000def8 ffff88001f98ffd8
0000000000014180
[16087.359872]  ffff88001f98ffd8 0000000000014180 ffff88001f983210
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] khelper       S ffff88001f9842c0  5232    16      2 0x00000000
[16087.359872]  ffff88001f999dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f9842c0 000000000000def8 ffff88001f999fd8
0000000000014180
[16087.359872]  ffff88001f999fd8 0000000000014180 ffff88001f9842c0
ffff88001f962160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff81052790>] ? __call_usermodehelper+0x0/0x6c
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] netns         S 0000000000000000  6816    19      2 0x00000000
[16087.359872]  ffff88001fa35dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f985370 000000000000def8 ffff88001fa35fd8
0000000000014180
[16087.359872]  ffff88001fa35fd8 0000000000014180 ffff88001f985370
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] async/mgr     S 0000000000000000  6152    22      2 0x00000000
[16087.359872]  ffff88001fa3de60 0000000000000046 ffff880002714180
0000000000000001
[16087.359872]  ffff88001fa3ddd0 000000000000def8 ffff88001fa3dfd8
0000000000014180
[16087.359872]  ffff88001fa3dfd8 0000000000014180 ffff88001f986420
ffff88001f9842c0
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81039436>] ? try_to_wake_up+0x2d5/0x2e7
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff8105d934>] ? async_manager_thread+0x0/0xe8
[16087.359872]  [<ffffffff8105d9f2>] async_manager_thread+0xbe/0xe8
[16087.359872]  [<ffffffff81039448>] ? default_wake_function+0x0/0xf
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] sync_supers   S 0000000000000000  6616   155      2 0x00000000
[16087.359872]  ffff88001fa41e90 0000000000000046 ffff88001fa50000
0000000000000000
[16087.359872]  0000000000000002 000000000000def8 ffff88001fa41fd8
0000000000014180
[16087.359872]  ffff88001fa41fd8 0000000000014180 ffff88001fa50000
ffff88001f966420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff810c6c16>] ? bdi_sync_supers+0x0/0x51
[16087.359872]  [<ffffffff81490bdd>] ? _raw_spin_unlock+0x26/0x2b
[16087.359872]  [<ffffffff810ecf84>] ? sync_supers+0x13/0xa5
[16087.359872]  [<ffffffff810c6c16>] ? bdi_sync_supers+0x0/0x51
[16087.359872]  [<ffffffff810c6c50>] bdi_sync_supers+0x3a/0x51
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] bdi-default   S ffff88001f9abe40  5816   157      2 0x00000000
[16087.359872]  ffff88001f9abd80 0000000000000046 ffff88001fa510b0
ffffffff81490baf
[16087.359872]  ffffffff81b71f00 000000000000def8 ffff88001f9abfd8
0000000000014180
[16087.359872]  ffff88001f9abfd8 0000000000014180 ffff88001fa510b0
ffff88001f0b5370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81490baf>] ? _raw_spin_unlock_irqrestore+0x3f/0x47
[16087.359872]  [<ffffffff8104b617>] ? __mod_timer+0x141/0x153
[16087.359872]  [<ffffffff8104b45f>] ? init_timer_key+0x8d/0x99
[16087.359872]  [<ffffffff8148efbb>] schedule_timeout+0xa2/0xc9
[16087.359872]  [<ffffffff8104b095>] ? process_timeout+0x0/0xb
[16087.359872]  [<ffffffff81490b3b>] ? _raw_spin_unlock_bh+0x2f/0x33
[16087.359872]  [<ffffffff810c776a>] bdi_forker_task+0x1a8/0x2da
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff810c75c2>] ? bdi_forker_task+0x0/0x2da
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kblockd/0     S ffff88001fa52160  5856   159      2 0x00000000
[16087.359872]  ffff88001fa37dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fa52160 000000000000def8 ffff88001fa37fd8
0000000000014180
[16087.359872]  ffff88001fa37fd8 0000000000014180 ffff88001fa52160
ffff88001f0b5370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff81269971>] ? blk_unplug_work+0x0/0x4d
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kblockd/1     S ffff88001fa53210  5624   160      2 0x00000000
[16087.359872]  ffff88001fa3fdc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fa53210 000000000000def8 ffff88001fa3ffd8
0000000000014180
[16087.359872]  ffff88001fa3ffd8 0000000000014180 ffff88001fa53210
ffff88001dcb5370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff8127a542>] ? cfq_kick_queue+0x0/0x3c
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kblockd/2     S ffff88001fa542c0  5856   161      2 0x00000000
[16087.359872]  ffff88001fa3bdc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fa542c0 000000000000def8 ffff88001fa3bfd8
0000000000014180
[16087.359872]  ffff88001fa3bfd8 0000000000014180 ffff88001fa542c0
ffff88001f152160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff8127a542>] ? cfq_kick_queue+0x0/0x3c
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kblockd/3     S ffff88001fa55370  5576   162      2 0x00000000
[16087.359872]  ffff88001fa47dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fa55370 000000000000def8 ffff88001fa47fd8
0000000000014180
[16087.359872]  ffff88001fa47fd8 0000000000014180 ffff88001fa55370
ffff88001f155370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff8127a542>] ? cfq_kick_queue+0x0/0x3c
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kacpid        S ffff88001fa56420  6232   165      2 0x00000000
[16087.359872]  ffff88001f9afdc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fa56420 000000000000def8 ffff88001f9affd8
0000000000014180
[16087.359872]  ffff88001f9affd8 0000000000014180 ffff88001fa56420
ffff88001f910000
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff812d5785>] ? bind_to_cpu0+0x0/0x2f
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kacpi_notify  S ffff88001fa65370  6232   166      2 0x00000000
[16087.359872]  ffff88001fa69dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fa65370 000000000000def8 ffff88001fa69fd8
0000000000014180
[16087.359872]  ffff88001fa69fd8 0000000000014180 ffff88001fa65370
ffff88001fa642c0
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff812d5785>] ? bind_to_cpu0+0x0/0x2f
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kacpi_hotplug S ffff88001fa642c0  6232   167      2 0x00000000
[16087.359872]  ffff88001fa6bdc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fa642c0 000000000000def8 ffff88001fa6bfd8
0000000000014180
[16087.359872]  ffff88001fa6bfd8 0000000000014180 ffff88001fa642c0
ffff88001f910000
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff812d5785>] ? bind_to_cpu0+0x0/0x2f
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kseriod       S ffff88001fa87e78  5352   239      2 0x00000000
[16087.359872]  ffff88001fa87e40 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fa63210 000000000000def8 ffff88001fa87fd8
0000000000014180
[16087.359872]  ffff88001fa87fd8 0000000000014180 ffff88001fa63210
ffff88001f910000
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff8136408e>] serio_thread+0x2cd/0x30a
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81363dc1>] ? serio_thread+0x0/0x30a
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] rpciod/0      S 0000000000000000  6664   269      2 0x00000000
[16087.359872]  ffff88001f08ddc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fa62160 000000000000def8 ffff88001f08dfd8
0000000000014180
[16087.359872]  ffff88001f08dfd8 0000000000014180 ffff88001fa62160
ffffffff81a0e020
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] rpciod/1      S 0000000000000000  6664   270      2 0x00000000
[16087.359872]  ffff88001f08fdc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fa60000 000000000000def8 ffff88001f08ffd8
0000000000014180
[16087.359872]  ffff88001f08ffd8 0000000000014180 ffff88001fa60000
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] rpciod/2      S 0000000000000000  6664   271      2 0x00000000
[16087.359872]  ffff88001fa7fdc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fa610b0 000000000000def8 ffff88001fa7ffd8
0000000000014180
[16087.359872]  ffff88001fa7ffd8 0000000000014180 ffff88001fa610b0
ffff88001f962160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] rpciod/3      S 0000000000000000  6664   272      2 0x00000000
[16087.359872]  ffff88001f091dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fa66420 000000000000def8 ffff88001f091fd8
0000000000014180
[16087.359872]  ffff88001f091fd8 0000000000014180 ffff88001fa66420
ffff88001f910000
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kswapd0       S 00000000ffffffff  4352   319      2 0x00000000
[16087.359872]  ffff88001f0bdd60 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fb58000 000000000000def8 ffff88001f0bdfd8
0000000000014180
[16087.359872]  ffff88001f0bdfd8 0000000000014180 ffff88001fb58000
ffff88001f152160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff810bf06c>] kswapd+0x853/0x867
[16087.359872]  [<ffffffff810bc216>] ? isolate_pages_global+0x0/0x1f2
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff810be819>] ? kswapd+0x0/0x867
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] aio/0         S 0000000000000000  6664   375      2 0x00000000
[16087.359872]  ffff88001f0bfdc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fb590b0 000000000000def8 ffff88001f0bffd8
0000000000014180
[16087.359872]  ffff88001f0bffd8 0000000000014180 ffff88001fb590b0
ffffffff81a0e020
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] aio/1         S 0000000000000000  6664   376      2 0x00000000
[16087.359872]  ffff88001f107dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fb5a160 000000000000def8 ffff88001f107fd8
0000000000014180
[16087.359872]  ffff88001f107fd8 0000000000014180 ffff88001fb5a160
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] aio/2         S 0000000000000000  6664   377      2 0x00000000
[16087.359872]  ffff88001f143dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fb5b210 000000000000def8 ffff88001f143fd8
0000000000014180
[16087.359872]  ffff88001f143fd8 0000000000014180 ffff88001fb5b210
ffff88001f962160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] aio/3         S 0000000000000000  6664   378      2 0x00000000
[16087.359872]  ffff88001f105dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fb5c2c0 000000000000def8 ffff88001f105fd8
0000000000014180
[16087.359872]  ffff88001f105fd8 0000000000014180 ffff88001fb5c2c0
ffff88001f965370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] nfsiod        S 0000000000000000  6664   390      2 0x00000000
[16087.359872]  ffff88001f0ffdc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001fb5d370 000000000000def8 ffff88001f0fffd8
0000000000014180
[16087.359872]  ffff88001f0fffd8 0000000000014180 ffff88001fb5d370
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kslowd000     S ffff88001fb5e420  6768   399      2 0x00000000
[16087.359872]  ffff88001f0fbe40 0000000000000046 0000000000000000
ffffffff81057e21
[16087.359872]  ffff88001fb5e420 000000000000def8 ffff88001f0fbfd8
0000000000014180
[16087.359872]  ffff88001f0fbfd8 0000000000014180 ffff88001fb5e420
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e21>] ? prepare_to_wait_exclusive+0x1e/0x7a
[16087.359872]  [<ffffffff81057e72>] ? prepare_to_wait_exclusive+0x6f/0x7a
[16087.359872]  [<ffffffff810aaced>] slow_work_thread+0x140/0x2da
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff810aabad>] ? slow_work_thread+0x0/0x2da
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kslowd001     S ffff88001f0b42c0  6792   400      2 0x00000000
[16087.359872]  ffff88001f16fe40 0000000000000046 0000000000000000
ffffffff81057e21
[16087.359872]  ffff88001f0b42c0 000000000000def8 ffff88001f16ffd8
0000000000014180
[16087.359872]  ffff88001f16ffd8 0000000000014180 ffff88001f0b42c0
ffff88001f962160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e21>] ? prepare_to_wait_exclusive+0x1e/0x7a
[16087.359872]  [<ffffffff81057e72>] ? prepare_to_wait_exclusive+0x6f/0x7a
[16087.359872]  [<ffffffff810aaced>] slow_work_thread+0x140/0x2da
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff810aabad>] ? slow_work_thread+0x0/0x2da
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfs_mru_cache S 0000000000000000  6664   409      2 0x00000000
[16087.359872]  ffff88001f195dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f0b3210 000000000000def8 ffff88001f195fd8
0000000000014180
[16087.359872]  ffff88001f195fd8 0000000000014180 ffff88001f0b3210
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfslogd/0     S 0000000000000000  6664   410      2 0x00000000
[16087.359872]  ffff88001f197dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f0b2160 000000000000def8 ffff88001f197fd8
0000000000014180
[16087.359872]  ffff88001f197fd8 0000000000014180 ffff88001f0b2160
ffffffff81a0e020
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfslogd/1     S ffff88001f0b10b0  6376   411      2 0x00000000
[16087.359872]  ffff88001f1a1dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f0b10b0 000000000000def8 ffff88001f1a1fd8
0000000000014180
[16087.359872]  ffff88001f1a1fd8 0000000000014180 ffff88001f0b10b0
ffff88001fa53210
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff8123f271>] ? xfs_buf_iodone_work+0x0/0xa1
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfslogd/2     S ffff88001f0b0000  5952   412      2 0x00000000
[16087.359872]  ffff88001f1a3dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f0b0000 000000000000def8 ffff88001f1a3fd8
0000000000014180
[16087.359872]  ffff88001f1a3fd8 0000000000014180 ffff88001f0b0000
ffff88001fa542c0
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff8123f271>] ? xfs_buf_iodone_work+0x0/0xa1
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfslogd/3     S ffff88001f0b6420  6256   413      2 0x00000000
[16087.359872]  ffff88001f1a5dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f0b6420 000000000000def8 ffff88001f1a5fd8
0000000000014180
[16087.359872]  ffff88001f1a5fd8 0000000000014180 ffff88001f0b6420
ffff88001fa55370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff8123f271>] ? xfs_buf_iodone_work+0x0/0xa1
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfsdatad/0    D ffff88001f0b5370  6216   414      2 0x00000000
[16087.359872]  ffff88001f1a7d00 0000000000000046 0000000000000000
ffffffff81490baf
[16087.359872]  ffffffff81b71f00 000000000000def8 ffff88001f1a7fd8
0000000000014180
[16087.359872]  ffff88001f1a7fd8 0000000000014180 ffff88001f0b5370
ffffffff81a0e020
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81490baf>] ? _raw_spin_unlock_irqrestore+0x3f/0x47
[16087.359872]  [<ffffffff8104b617>] ? __mod_timer+0x141/0x153
[16087.359872]  [<ffffffff8104b45f>] ? init_timer_key+0x8d/0x99
[16087.359872]  [<ffffffff8148efbb>] schedule_timeout+0xa2/0xc9
[16087.359872]  [<ffffffff8104b095>] ? process_timeout+0x0/0xb
[16087.359872]  [<ffffffff81054688>] ? queue_work+0x1a/0x1c
[16087.359872]  [<ffffffff8148effb>] schedule_timeout_uninterruptible+0x19/0x1b
[16087.359872]  [<ffffffff8123bc95>] xfs_end_io+0x7d/0x8c
[16087.359872]  [<ffffffff81053dfb>] worker_thread+0x22c/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff8123bc18>] ? xfs_end_io+0x0/0x8c
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfsdatad/1    S ffff88001f153210  6216   415      2 0x00000000
[16087.359872]  ffff88001f1b1dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f153210 000000000000def8 ffff88001f1b1fd8
0000000000014180
[16087.359872]  ffff88001f1b1fd8 0000000000014180 ffff88001f153210
ffff88001e5e42c0
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff8123bc18>] ? xfs_end_io+0x0/0x8c
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfsdatad/2    D ffff88001f152160  6112   416      2 0x00000000
[16087.359872]  ffff88001f1b3d00 0000000000000046 0000000000000000
ffffffff81490baf
[16087.359872]  ffff88001f95c000 000000000000def8 ffff88001f1b3fd8
0000000000014180
[16087.359872]  ffff88001f1b3fd8 0000000000014180 ffff88001f152160
ffff88001f962160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81490baf>] ? _raw_spin_unlock_irqrestore+0x3f/0x47
[16087.359872]  [<ffffffff8104b617>] ? __mod_timer+0x141/0x153
[16087.359872]  [<ffffffff8104b45f>] ? init_timer_key+0x8d/0x99
[16087.359872]  [<ffffffff8148efbb>] schedule_timeout+0xa2/0xc9
[16087.359872]  [<ffffffff8104b095>] ? process_timeout+0x0/0xb
[16087.359872]  [<ffffffff81054688>] ? queue_work+0x1a/0x1c
[16087.359872]  [<ffffffff8148effb>] schedule_timeout_uninterruptible+0x19/0x1b
[16087.359872]  [<ffffffff8123bc95>] xfs_end_io+0x7d/0x8c
[16087.359872]  [<ffffffff81053dfb>] worker_thread+0x22c/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff8123bc18>] ? xfs_end_io+0x0/0x8c
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfsdatad/3    S ffff88001f1510b0  6176   417      2 0x00000000
[16087.359872]  ffff88001f1b5dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f1510b0 000000000000def8 ffff88001f1b5fd8
0000000000014180
[16087.359872]  ffff88001f1b5fd8 0000000000014180 ffff88001f1510b0
ffff88001fb58000
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81053da4>] ? worker_thread+0x1d5/0x332
[16087.359872]  [<ffffffff8123bc18>] ? xfs_end_io+0x0/0x8c
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfsconvertd/0 S 0000000000000000  6664   418      2 0x00000000
[16087.359872]  ffff88001f1b7dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f1b8000 000000000000def8 ffff88001f1b7fd8
0000000000014180
[16087.359872]  ffff88001f1b7fd8 0000000000014180 ffff88001f1b8000
ffffffff81a0e020
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfsconvertd/1 S 0000000000000000  6664   419      2 0x00000000
[16087.359872]  ffff88001f1c1dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f1b90b0 000000000000def8 ffff88001f1c1fd8
0000000000014180
[16087.359872]  ffff88001f1c1fd8 0000000000014180 ffff88001f1b90b0
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfsconvertd/2 S 0000000000000000  6664   420      2 0x00000000
[16087.359872]  ffff88001f1c3dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f1ba160 000000000000def8 ffff88001f1c3fd8
0000000000014180
[16087.359872]  ffff88001f1c3fd8 0000000000014180 ffff88001f1ba160
ffff88001f962160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfsconvertd/3 S 0000000000000000  6664   421      2 0x00000000
[16087.359872]  ffff88001f1c5dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f1bb210 000000000000def8 ffff88001f1c5fd8
0000000000014180
[16087.359872]  ffff88001f1c5fd8 0000000000014180 ffff88001f1bb210
ffff88001f965370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] crypto/0      S 0000000000000000  6664   423      2 0x00000000
[16087.359872]  ffff88001f135dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f1bc2c0 000000000000def8 ffff88001f135fd8
0000000000014180
[16087.359872]  ffff88001f135fd8 0000000000014180 ffff88001f1bc2c0
ffffffff81a0e020
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] crypto/1      S 0000000000000000  6664   424      2 0x00000000
[16087.359872]  ffff88001fa07dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f1bd370 000000000000def8 ffff88001fa07fd8
0000000000014180
[16087.359872]  ffff88001fa07fd8 0000000000014180 ffff88001f1bd370
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] crypto/2      S 0000000000000000  6664   425      2 0x00000000
[16087.359872]  ffff88001f191dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f1be420 000000000000def8 ffff88001f191fd8
0000000000014180
[16087.359872]  ffff88001f191fd8 0000000000014180 ffff88001f1be420
ffff88001f962160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] crypto/3      S 0000000000000000  6664   426      2 0x00000000
[16087.359872]  ffff88001f1c7dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f150000 000000000000def8 ffff88001f1c7fd8
0000000000014180
[16087.359872]  ffff88001f1c7fd8 0000000000014180 ffff88001f150000
ffff88001f965370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kpsmoused     S 0000000000000000  6664   575      2 0x00000000
[16087.359872]  ffff88001f22ddc0 0000000000000046 ffff88001f22dd90
ffff88001f22dd8c
[16087.359872]  ffff88001f1e6420 000000000000def8 ffff88001f22dfd8
0000000000014180
[16087.359872]  ffff88001f22dfd8 0000000000014180 ffff88001f1e6420
ffff88001f962160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] edac-poller   S 0000000000000000  6664   583      2 0x00000000
[16087.359872]  ffff88001f283dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f1e5370 000000000000def8 ffff88001f283fd8
0000000000014180
[16087.359872]  ffff88001f283fd8 0000000000014180 ffff88001f1e5370
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kondemand/0   S 0000000000000000  6664   586      2 0x00000000
[16087.359872]  ffff88001f289dc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f1e3210 000000000000def8 ffff88001f289fd8
0000000000014180
[16087.359872]  ffff88001f289fd8 0000000000014180 ffff88001f1e3210
ffffffff81a0e020
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kondemand/1   S 0000000000000000  6664   587      2 0x00000000
[16087.359872]  ffff88001f28bdc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f1e2160 000000000000def8 ffff88001f28bfd8
0000000000014180
[16087.359872]  ffff88001f28bfd8 0000000000014180 ffff88001f1e2160
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kondemand/2   S 0000000000000000  6664   588      2 0x00000000
[16087.359872]  ffff88001f28ddc0 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001f1e10b0 000000000000def8 ffff88001f28dfd8
0000000000014180
[16087.359872]  ffff88001f28dfd8 0000000000014180 ffff88001f1e10b0
ffff88001f962160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kondemand/3   S 0000000000000000  6664   589      2 0x00000000
[16087.359872]  ffff88001f28fdc0 0000000000000046 ffff88001f28fd90
ffff88001f28fd8c
[16087.359872]  ffff88001f1e0000 000000000000def8 ffff88001f28ffd8
0000000000014180
[16087.359872]  ffff88001f28ffd8 0000000000014180 ffff88001f1e0000
ffff88001f965370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81053ccc>] worker_thread+0xfd/0x332
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81053bcf>] ? worker_thread+0x0/0x332
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] vballoon      S ffff88001ec37e60  6736   605      2 0x00000000
[16087.359872]  ffff88001ec37e20 0000000000000046 0000000000000000
ffffffff81057e9b
[16087.359872]  ffff88001ec38000 000000000000def8 ffff88001ec37fd8
0000000000014180
[16087.359872]  ffff88001ec37fd8 0000000000014180 ffff88001ec38000
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81057e9b>] ? prepare_to_wait+0x1e/0x7b
[16087.359872]  [<ffffffff8138e267>] ? vp_get+0x48/0x5d
[16087.359872]  [<ffffffff8138f1b7>] balloon+0x143/0x2c0
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff8138f074>] ? balloon+0x0/0x2c0
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] kjournald     S ffff88001f09e970  3984   620      2 0x00000000
[16087.359872]  ffff88001ec4de20 0000000000000046 ffff88001ec4dd90
ffffffff81067901
[16087.359872]  ffff88001ec390b0 000000000000def8 ffff88001ec4dfd8
0000000000014180
[16087.359872]  ffff88001ec4dfd8 0000000000014180 ffff88001ec390b0
ffff88001f0b5370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81067901>] ? mark_held_locks+0x4d/0x6b
[16087.359872]  [<ffffffff811602bb>] kjournald+0x1cf/0x25a
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff811600ec>] ? kjournald+0x0/0x25a
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] udevd         S 0000000000000000  5616   711      1 0x00000000
[16087.359872]  ffff88001ed73a58 0000000000000046 0000000000000000
000000000000004c
[16087.359872]  ffff88001ed73a88 000000000000def8 ffff88001ed73fd8
0000000000014180
[16087.359872]  ffff88001ed73fd8 0000000000014180 ffff88001edb2160
ffffffff81a0e020
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff8148f8c5>] schedule_hrtimeout_range+0x3f/0xf7
[16087.359872]  [<ffffffff81490bdd>] ? _raw_spin_unlock+0x26/0x2b
[16087.359872]  [<ffffffff81423dca>] ? unix_peer_get+0x19/0x3d
[16087.359872]  [<ffffffff81423de6>] ? unix_peer_get+0x35/0x3d
[16087.359872]  [<ffffffff810f8b92>] poll_schedule_timeout+0x46/0x69
[16087.359872]  [<ffffffff810f8fac>] do_sys_poll+0x36a/0x40b
[16087.359872]  [<ffffffff810f9d51>] ? __pollwait+0x0/0xcf
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810b7d5a>] ? __pagevec_free+0x70/0x85
[16087.359872]  [<ffffffff810bb1b8>] ? release_pages+0x1aa/0x1bc
[16087.359872]  [<ffffffff810d32b2>] ? anon_vma_unlink+0x1b/0x71
[16087.359872]  [<ffffffff81067901>] ? mark_held_locks+0x4d/0x6b
[16087.359872]  [<ffffffff810e4920>] ? kmem_cache_free+0x99/0xd3
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff810ce526>] ? remove_vma+0x7f/0x87
[16087.359872]  [<ffffffff810ce526>] ? remove_vma+0x7f/0x87
[16087.359872]  [<ffffffff81002a9c>] ? sysret_check+0x27/0x62
[16087.359872]  [<ffffffff810f91e2>] sys_poll+0x50/0xba
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] dhcpcd        S 0000000000000000  3536  2636      1 0x00000000
[16087.359872]  ffff88001e785a58 0000000000000046 0000000000000000
0000000000000282
[16087.359872]  0000000000000040 000000000000def8 ffff88001e785fd8
0000000000014180
[16087.359872]  ffff88001e785fd8 0000000000014180 ffff88001ec3c2c0
ffff88001f965370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff8148f948>] schedule_hrtimeout_range+0xc2/0xf7
[16087.359872]  [<ffffffff8105a9e0>] ? hrtimer_wakeup+0x0/0x21
[16087.359872]  [<ffffffff8105b4eb>] ? hrtimer_start_range_ns+0xf/0x11
[16087.359872]  [<ffffffff810f9e17>] ? __pollwait+0xc6/0xcf
[16087.359872]  [<ffffffff810f8b92>] poll_schedule_timeout+0x46/0x69
[16087.359872]  [<ffffffff810f8fac>] do_sys_poll+0x36a/0x40b
[16087.359872]  [<ffffffff810f9d51>] ? __pollwait+0x0/0xcf
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff811154ca>] ? fsnotify_clear_marks_by_inode+0x2b/0xe0
[16087.359872]  [<ffffffff810e4920>] ? kmem_cache_free+0x99/0xd3
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff810e4920>] ? kmem_cache_free+0x99/0xd3
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff810fa2a0>] ? __d_free+0x51/0x55
[16087.359872]  [<ffffffff810fa2d6>] ? d_free+0x32/0x4b
[16087.359872]  [<ffffffff81101451>] ? mntput_no_expire+0x24/0xf8
[16087.359872]  [<ffffffff81060016>] ? ktime_get_ts+0xb1/0xbe
[16087.359872]  [<ffffffff810f8b31>] ? poll_select_set_timeout+0x5c/0x77
[16087.359872]  [<ffffffff810f91e2>] sys_poll+0x50/0xba
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] login         S ffff88001edb42c0  4896  2847      1 0x00000000
[16087.359872]  ffff88001dc61e58 0000000000000046 0000000000000000
0000000000000000
[16087.359872]  ffff88001dc61e08 000000000000def8 ffff88001dc61fd8
0000000000014180
[16087.359872]  ffff88001dc61fd8 0000000000014180 ffff88001edb42c0
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81040671>] do_wait+0x1bd/0x22b
[16087.359872]  [<ffffffff81040785>] sys_wait4+0xa6/0xc3
[16087.359872]  [<ffffffff8103ee89>] ? child_wait_callback+0x0/0x53
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] agetty        S ffff88001f9ec290  5784  2848      1 0x00000000
[16087.359872]  ffff88001dcddd08 0000000000000046 0000000000000000
0000000000000000
[16087.359872]  ffff88001edb6420 000000000000def8 ffff88001dcddfd8
0000000000014180
[16087.359872]  ffff88001dcddfd8 0000000000014180 ffff88001edb6420
ffff88001f962160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff8105497d>] ? flush_work+0x2b/0xf2
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff8148ef3b>] schedule_timeout+0x22/0xc9
[16087.359872]  [<ffffffff8104b009>] ? del_timer_sync+0x73/0x84
[16087.359872]  [<ffffffff8104af96>] ? del_timer_sync+0x0/0x84
[16087.359872]  [<ffffffff81054c01>] ? flush_delayed_work+0x50/0x55
[16087.359872]  [<ffffffff812a97d8>] n_tty_read+0x517/0x7f3
[16087.359872]  [<ffffffff81039448>] ? default_wake_function+0x0/0xf
[16087.359872]  [<ffffffff812a4e53>] tty_read+0x7d/0xba
[16087.359872]  [<ffffffff810eb75e>] vfs_read+0xa8/0x102
[16087.359872]  [<ffffffff810eb87c>] sys_read+0x47/0x70
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] agetty        S ffff88001f9bd350  5320  2849      1 0x00000000
[16087.359872]  ffff88001f231d08 0000000000000046 0000000000000000
0000000000000000
[16087.359872]  ffff88001edb3210 000000000000def8 ffff88001f231fd8
0000000000014180
[16087.359872]  ffff88001f231fd8 0000000000014180 ffff88001edb3210
ffff88001f962160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff8105497d>] ? flush_work+0x2b/0xf2
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff8148ef3b>] schedule_timeout+0x22/0xc9
[16087.359872]  [<ffffffff8104b009>] ? del_timer_sync+0x73/0x84
[16087.359872]  [<ffffffff8104af96>] ? del_timer_sync+0x0/0x84
[16087.359872]  [<ffffffff81054c01>] ? flush_delayed_work+0x50/0x55
[16087.359872]  [<ffffffff812a97d8>] n_tty_read+0x517/0x7f3
[16087.359872]  [<ffffffff81039448>] ? default_wake_function+0x0/0xf
[16087.359872]  [<ffffffff812a4e53>] tty_read+0x7d/0xba
[16087.359872]  [<ffffffff810eb75e>] vfs_read+0xa8/0x102
[16087.359872]  [<ffffffff810eb87c>] sys_read+0x47/0x70
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] agetty        S ffff88001f9bd360  5224  2850      1 0x00000000
[16087.359872]  ffff88001ed71d08 0000000000000046 0000000000000000
0000000000000000
[16087.359872]  ffff88001edb0000 000000000000def8 ffff88001ed71fd8
0000000000014180
[16087.359872]  ffff88001ed71fd8 0000000000014180 ffff88001edb0000
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff8148ef3b>] schedule_timeout+0x22/0xc9
[16087.359872]  [<ffffffff8104b009>] ? del_timer_sync+0x73/0x84
[16087.359872]  [<ffffffff8104af96>] ? del_timer_sync+0x0/0x84
[16087.359872]  [<ffffffff81054c01>] ? flush_delayed_work+0x50/0x55
[16087.359872]  [<ffffffff812a97d8>] n_tty_read+0x517/0x7f3
[16087.359872]  [<ffffffff81039448>] ? default_wake_function+0x0/0xf
[16087.359872]  [<ffffffff812a4e53>] tty_read+0x7d/0xba
[16087.359872]  [<ffffffff810eb75e>] vfs_read+0xa8/0x102
[16087.359872]  [<ffffffff810eb87c>] sys_read+0x47/0x70
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] agetty        S ffff88001f21c5e0  5224  2851      1 0x00000000
[16087.359872]  ffff88001e425d08 0000000000000046 0000000000000000
0000000000000000
[16087.359872]  ffff88001edb10b0 000000000000def8 ffff88001e425fd8
0000000000014180
[16087.359872]  ffff88001e425fd8 0000000000014180 ffff88001edb10b0
ffff88001f965370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff8105497d>] ? flush_work+0x2b/0xf2
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff8148ef3b>] schedule_timeout+0x22/0xc9
[16087.359872]  [<ffffffff8104b009>] ? del_timer_sync+0x73/0x84
[16087.359872]  [<ffffffff8104af96>] ? del_timer_sync+0x0/0x84
[16087.359872]  [<ffffffff81054c01>] ? flush_delayed_work+0x50/0x55
[16087.359872]  [<ffffffff812a97d8>] n_tty_read+0x517/0x7f3
[16087.359872]  [<ffffffff81039448>] ? default_wake_function+0x0/0xf
[16087.359872]  [<ffffffff812a4e53>] tty_read+0x7d/0xba
[16087.359872]  [<ffffffff810eb75e>] vfs_read+0xa8/0x102
[16087.359872]  [<ffffffff810eb87c>] sys_read+0x47/0x70
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] agetty        S ffff88001fa78a80  5784  2852      1 0x00000000
[16087.359872]  ffff88001dd1bd08 0000000000000046 0000000000000000
0000000000000000
[16087.359872]  ffff88001edb5370 000000000000def8 ffff88001dd1bfd8
0000000000014180
[16087.359872]  ffff88001dd1bfd8 0000000000014180 ffff88001edb5370
ffffffff81a0e020
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff8148ef3b>] schedule_timeout+0x22/0xc9
[16087.359872]  [<ffffffff8104b009>] ? del_timer_sync+0x73/0x84
[16087.359872]  [<ffffffff8104af96>] ? del_timer_sync+0x0/0x84
[16087.359872]  [<ffffffff81054c01>] ? flush_delayed_work+0x50/0x55
[16087.359872]  [<ffffffff812a97d8>] n_tty_read+0x517/0x7f3
[16087.359872]  [<ffffffff81039448>] ? default_wake_function+0x0/0xf
[16087.359872]  [<ffffffff812a4e53>] tty_read+0x7d/0xba
[16087.359872]  [<ffffffff810eb75e>] vfs_read+0xa8/0x102
[16087.359872]  [<ffffffff810eb87c>] sys_read+0x47/0x70
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] bash          R  running task     3968  2853   2847 0x00000000
[16087.359872]  332e37383036315b ffffffff8148e017 0000003000000010
ffff88001eda3d98
[16087.359872]  ffff88001eda3cd8 0000000137383935 ffff88001eda2000
ffff88001eda2000
[16087.359872]  ffff880002780000 0000000000000000 0000000000000082
0000000000000004
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff810057a3>] ? dump_trace+0x2fe/0x30f
[16087.359872]  [<ffffffff8100618f>] show_trace_log_lvl+0x4c/0x58
[16087.359872]  [<ffffffff8100528f>] show_stack_log_lvl+0xff/0x10e
[16087.359872]  [<ffffffff81006141>] show_stack+0x17/0x19
[16087.359872]  [<ffffffff81036d04>] sched_show_task+0xc0/0xc5
[16087.359872]  [<ffffffff81036d59>] show_state_filter+0x50/0x9b
[16087.359872]  [<ffffffff812ba8d3>] sysrq_handle_showstate+0xb/0xd
[16087.359872]  [<ffffffff812bac43>] __handle_sysrq+0xbb/0x158
[16087.359872]  [<ffffffff812bace0>] ? write_sysrq_trigger+0x0/0x44
[16087.359872]  [<ffffffff812bad19>] write_sysrq_trigger+0x39/0x44
[16087.359872]  [<ffffffff8113397a>] proc_reg_write+0x84/0xa1
[16087.359872]  [<ffffffff810eb56f>] vfs_write+0xab/0x105
[16087.359872]  [<ffffffff810eb68d>] sys_write+0x47/0x70
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] xfsbufd       S ffff88001eecde80  6208  2861      2 0x00000000
[16087.359872]  ffff88001eecddd0 0000000000000046 0000000000000000
ffffffff81490baf
[16087.359872]  ffff88001f970000 000000000000def8 ffff88001eecdfd8
0000000000014180
[16087.359872]  ffff88001eecdfd8 0000000000014180 ffff88001e5890b0
ffff88001f965370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81490baf>] ? _raw_spin_unlock_irqrestore+0x3f/0x47
[16087.359872]  [<ffffffff8104b617>] ? __mod_timer+0x141/0x153
[16087.359872]  [<ffffffff8104b45f>] ? init_timer_key+0x8d/0x99
[16087.359872]  [<ffffffff8148efbb>] schedule_timeout+0xa2/0xc9
[16087.359872]  [<ffffffff8104b095>] ? process_timeout+0x0/0xb
[16087.359872]  [<ffffffff8148f031>] schedule_timeout_interruptible+0x19/0x1b
[16087.359872]  [<ffffffff8123f1d7>] xfsbufd+0x73/0x10d
[16087.359872]  [<ffffffff8123f164>] ? xfsbufd+0x0/0x10d
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfsaild       S 0000000000000000  6440  2862      2 0x00000000
[16087.359872]  ffff88001dc81de0 0000000000000046 ffff88001e58c2c0
ffffffff81490baf
[16087.359872]  ffffffff81b71f00 000000000000def8 ffff88001dc81fd8
0000000000014180
[16087.359872]  ffff88001dc81fd8 0000000000014180 ffff88001e58c2c0
ffff88001f0b5370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81490baf>] ? _raw_spin_unlock_irqrestore+0x3f/0x47
[16087.359872]  [<ffffffff8104b617>] ? __mod_timer+0x141/0x153
[16087.359872]  [<ffffffff8104b45f>] ? init_timer_key+0x8d/0x99
[16087.359872]  [<ffffffff8148efbb>] schedule_timeout+0xa2/0xc9
[16087.359872]  [<ffffffff8104b095>] ? process_timeout+0x0/0xb
[16087.359872]  [<ffffffff8148f031>] schedule_timeout_interruptible+0x19/0x1b
[16087.359872]  [<ffffffff81245c5c>] xfsaild+0x3f/0x84
[16087.359872]  [<ffffffff81245c1d>] ? xfsaild+0x0/0x84
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] xfssyncd      S 0000000000007530  5664  2863      2 0x00000000
[16087.359872]  ffff88001ef3dda0 0000000000000046 0000000000000000
ffffffff81490baf
[16087.359872]  ffff88001f934000 000000000000def8 ffff88001ef3dfd8
0000000000014180
[16087.359872]  ffff88001ef3dfd8 0000000000014180 ffff88001e58e420
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81490baf>] ? _raw_spin_unlock_irqrestore+0x3f/0x47
[16087.359872]  [<ffffffff8104b617>] ? __mod_timer+0x141/0x153
[16087.359872]  [<ffffffff8104b45f>] ? init_timer_key+0x8d/0x99
[16087.359872]  [<ffffffff8148efbb>] schedule_timeout+0xa2/0xc9
[16087.359872]  [<ffffffff8104b095>] ? process_timeout+0x0/0xb
[16087.359872]  [<ffffffff8148f031>] schedule_timeout_interruptible+0x19/0x1b
[16087.359872]  [<ffffffff81245fb3>] xfssyncd+0x80/0x1bd
[16087.359872]  [<ffffffff81245f33>] ? xfssyncd+0x0/0x1bd
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] screen        S 0000000000000000  5008  2876      1 0x00000000
[16087.359872]  ffff88001f2d1908 0000000000000046 0000000000000000
ffffffff81067b93
[16087.359872]  ffff88001f2d1878 000000000000def8 ffff88001f2d1fd8
0000000000014180
[16087.359872]  ffff88001f2d1fd8 0000000000014180 ffff88001e58d370
ffff88001f965370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff8104af88>] ? try_to_del_timer_sync+0x66/0x74
[16087.359872]  [<ffffffff8148f8c5>] schedule_hrtimeout_range+0x3f/0xf7
[16087.359872]  [<ffffffff812ac13e>] ? put_ldisc+0x91/0x98
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff810f8b92>] poll_schedule_timeout+0x46/0x69
[16087.359872]  [<ffffffff810f985a>] do_select+0x5c1/0x605
[16087.359872]  [<ffffffff810f9299>] ? do_select+0x0/0x605
[16087.359872]  [<ffffffff810f9d51>] ? __pollwait+0x0/0xcf
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9a57>] core_sys_select+0x1b9/0x255
[16087.359872]  [<ffffffff810f98df>] ? core_sys_select+0x41/0x255
[16087.359872]  [<ffffffff810698a9>] ? lock_release_non_nested+0x97/0x20d
[16087.359872]  [<ffffffff810698a9>] ? lock_release_non_nested+0x97/0x20d
[16087.359872]  [<ffffffff81490b6a>] ? _raw_spin_unlock_irq+0x2b/0x31
[16087.359872]  [<ffffffff810c885b>] ? might_fault+0x57/0xa4
[16087.359872]  [<ffffffff810c885b>] ? might_fault+0x57/0xa4
[16087.359872]  [<ffffffff810f9d29>] sys_select+0x94/0xbc
[16087.359872]  [<ffffffff814903d9>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] bash          S ffff88001f1542c0  5384  2877   2876 0x00000000
[16087.359872]  ffff88001dc17e58 0000000000000046 0000000000000000
0000000000000000
[16087.359872]  ffff88001dc17e08 000000000000def8 ffff88001dc17fd8
0000000000014180
[16087.359872]  ffff88001dc17fd8 0000000000014180 ffff88001f1542c0
ffff88001f965370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81040671>] do_wait+0x1bd/0x22b
[16087.359872]  [<ffffffff810c885b>] ? might_fault+0x57/0xa4
[16087.359872]  [<ffffffff81040785>] sys_wait4+0xa6/0xc3
[16087.359872]  [<ffffffff8103ee89>] ? child_wait_callback+0x0/0x53
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] test-mysqld   S 0000000000000000  5096  2881   2877 0x00000000
[16087.359872]  ffff88001ee51bf8 0000000000000046 0000000000000000
ffffffff8106a8a6
[16087.359872]  0000000100000000 000000000000def8 ffff88001ee51fd8
0000000000014180
[16087.359872]  ffff88001ee51fd8 0000000000014180 ffff88001dcb42c0
ffff88001f962160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff8106a8a6>] ? queue_lock+0x57/0x5f
[16087.359872]  [<ffffffff81490bdd>] ? _raw_spin_unlock+0x26/0x2b
[16087.359872]  [<ffffffff8106ab41>] futex_wait_queue_me+0x82/0x9a
[16087.359872]  [<ffffffff8106ac9c>] futex_wait+0x143/0x2b1
[16087.359872]  [<ffffffff810325c3>] ? select_task_rq_fair+0x7a7/0x81e
[16087.359872]  [<ffffffff8106c958>] do_futex+0x98/0x9a2
[16087.359872]  [<ffffffff81066f14>] ? trace_hardirqs_off+0xd/0xf
[16087.359872]  [<ffffffff81490baf>] ? _raw_spin_unlock_irqrestore+0x3f/0x47
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff8103a8da>] ? wake_up_new_task+0xd4/0xe2
[16087.359872]  [<ffffffff8103cbf3>] ? do_fork+0x279/0x33e
[16087.359872]  [<ffffffff812873d8>] ? call_rwsem_wake+0x18/0x30
[16087.359872]  [<ffffffff8106d375>] sys_futex+0x113/0x131
[16087.359872]  [<ffffffff814903d9>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] test-mysqld   D 0000000000000002  3440  2882   2877 0x00000000
[16087.359872]  ffff88001edfba88 0000000000000046 ffff88001edfba08
ffffffff810676b1
[16087.359872]  ffff88001ec1e420 000000000000def8 ffff88001edfbfd8
0000000000014180
[16087.359872]  ffff88001edfbfd8 0000000000014180 ffff88001ec1e420
ffff88001f152160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff81060016>] ? ktime_get_ts+0xb1/0xbe
[16087.359872]  [<ffffffff8148ec26>] io_schedule+0x7b/0xc2
[16087.359872]  [<ffffffff810b2a58>] sync_page+0x44/0x48
[16087.359872]  [<ffffffff8148f078>] __wait_on_bit_lock+0x45/0x8c
[16087.359872]  [<ffffffff810b2a14>] ? sync_page+0x0/0x48
[16087.359872]  [<ffffffff810b29c4>] __lock_page+0x63/0x6a
[16087.359872]  [<ffffffff81057cec>] ? wake_bit_function+0x0/0x2a
[16087.359872]  [<ffffffff810bb7bc>] lock_page+0x37/0x3b
[16087.359872]  [<ffffffff810bb842>] invalidate_inode_pages2_range+0x82/0x2cb
[16087.359872]  [<ffffffff8123bbed>] ? xfs_vm_direct_IO+0x90/0xbb
[16087.359872]  [<ffffffff8123be2b>] ? xfs_get_blocks_direct+0x0/0x17
[16087.359872]  [<ffffffff8123ba4b>] ? xfs_end_io_direct+0x0/0x4e
[16087.359872]  [<ffffffff810b370a>] generic_file_direct_write+0x102/0x158
[16087.359872]  [<ffffffff81243606>] xfs_write+0x4a5/0x793
[16087.359872]  [<ffffffff810309a6>] ? cpuacct_charge+0x8e/0x97
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff8123ffa5>] xfs_file_aio_write+0x5a/0x5c
[16087.359872]  [<ffffffff810eabe6>] do_sync_write+0xc0/0x106
[16087.359872]  [<ffffffff810ebd8c>] ? fget_light+0x4f/0xfb
[16087.359872]  [<ffffffff81256e08>] ? security_file_permission+0x11/0x13
[16087.359872]  [<ffffffff810eb56f>] vfs_write+0xab/0x105
[16087.359872]  [<ffffffff810eb68d>] sys_write+0x47/0x70
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] test-mysqld   D ffff88001ec190b0  3152  2883   2877 0x00000000
[16087.359872]  ffff88001e4bde28 0000000000000046 ffff88001e4bdda8
ffffffff810676b1
[16087.359872]  ffff88001ec190b0 000000000000def8 ffff88001e4bdfd8
0000000000014180
[16087.359872]  ffff88001e4bdfd8 0000000000014180 ffff88001ec190b0
ffff88001ec3b210
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff81067901>] ? mark_held_locks+0x4d/0x6b
[16087.359872]  [<ffffffff8148f6b8>] ? __mutex_lock_common+0x216/0x330
[16087.359872]  [<ffffffff8148f6c0>] __mutex_lock_common+0x21e/0x330
[16087.359872]  [<ffffffff810eba12>] ? generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810eba12>] ? generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810ebd8c>] ? fget_light+0x4f/0xfb
[16087.359872]  [<ffffffff8148f881>] mutex_lock_nested+0x37/0x3c
[16087.359872]  [<ffffffff810eba12>] generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810ea6b4>] vfs_llseek+0x35/0x37
[16087.359872]  [<ffffffff810ea7cf>] sys_lseek+0x48/0x67
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] test-mysqld   D 0000000000000000  3360  2884   2877 0x00000000
[16087.359872]  ffff88001ece7e28 0000000000000046 ffff88001ece7df8
ffff88001ece7df4
[16087.359872]  ffff88001e9e0000 000000000000def8 ffff88001ece7fd8
0000000000014180
[16087.359872]  ffff88001ece7fd8 0000000000014180 ffff88001ec1a160
ffffffff81a0e020
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff8148f6b8>] ? __mutex_lock_common+0x216/0x330
[16087.359872]  [<ffffffff8148f6c0>] __mutex_lock_common+0x21e/0x330
[16087.359872]  [<ffffffff810eba12>] ? generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810eba12>] ? generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810ebd8c>] ? fget_light+0x4f/0xfb
[16087.359872]  [<ffffffff8148f881>] mutex_lock_nested+0x37/0x3c
[16087.359872]  [<ffffffff810eba12>] generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810ea6b4>] vfs_llseek+0x35/0x37
[16087.359872]  [<ffffffff810ea7cf>] sys_lseek+0x48/0x67
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] test-mysqld   D 0000000000000000  2664  2885   2877 0x00000000
[16087.359872]  ffff88001ecc9bb8 0000000000000046 ffff88001ecc9b38
ffffffff810676b1
[16087.359872]  ffff880002614180 000000000000def8 ffff88001ecc9fd8
0000000000014180
[16087.359872]  ffff88001ecc9fd8 0000000000014180 ffff88001ec1c2c0
ffff88001ec3b210
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff8149022b>] rwsem_down_failed_common+0x9a/0xcc
[16087.359872]  [<ffffffff8149027d>] rwsem_down_write_failed+0x20/0x2a
[16087.359872]  [<ffffffff812873b3>] call_rwsem_down_write_failed+0x13/0x20
[16087.359872]  [<ffffffff8105b78b>] ? down_write_nested+0x66/0x6d
[16087.359872]  [<ffffffff81221001>] ? xfs_ilock+0x2c/0xb2
[16087.359872]  [<ffffffff81221001>] xfs_ilock+0x2c/0xb2
[16087.359872]  [<ffffffff8124342d>] xfs_write+0x2cc/0x793
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff8123ffa5>] xfs_file_aio_write+0x5a/0x5c
[16087.359872]  [<ffffffff810eabe6>] do_sync_write+0xc0/0x106
[16087.359872]  [<ffffffff810ebd8c>] ? fget_light+0x4f/0xfb
[16087.359872]  [<ffffffff81256e08>] ? security_file_permission+0x11/0x13
[16087.359872]  [<ffffffff810eb56f>] vfs_write+0xab/0x105
[16087.359872]  [<ffffffff810eb68d>] sys_write+0x47/0x70
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] test-mysqld   D ffff88001ec1d370  2664  2886   2877 0x00000000
[16087.359872]  ffff88001eea1e28 0000000000000046 ffff88001eea1da8
ffffffff810676b1
[16087.359872]  000000000000000d 000000000000def8 ffff88001eea1fd8
0000000000014180
[16087.359872]  ffff88001eea1fd8 0000000000014180 ffff88001ec1d370
ffff88001fa53210
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff81067901>] ? mark_held_locks+0x4d/0x6b
[16087.359872]  [<ffffffff8148f6b8>] ? __mutex_lock_common+0x216/0x330
[16087.359872]  [<ffffffff8148f6c0>] __mutex_lock_common+0x21e/0x330
[16087.359872]  [<ffffffff810eba12>] ? generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810eba12>] ? generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810ebd8c>] ? fget_light+0x4f/0xfb
[16087.359872]  [<ffffffff8148f881>] mutex_lock_nested+0x37/0x3c
[16087.359872]  [<ffffffff810eba12>] generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810ea6b4>] vfs_llseek+0x35/0x37
[16087.359872]  [<ffffffff810ea7cf>] sys_lseek+0x48/0x67
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] test-mysqld   D 0000000000000002  3440  2887   2877 0x00000000
[16087.359872]  ffff88001ee87a88 0000000000000046 0000000000000000
ffff88000240a980
[16087.359872]  0000000046e48524 000000000000def8 ffff88001ee87fd8
0000000000014180
[16087.359872]  ffff88001ee87fd8 0000000000014180 ffff88001ec1b210
ffffffff81a0e020
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81060016>] ? ktime_get_ts+0xb1/0xbe
[16087.359872]  [<ffffffff8148ec26>] io_schedule+0x7b/0xc2
[16087.359872]  [<ffffffff810b2a58>] sync_page+0x44/0x48
[16087.359872]  [<ffffffff8148f078>] __wait_on_bit_lock+0x45/0x8c
[16087.359872]  [<ffffffff810b2a14>] ? sync_page+0x0/0x48
[16087.359872]  [<ffffffff810b29c4>] __lock_page+0x63/0x6a
[16087.359872]  [<ffffffff81057cec>] ? wake_bit_function+0x0/0x2a
[16087.359872]  [<ffffffff810bb7bc>] lock_page+0x37/0x3b
[16087.359872]  [<ffffffff810bb842>] invalidate_inode_pages2_range+0x82/0x2cb
[16087.359872]  [<ffffffff8123bbed>] ? xfs_vm_direct_IO+0x90/0xbb
[16087.359872]  [<ffffffff8123be2b>] ? xfs_get_blocks_direct+0x0/0x17
[16087.359872]  [<ffffffff8123ba4b>] ? xfs_end_io_direct+0x0/0x4e
[16087.359872]  [<ffffffff810b370a>] generic_file_direct_write+0x102/0x158
[16087.359872]  [<ffffffff81243606>] xfs_write+0x4a5/0x793
[16087.359872]  [<ffffffff810309a6>] ? cpuacct_charge+0x8e/0x97
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff8123ffa5>] xfs_file_aio_write+0x5a/0x5c
[16087.359872]  [<ffffffff810eabe6>] do_sync_write+0xc0/0x106
[16087.359872]  [<ffffffff810ebd8c>] ? fget_light+0x4f/0xfb
[16087.359872]  [<ffffffff81256e08>] ? security_file_permission+0x11/0x13
[16087.359872]  [<ffffffff810eb56f>] vfs_write+0xab/0x105
[16087.359872]  [<ffffffff810eb68d>] sys_write+0x47/0x70
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] test-mysqld   D ffff88001f1e42c0  2664  2888   2877 0x00000000
[16087.359872]  ffff88001dcc5e28 0000000000000046 0000000000000000
ffffffff810676b1
[16087.359872]  ffff88001e9e0000 000000000000def8 ffff88001dcc5fd8
0000000000014180
[16087.359872]  ffff88001dcc5fd8 0000000000014180 ffff88001f1e42c0
ffff88001f962160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff8148f6b8>] ? __mutex_lock_common+0x216/0x330
[16087.359872]  [<ffffffff8148f6c0>] __mutex_lock_common+0x21e/0x330
[16087.359872]  [<ffffffff810eba12>] ? generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810eba12>] ? generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810ebd8c>] ? fget_light+0x4f/0xfb
[16087.359872]  [<ffffffff8148f881>] mutex_lock_nested+0x37/0x3c
[16087.359872]  [<ffffffff810eba12>] generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810ea6b4>] vfs_llseek+0x35/0x37
[16087.359872]  [<ffffffff810ea7cf>] sys_lseek+0x48/0x67
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] test-mysqld   D ffff88001ec3d370  2664  2889   2877 0x00000000
[16087.359872]  ffff88001ede5e28 0000000000000046 ffff88001ede5da8
ffffffff810676b1
[16087.359872]  ffff88001ec3d370 000000000000def8 ffff88001ede5fd8
0000000000014180
[16087.359872]  ffff88001ede5fd8 0000000000014180 ffff88001ec3d370
ffff88001f0b5370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff81067901>] ? mark_held_locks+0x4d/0x6b
[16087.359872]  [<ffffffff8148f6b8>] ? __mutex_lock_common+0x216/0x330
[16087.359872]  [<ffffffff8148f6c0>] __mutex_lock_common+0x21e/0x330
[16087.359872]  [<ffffffff810eba12>] ? generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810eba12>] ? generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810ebd8c>] ? fget_light+0x4f/0xfb
[16087.359872]  [<ffffffff8148f881>] mutex_lock_nested+0x37/0x3c
[16087.359872]  [<ffffffff810eba12>] generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810ea6b4>] vfs_llseek+0x35/0x37
[16087.359872]  [<ffffffff810ea7cf>] sys_lseek+0x48/0x67
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] test-mysqld   D 0000000000000000  3152  2890   2877 0x00000000
[16087.359872]  ffff88001e6fbc08 0000000000000046 ffff88001e6fbbd8
ffff88001e6fbbd4
[16087.359872]  ffff88001ec3b210 000000000000def8 ffff88001e6fbfd8
0000000000014180
[16087.359872]  ffff88001e6fbfd8 0000000000014180 ffff88001ec3b210
ffff88001ec1a160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff8148f6b8>] ? __mutex_lock_common+0x216/0x330
[16087.359872]  [<ffffffff8148f6c0>] __mutex_lock_common+0x21e/0x330
[16087.359872]  [<ffffffff81243420>] ? xfs_write+0x2bf/0x793
[16087.359872]  [<ffffffff81243420>] ? xfs_write+0x2bf/0x793
[16087.359872]  [<ffffffff8148f881>] mutex_lock_nested+0x37/0x3c
[16087.359872]  [<ffffffff81243420>] xfs_write+0x2bf/0x793
[16087.359872]  [<ffffffff810309a6>] ? cpuacct_charge+0x8e/0x97
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff8123ffa5>] xfs_file_aio_write+0x5a/0x5c
[16087.359872]  [<ffffffff810eabe6>] do_sync_write+0xc0/0x106
[16087.359872]  [<ffffffff810ebd8c>] ? fget_light+0x4f/0xfb
[16087.359872]  [<ffffffff81256e08>] ? security_file_permission+0x11/0x13
[16087.359872]  [<ffffffff810eb56f>] vfs_write+0xab/0x105
[16087.359872]  [<ffffffff810eb68d>] sys_write+0x47/0x70
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] test-mysqld   D ffff88001ec3e420  3232  2891   2877 0x00000000
[16087.359872]  ffff88001ee91e28 0000000000000046 ffff88001ee91da8
ffffffff810676b1
[16087.359872]  ffff88001ec3e420 000000000000def8 ffff88001ee91fd8
0000000000014180
[16087.359872]  ffff88001ee91fd8 0000000000014180 ffff88001ec3e420
ffff88001f152160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff81067901>] ? mark_held_locks+0x4d/0x6b
[16087.359872]  [<ffffffff8148f6b8>] ? __mutex_lock_common+0x216/0x330
[16087.359872]  [<ffffffff8148f6c0>] __mutex_lock_common+0x21e/0x330
[16087.359872]  [<ffffffff810eba12>] ? generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810eba12>] ? generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810ebd8c>] ? fget_light+0x4f/0xfb
[16087.359872]  [<ffffffff8148f881>] mutex_lock_nested+0x37/0x3c
[16087.359872]  [<ffffffff810eba12>] generic_file_llseek+0x2c/0x5e
[16087.359872]  [<ffffffff810ea6b4>] vfs_llseek+0x35/0x37
[16087.359872]  [<ffffffff810ea7cf>] sys_lseek+0x48/0x67
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] test-mysqld   D ffff88001ec3a160  4136  2892   2877 0x00000000
[16087.359872]  ffff88001e6d7e18 0000000000000046 ffffffff810b242d
ffffffff8105b193
[16087.359872]  000000010000000e 000000000000def8 ffff88001e6d7fd8
0000000000014180
[16087.359872]  ffff88001e6d7fd8 0000000000014180 ffff88001ec3a160
ffff88001e5e42c0
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff810b242d>] ? find_get_pages_tag+0x0/0x12a
[16087.359872]  [<ffffffff8105b193>] ? lock_hrtimer_base+0x25/0x4a
[16087.359872]  [<ffffffff81067901>] ? mark_held_locks+0x4d/0x6b
[16087.359872]  [<ffffffff8148f6b8>] ? __mutex_lock_common+0x216/0x330
[16087.359872]  [<ffffffff8148f6c0>] __mutex_lock_common+0x21e/0x330
[16087.359872]  [<ffffffff8110a3af>] ? vfs_fsync_range+0x6e/0xa6
[16087.359872]  [<ffffffff8110a3af>] ? vfs_fsync_range+0x6e/0xa6
[16087.359872]  [<ffffffff8148f881>] mutex_lock_nested+0x37/0x3c
[16087.359872]  [<ffffffff8110a3af>] vfs_fsync_range+0x6e/0xa6
[16087.359872]  [<ffffffff8110a454>] vfs_fsync+0x18/0x1a
[16087.359872]  [<ffffffff8110a488>] do_fsync+0x32/0x48
[16087.359872]  [<ffffffff8110a4bb>] sys_fsync+0xb/0xf
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] bash          S ffff88001e5e0000  5776  2893   2876 0x00000000
[16087.359872]  ffff88001ee57e58 0000000000000046 0000000000000000
0000000000000000
[16087.359872]  ffff88001ee57e08 000000000000def8 ffff88001ee57fd8
0000000000014180
[16087.359872]  ffff88001ee57fd8 0000000000014180 ffff88001e5e0000
ffffffff81a0e020
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81040671>] do_wait+0x1bd/0x22b
[16087.359872]  [<ffffffff810c885b>] ? might_fault+0x57/0xa4
[16087.359872]  [<ffffffff81040785>] sys_wait4+0xa6/0xc3
[16087.359872]  [<ffffffff8103ee89>] ? child_wait_callback+0x0/0x53
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] test-tar      D 0000000000000000  3320  2897   2893 0x00000000
[16087.359872]  ffff88001e6eb700 0000000000000046 ffff88001e6eb680
ffffffff810676b1
[16087.359872]  0000000000000000 000000000000def8 ffff88001e6ebfd8
0000000000014180
[16087.359872]  ffff88001e6ebfd8 0000000000014180 ffff88001e58a160
ffff88001ec1c2c0
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff8149022b>] rwsem_down_failed_common+0x9a/0xcc
[16087.359872]  [<ffffffff8127b18e>] ? cfq_close_cooperator+0xd9/0x184
[16087.359872]  [<ffffffff814902ad>] rwsem_down_read_failed+0x26/0x30
[16087.359872]  [<ffffffff81287384>] call_rwsem_down_read_failed+0x14/0x30
[16087.359872]  [<ffffffff8105b7ed>] ? down_read_nested+0x5b/0x62
[16087.359872]  [<ffffffff81221050>] ? xfs_ilock+0x7b/0xb2
[16087.359872]  [<ffffffff81221050>] xfs_ilock+0x7b/0xb2
[16087.359872]  [<ffffffff812210b0>] xfs_ilock_map_shared+0x29/0x2f
[16087.359872]  [<ffffffff8122762b>] xfs_iomap+0x9f/0x39c
[16087.359872]  [<ffffffff81066948>] ? register_lock_class+0x20/0x327
[16087.359872]  [<ffffffff8123bd40>] __xfs_get_blocks+0x71/0x15c
[16087.359872]  [<ffffffff8123be54>] xfs_get_blocks+0x12/0x14
[16087.359872]  [<ffffffff811137a1>] do_mpage_readpage+0x173/0x467
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff810b2d46>] ? add_to_page_cache_locked+0xa4/0xcd
[16087.359872]  [<ffffffff81113bc8>] mpage_readpages+0xd6/0x11a
[16087.359872]  [<ffffffff8123be42>] ? xfs_get_blocks+0x0/0x14
[16087.359872]  [<ffffffff8123be42>] ? xfs_get_blocks+0x0/0x14
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff8123cee4>] xfs_vm_readpages+0x1a/0x1c
[16087.359872]  [<ffffffff810ba6a2>] __do_page_cache_readahead+0x163/0x1fc
[16087.359872]  [<ffffffff810ba5ba>] ? __do_page_cache_readahead+0x7b/0x1fc
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff810ba757>] ra_submit+0x1c/0x20
[16087.359872]  [<ffffffff810baa7b>] ondemand_readahead+0x243/0x256
[16087.359872]  [<ffffffff810bab6e>] page_cache_sync_readahead+0x38/0x3a
[16087.359872]  [<ffffffff810b3cd7>] generic_file_aio_read+0x242/0x5a7
[16087.359872]  [<ffffffff81243d6d>] xfs_read+0x1a8/0x1ff
[16087.359872]  [<ffffffff8123ffff>] xfs_file_aio_read+0x58/0x5d
[16087.359872]  [<ffffffff810eacec>] do_sync_read+0xc0/0x106
[16087.359872]  [<ffffffff810372ec>] ? finish_task_switch+0x6f/0xad
[16087.359872]  [<ffffffff8103727d>] ? finish_task_switch+0x0/0xad
[16087.359872]  [<ffffffff8148eb33>] ? schedule+0x845/0x8bd
[16087.359872]  [<ffffffff81256e08>] ? security_file_permission+0x11/0x13
[16087.359872]  [<ffffffff810eb75e>] vfs_read+0xa8/0x102
[16087.359872]  [<ffffffff810eb87c>] sys_read+0x47/0x70
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] bash          S ffff88001e6e0000  4880  2898   2876 0x00000000
[16087.359872]  ffff88001f3dfe58 0000000000000046 0000000000000000
0000000000000000
[16087.359872]  ffff88001f3dfe08 000000000000def8 ffff88001f3dffd8
0000000000014180
[16087.359872]  ffff88001f3dffd8 0000000000014180 ffff88001e6e0000
ffff88001f965370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81040671>] do_wait+0x1bd/0x22b
[16087.359872]  [<ffffffff810c885b>] ? might_fault+0x57/0xa4
[16087.359872]  [<ffffffff81040785>] sys_wait4+0xa6/0xc3
[16087.359872]  [<ffffffff8103ee89>] ? child_wait_callback+0x0/0x53
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] syslog-ng     S ffff88001e6e10b0  6248  3348      1 0x00000000
[16087.359872]  ffff88001f373e58 0000000000000046 0000000000000002
0000000000000000
[16087.359872]  ffff88001f373e08 000000000000def8 ffff88001f373fd8
0000000000014180
[16087.359872]  ffff88001f373fd8 0000000000014180 ffff88001e6e10b0
ffff88001e5e6420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81040671>] do_wait+0x1bd/0x22b
[16087.359872]  [<ffffffff81040785>] sys_wait4+0xa6/0xc3
[16087.359872]  [<ffffffff8103ee89>] ? child_wait_callback+0x0/0x53
[16087.359872]  [<ffffffff810e9148>] ? sys_close+0x2d/0xe4
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] syslog-ng     S 0000000000000000  3832  3349   3348 0x00000000
[16087.359872]  ffff88001f3e9a58 0000000000000046 0000000000000000
0000000000000282
[16087.359872]  0000000000000040 000000000000def8 ffff88001f3e9fd8
0000000000014180
[16087.359872]  ffff88001f3e9fd8 0000000000014180 ffff88001e5e6420
ffff88001f965370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff8148f948>] schedule_hrtimeout_range+0xc2/0xf7
[16087.359872]  [<ffffffff8105a9e0>] ? hrtimer_wakeup+0x0/0x21
[16087.359872]  [<ffffffff8105b4eb>] ? hrtimer_start_range_ns+0xf/0x11
[16087.359872]  [<ffffffff810f8b92>] poll_schedule_timeout+0x46/0x69
[16087.359872]  [<ffffffff810f8fac>] do_sys_poll+0x36a/0x40b
[16087.359872]  [<ffffffff810f9d51>] ? __pollwait+0x0/0xcf
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810f9e20>] ? pollwake+0x0/0x5c
[16087.359872]  [<ffffffff810676b1>] ? mark_lock+0x22/0x225
[16087.359872]  [<ffffffff81067901>] ? mark_held_locks+0x4d/0x6b
[16087.359872]  [<ffffffff8103077e>] ? __wake_up+0x22/0x50
[16087.359872]  [<ffffffff81490baf>] ? _raw_spin_unlock_irqrestore+0x3f/0x47
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff81067b62>] ? trace_hardirqs_on_caller+0x10b/0x12f
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff81060016>] ? ktime_get_ts+0xb1/0xbe
[16087.359872]  [<ffffffff810f8b31>] ? poll_select_set_timeout+0x5c/0x77
[16087.359872]  [<ffffffff810f91e2>] sys_poll+0x50/0xba
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] flush-254:0   S 00000000000493e0  5216  4467      2 0x00000000
[16087.359872]  ffff88001e783db0 0000000000000046 ffff88001e623210
ffffffff81490baf
[16087.359872]  ffff88001f95c000 000000000000def8 ffff88001e783fd8
0000000000014180
[16087.359872]  ffff88001e783fd8 0000000000014180 ffff88001e623210
ffff88001f152160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81490baf>] ? _raw_spin_unlock_irqrestore+0x3f/0x47
[16087.359872]  [<ffffffff8104b617>] ? __mod_timer+0x141/0x153
[16087.359872]  [<ffffffff8104b45f>] ? init_timer_key+0x8d/0x99
[16087.359872]  [<ffffffff8148efbb>] schedule_timeout+0xa2/0xc9
[16087.359872]  [<ffffffff8104b095>] ? process_timeout+0x0/0xb
[16087.359872]  [<ffffffff8148f031>] schedule_timeout_interruptible+0x19/0x1b
[16087.359872]  [<ffffffff811075a4>] bdi_writeback_task+0x82/0xa9
[16087.359872]  [<ffffffff810c789c>] ? bdi_start_fn+0x0/0xd4
[16087.359872]  [<ffffffff810c790d>] bdi_start_fn+0x71/0xd4
[16087.359872]  [<ffffffff810c789c>] ? bdi_start_fn+0x0/0xd4
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] flush-254:16  S 00000000000493e0  6208  4702      2 0x00000000
[16087.359872]  ffff88001e6cddb0 0000000000000046 0000000000000000
ffffffff81490baf
[16087.359872]  ffff88001f970000 000000000000def8 ffff88001e6cdfd8
0000000000014180
[16087.359872]  ffff88001e6cdfd8 0000000000014180 ffff88001e5e5370
ffff88001f965370
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81490baf>] ? _raw_spin_unlock_irqrestore+0x3f/0x47
[16087.359872]  [<ffffffff8104b617>] ? __mod_timer+0x141/0x153
[16087.359872]  [<ffffffff8104b45f>] ? init_timer_key+0x8d/0x99
[16087.359872]  [<ffffffff8148efbb>] schedule_timeout+0xa2/0xc9
[16087.359872]  [<ffffffff8104b095>] ? process_timeout+0x0/0xb
[16087.359872]  [<ffffffff8148f031>] schedule_timeout_interruptible+0x19/0x1b
[16087.359872]  [<ffffffff811075a4>] bdi_writeback_task+0x82/0xa9
[16087.359872]  [<ffffffff810c789c>] ? bdi_start_fn+0x0/0xd4
[16087.359872]  [<ffffffff810c790d>] bdi_start_fn+0x71/0xd4
[16087.359872]  [<ffffffff810c789c>] ? bdi_start_fn+0x0/0xd4
[16087.359872]  [<ffffffff81057842>] kthread+0x7d/0x85
[16087.359872]  [<ffffffff81003894>] kernel_thread_helper+0x4/0x10
[16087.359872]  [<ffffffff81490f3c>] ? restore_args+0x0/0x30
[16087.359872]  [<ffffffff810577c5>] ? kthread+0x0/0x85
[16087.359872]  [<ffffffff81003890>] ? kernel_thread_helper+0x0/0x10
[16087.359872] bash          S ffff88001dcb10b0  4616  4754   2876 0x00000000
[16087.359872]  ffff88001f371e58 0000000000000046 0000000000000002
0000000000000000
[16087.359872]  ffff88001f371e08 000000000000def8 ffff88001f371fd8
0000000000014180
[16087.359872]  ffff88001f371fd8 0000000000014180 ffff88001dcb10b0
ffff88001f152160
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81040671>] do_wait+0x1bd/0x22b
[16087.359872]  [<ffffffff810c885b>] ? might_fault+0x57/0xa4
[16087.359872]  [<ffffffff81040785>] sys_wait4+0xa6/0xc3
[16087.359872]  [<ffffffff8103ee89>] ? child_wait_callback+0x0/0x53
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] sync          D 0000000000000000  4896  4761   4754 0x00000000
[16087.359872]  ffff88001edafcd8 0000000000000046 0000000000000000
ffffffff81067901
[16087.359872]  ffff88001e5e42c0 000000000000def8 ffff88001edaffd8
0000000000014180
[16087.359872]  ffff88001edaffd8 0000000000014180 ffff88001e5e42c0
ffff88001f916420
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81067901>] ? mark_held_locks+0x4d/0x6b
[16087.359872]  [<ffffffff81057eed>] ? prepare_to_wait+0x70/0x7b
[16087.359872]  [<ffffffff8123d0e0>] xfs_ioend_wait+0x83/0x9b
[16087.359872]  [<ffffffff81057cb4>] ? autoremove_wake_function+0x0/0x38
[16087.359872]  [<ffffffff810b90a6>] ? mapping_tagged+0x60/0x69
[16087.359872]  [<ffffffff810b9046>] ? mapping_tagged+0x0/0x69
[16087.359872]  [<ffffffff81246621>] xfs_sync_inode_data+0x98/0xbf
[16087.359872]  [<ffffffff81246773>] xfs_inode_ag_walk+0x9d/0xdc
[16087.359872]  [<ffffffff81246589>] ? xfs_sync_inode_data+0x0/0xbf
[16087.359872]  [<ffffffff81246808>] xfs_inode_ag_iterator+0x56/0x84
[16087.359872]  [<ffffffff81246589>] ? xfs_sync_inode_data+0x0/0xbf
[16087.359872]  [<ffffffff812469d0>] xfs_sync_data+0x26/0x4e
[16087.359872]  [<ffffffff81246a47>] xfs_quiesce_data+0x1e/0x57
[16087.359872]  [<ffffffff81244359>] xfs_fs_sync_fs+0x26/0xd7
[16087.359872]  [<ffffffff8112fa4e>] ? vfs_quota_sync+0x1b7/0x1c8
[16087.359872]  [<ffffffff81131394>] sync_quota_sb+0x4e/0x103
[16087.359872]  [<ffffffff8110a5ff>] __sync_filesystem+0x41/0x72
[16087.359872]  [<ffffffff8110a6da>] sync_filesystems+0xaa/0x100
[16087.359872]  [<ffffffff8110a781>] sys_sync+0x1c/0x2e
[16087.359872]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[16087.359872] test-malloc   R  running task     4616  4765   2898 0x00000000
[16087.359872]  ffff88001f3abdb8 0000000000000040 0000000000000001
ffff88001f3abea8
[16087.359872]  ffff88000268e720 ffffffff81063fa4 ffff88001f3abe48
ffffffff8105b465
[16087.359872]  ffff88001dcb5370 000000000000e6b0 0000000100000001
000000000000c350
[16087.359872] Call Trace:
[16087.359872]  [<ffffffff81067b93>] ? trace_hardirqs_on+0xd/0xf
[16087.359872]  [<ffffffff8148f9ff>] ? do_nanosleep+0x73/0xb3
[16087.359872]  [<ffffffff8105b578>] ? hrtimer_nanosleep+0x8b/0xf9
[16087.359872]  [<ffffffff8105a9e0>] ? hrtimer_wakeup+0x0/0x21
[16087.359872]  [<ffffffff8105b4eb>] ? hrtimer_start_range_ns+0xf/0x11
[16087.359872]  [<ffffffff8105b644>] ? sys_nanosleep+0x5e/0x78
[16087.359872]  [<ffffffff81002a6b>] ? system_call_fastpath+0x16/0x1b
[16087.359872]
[16087.359872] Showing all locks held in the system:
[16087.359872] 2 locks held by xfsdatad/0/414:
[16087.359872]  #0:  (xfsdatad){+.+...}, at: [<ffffffff81053da4>]
worker_thread+0x1d5/0x332
[16087.359872]  #1:  ((&ioend->io_work)){+.+...}, at:
[<ffffffff81053da4>] worker_thread+0x1d5/0x332
[16087.359872] 2 locks held by xfsdatad/2/416:
[16087.359872]  #0:  (xfsdatad){+.+...}, at: [<ffffffff81053da4>]
worker_thread+0x1d5/0x332
[16087.359872]  #1:  ((&ioend->io_work)){+.+...}, at:
[<ffffffff81053da4>] worker_thread+0x1d5/0x332
[16087.359872] 1 lock held by agetty/2848:
[16087.359872]  #0:  (&tty->atomic_read_lock){+.+.+.}, at:
[<ffffffff812a9599>] n_tty_read+0x2d8/0x7f3
[16087.359872] 1 lock held by agetty/2849:
[16087.359872]  #0:  (&tty->atomic_read_lock){+.+.+.}, at:
[<ffffffff812a9599>] n_tty_read+0x2d8/0x7f3
[16087.359872] 1 lock held by agetty/2850:
[16087.359872]  #0:  (&tty->atomic_read_lock){+.+.+.}, at:
[<ffffffff812a9599>] n_tty_read+0x2d8/0x7f3
[16087.359872] 1 lock held by agetty/2851:
[16087.359872]  #0:  (&tty->atomic_read_lock){+.+.+.}, at:
[<ffffffff812a9599>] n_tty_read+0x2d8/0x7f3
[16087.359872] 1 lock held by agetty/2852:
[16087.359872]  #0:  (&tty->atomic_read_lock){+.+.+.}, at:
[<ffffffff812a9599>] n_tty_read+0x2d8/0x7f3
[16087.359872] 2 locks held by bash/2853:
[16087.359872]  #0:  (sysrq_key_table_lock){......}, at:
[<ffffffff812babae>] __handle_sysrq+0x26/0x158
[16087.359872]  #1:  (tasklist_lock){.+.+..}, at: [<ffffffff8106733d>]
debug_show_all_locks+0x53/0x188
[16087.359872] 1 lock held by test-mysqld/2882:
[16087.359872]  #0:  (&(&ip->i_iolock)->mr_lock){++++++}, at:
[<ffffffff8122101e>] xfs_ilock+0x49/0xb2
[16087.359872] 1 lock held by test-mysqld/2883:
[16087.359872]  #0:  (&sb->s_type->i_mutex_key#12){+.+.+.}, at:
[<ffffffff810eba12>] generic_file_llseek+0x2c/0x5e
[16087.359872] 1 lock held by test-mysqld/2884:
[16087.359872]  #0:  (&sb->s_type->i_mutex_key#12){+.+.+.}, at:
[<ffffffff810eba12>] generic_file_llseek+0x2c/0x5e
[16087.359872] 2 locks held by test-mysqld/2885:
[16087.359872]  #0:  (&sb->s_type->i_mutex_key#12){+.+.+.}, at:
[<ffffffff81243420>] xfs_write+0x2bf/0x793
[16087.359872]  #1:  (&(&ip->i_iolock)->mr_lock){++++++}, at:
[<ffffffff81221001>] xfs_ilock+0x2c/0xb2
[16087.359872] 1 lock held by test-mysqld/2886:
[16087.359872]  #0:  (&sb->s_type->i_mutex_key#12){+.+.+.}, at:
[<ffffffff810eba12>] generic_file_llseek+0x2c/0x5e
[16087.359872] 1 lock held by test-mysqld/2887:
[16087.359872]  #0:  (&(&ip->i_iolock)->mr_lock){++++++}, at:
[<ffffffff8122101e>] xfs_ilock+0x49/0xb2
[16087.359872] 1 lock held by test-mysqld/2888:
[16087.359872]  #0:  (&sb->s_type->i_mutex_key#12){+.+.+.}, at:
[<ffffffff810eba12>] generic_file_llseek+0x2c/0x5e
[16087.359872] 1 lock held by test-mysqld/2889:
[16087.359872]  #0:  (&sb->s_type->i_mutex_key#12){+.+.+.}, at:
[<ffffffff810eba12>] generic_file_llseek+0x2c/0x5e
[16087.359872] 1 lock held by test-mysqld/2890:
[16087.359872]  #0:  (&sb->s_type->i_mutex_key#12){+.+.+.}, at:
[<ffffffff81243420>] xfs_write+0x2bf/0x793
[16087.359872] 1 lock held by test-mysqld/2891:
[16087.359872]  #0:  (&sb->s_type->i_mutex_key#12){+.+.+.}, at:
[<ffffffff810eba12>] generic_file_llseek+0x2c/0x5e
[16087.359872] 1 lock held by test-mysqld/2892:
[16087.359872]  #0:  (&sb->s_type->i_mutex_key#12){+.+.+.}, at:
[<ffffffff8110a3af>] vfs_fsync_range+0x6e/0xa6
[16087.359872] 2 locks held by test-tar/2897:
[16087.359872]  #0:  (&(&ip->i_iolock)->mr_lock){++++++}, at:
[<ffffffff8122101e>] xfs_ilock+0x49/0xb2
[16087.359872]  #1:  (&(&ip->i_lock)->mr_lock){++++..}, at:
[<ffffffff81221050>] xfs_ilock+0x7b/0xb2
[16087.359872] 2 locks held by sync/4761:
[16087.359872]  #0:  (mutex){+.+...}, at: [<ffffffff8110a64e>]
sync_filesystems+0x1e/0x100
[16087.359872]  #1:  (&type->s_umount_key#24){.+.+..}, at:
[<ffffffff8110a6b8>] sync_filesystems+0x88/0x100
[16087.359872]

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Dave Chinner
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Dave Chinner
Ilia,

Can you send me the output of this for your kernel that the
traces came from:

$ gdb <path/to/vmlinux>
(gdb) l *( xfs_write+0x2cc)

You can run it against the vmlinux file in the kernel build
directory.  Basically I need to know which xfs_ilock() call in
xfs_write() one of the mysqld-test processes is stuck on.

Cheers,

Dave.
--
Dave Chinner
[hidden email]

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Ilia Mirkin-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Ilia Mirkin-2
On Tue, Aug 3, 2010 at 8:47 PM, Dave Chinner <[hidden email]> wrote:

> Ilia,
>
> Can you send me the output of this for your kernel that the
> traces came from:
>
> $ gdb <path/to/vmlinux>
> (gdb) l *( xfs_write+0x2cc)
>
> You can run it against the vmlinux file in the kernel build
> directory.  Basically I need to know which xfs_ilock() call in
> xfs_write() one of the mysqld-test processes is stuck on.

No problem - BTW, I'm running this on a 2.6.33.3 kernel (same as the
one before, although diff hardware). If you want (and are fine with me
"destroying" the current state), I can upgrade it to a kernel of your
choice and repeat the test overnight.

Naturally I didn't have CONFIG_DEBUG_INFO in there... just changed
that to Y and recompiled. I'm not entirely sure that this preserves
all the offsets, but at least the BUG-HUNTING doc makes allusions that
it would.

(gdb) l *( xfs_write+0x2cc)
0xffffffff8124342d is in xfs_write (fs/xfs/linux-2.6/xfs_lrw.c:597).
592                     if (!need_i_mutex && (mapping->nrpages || pos
> xip->i_size)) {
593                             xfs_iunlock(xip, XFS_ILOCK_EXCL|iolock);
594                             iolock = XFS_IOLOCK_EXCL;
595                             need_i_mutex = 1;
596                             mutex_lock(&inode->i_mutex);
597                             xfs_ilock(xip, XFS_ILOCK_EXCL|iolock);
598                             goto start;
599                     }
600             }
601

Well, at least it points at an xfs_ilock, so pretty good chance that
adding debug info didn't change anything else.

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Dave Chinner
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Dave Chinner
On Tue, Aug 03, 2010 at 09:15:53PM -0400, Ilia Mirkin wrote:

> On Tue, Aug 3, 2010 at 8:47 PM, Dave Chinner <[hidden email]> wrote:
> > Ilia,
> >
> > Can you send me the output of this for your kernel that the
> > traces came from:
> >
> > $ gdb <path/to/vmlinux>
> > (gdb) l *( xfs_write+0x2cc)
> >
> > You can run it against the vmlinux file in the kernel build
> > directory.  Basically I need to know which xfs_ilock() call in
> > xfs_write() one of the mysqld-test processes is stuck on.
>
> No problem - BTW, I'm running this on a 2.6.33.3 kernel (same as the
> one before, although diff hardware). If you want (and are fine with me
> "destroying" the current state), I can upgrade it to a kernel of your
> choice and repeat the test overnight.
>
> Naturally I didn't have CONFIG_DEBUG_INFO in there... just changed
> that to Y and recompiled. I'm not entirely sure that this preserves
> all the offsets, but at least the BUG-HUNTING doc makes allusions that
> it would.
>
> (gdb) l *( xfs_write+0x2cc)
> 0xffffffff8124342d is in xfs_write (fs/xfs/linux-2.6/xfs_lrw.c:597).
> 592                     if (!need_i_mutex && (mapping->nrpages || pos
> > xip->i_size)) {
> 593                             xfs_iunlock(xip, XFS_ILOCK_EXCL|iolock);
> 594                             iolock = XFS_IOLOCK_EXCL;
> 595                             need_i_mutex = 1;
> 596                             mutex_lock(&inode->i_mutex);
> 597                             xfs_ilock(xip, XFS_ILOCK_EXCL|iolock);

Make sense. Can you run 'l *(xfs_ilock+0x2c)' as well? I just need to
confirm which lock it has blocked on.

Cheers,

Dave.
--
Dave Chinner
[hidden email]

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Ilia Mirkin-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Ilia Mirkin-2
On Wed, Aug 4, 2010 at 12:27 AM, Dave Chinner <[hidden email]> wrote:

> On Tue, Aug 03, 2010 at 09:15:53PM -0400, Ilia Mirkin wrote:
>> On Tue, Aug 3, 2010 at 8:47 PM, Dave Chinner <[hidden email]> wrote:
>> > Ilia,
>> >
>> > Can you send me the output of this for your kernel that the
>> > traces came from:
>> >
>> > $ gdb <path/to/vmlinux>
>> > (gdb) l *( xfs_write+0x2cc)
>> >
>> > You can run it against the vmlinux file in the kernel build
>> > directory.  Basically I need to know which xfs_ilock() call in
>> > xfs_write() one of the mysqld-test processes is stuck on.
>>
>> No problem - BTW, I'm running this on a 2.6.33.3 kernel (same as the
>> one before, although diff hardware). If you want (and are fine with me
>> "destroying" the current state), I can upgrade it to a kernel of your
>> choice and repeat the test overnight.
>>
>> Naturally I didn't have CONFIG_DEBUG_INFO in there... just changed
>> that to Y and recompiled. I'm not entirely sure that this preserves
>> all the offsets, but at least the BUG-HUNTING doc makes allusions that
>> it would.
>>
>> (gdb) l *( xfs_write+0x2cc)
>> 0xffffffff8124342d is in xfs_write (fs/xfs/linux-2.6/xfs_lrw.c:597).
>> 592                     if (!need_i_mutex && (mapping->nrpages || pos
>> > xip->i_size)) {
>> 593                             xfs_iunlock(xip, XFS_ILOCK_EXCL|iolock);
>> 594                             iolock = XFS_IOLOCK_EXCL;
>> 595                             need_i_mutex = 1;
>> 596                             mutex_lock(&inode->i_mutex);
>> 597                             xfs_ilock(xip, XFS_ILOCK_EXCL|iolock);
>
> Make sense. Can you run 'l *(xfs_ilock+0x2c)' as well? I just need to
> confirm which lock it has blocked on.

(gdb) l *(xfs_ilock+0x2c)
0xffffffff81221001 is in xfs_ilock (fs/xfs/linux-2.6/mrlock.h:48).
43              down_read_nested(&mrp->mr_lock, subclass);
44      }
45
46      static inline void mrupdate_nested(mrlock_t *mrp, int subclass)
47      {
48              down_write_nested(&mrp->mr_lock, subclass);
49      #ifdef DEBUG
50              mrp->mr_writer = 1;
51      #endif
52      }

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Dave Chinner
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Dave Chinner
On Wed, Aug 04, 2010 at 12:39:08AM -0400, Ilia Mirkin wrote:

> On Wed, Aug 4, 2010 at 12:27 AM, Dave Chinner <[hidden email]> wrote:
> > On Tue, Aug 03, 2010 at 09:15:53PM -0400, Ilia Mirkin wrote:
> >> On Tue, Aug 3, 2010 at 8:47 PM, Dave Chinner <[hidden email]> wrote:
> >> > Ilia,
> >> >
> >> > Can you send me the output of this for your kernel that the
> >> > traces came from:
> >> >
> >> > $ gdb <path/to/vmlinux>
> >> > (gdb) l *( xfs_write+0x2cc)
> >> >
> >> > You can run it against the vmlinux file in the kernel build
> >> > directory.  Basically I need to know which xfs_ilock() call in
> >> > xfs_write() one of the mysqld-test processes is stuck on.
> >>
> >> No problem - BTW, I'm running this on a 2.6.33.3 kernel (same as the
> >> one before, although diff hardware). If you want (and are fine with me
> >> "destroying" the current state), I can upgrade it to a kernel of your
> >> choice and repeat the test overnight.
> >>
> >> Naturally I didn't have CONFIG_DEBUG_INFO in there... just changed
> >> that to Y and recompiled. I'm not entirely sure that this preserves
> >> all the offsets, but at least the BUG-HUNTING doc makes allusions that
> >> it would.
> >>
> >> (gdb) l *( xfs_write+0x2cc)
> >> 0xffffffff8124342d is in xfs_write (fs/xfs/linux-2.6/xfs_lrw.c:597).
> >> 592                     if (!need_i_mutex && (mapping->nrpages || pos
> >> > xip->i_size)) {
> >> 593                             xfs_iunlock(xip, XFS_ILOCK_EXCL|iolock);
> >> 594                             iolock = XFS_IOLOCK_EXCL;
> >> 595                             need_i_mutex = 1;
> >> 596                             mutex_lock(&inode->i_mutex);
> >> 597                             xfs_ilock(xip, XFS_ILOCK_EXCL|iolock);
> >
> > Make sense. Can you run 'l *(xfs_ilock+0x2c)' as well? I just need to
> > confirm which lock it has blocked on.
>
> (gdb) l *(xfs_ilock+0x2c)
> 0xffffffff81221001 is in xfs_ilock (fs/xfs/linux-2.6/mrlock.h:48).
> 43              down_read_nested(&mrp->mr_lock, subclass);
> 44      }
> 45
> 46      static inline void mrupdate_nested(mrlock_t *mrp, int subclass)
> 47      {
> 48              down_write_nested(&mrp->mr_lock, subclass);
> 49      #ifdef DEBUG
> 50              mrp->mr_writer = 1;
> 51      #endif
> 52      }

OK, that doesn't help - it followed into the inline function
rather than telling me which of the two calls in the function it
was. I guess I'll need the disassembly output to work it out.
Can you send the output of "disass xfs_ilock" instead? Thanks.

Cheers,

Dave.
--
Dave Chinner
[hidden email]

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Ilia Mirkin-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Ilia Mirkin-2
On Wed, Aug 4, 2010 at 2:53 AM, Dave Chinner <[hidden email]> wrote:

> On Wed, Aug 04, 2010 at 12:39:08AM -0400, Ilia Mirkin wrote:
>> On Wed, Aug 4, 2010 at 12:27 AM, Dave Chinner <[hidden email]> wrote:
>> > On Tue, Aug 03, 2010 at 09:15:53PM -0400, Ilia Mirkin wrote:
>> >> On Tue, Aug 3, 2010 at 8:47 PM, Dave Chinner <[hidden email]> wrote:
>> >> > Ilia,
>> >> >
>> >> > Can you send me the output of this for your kernel that the
>> >> > traces came from:
>> >> >
>> >> > $ gdb <path/to/vmlinux>
>> >> > (gdb) l *( xfs_write+0x2cc)
>> >> >
>> >> > You can run it against the vmlinux file in the kernel build
>> >> > directory.  Basically I need to know which xfs_ilock() call in
>> >> > xfs_write() one of the mysqld-test processes is stuck on.
>> >>
>> >> No problem - BTW, I'm running this on a 2.6.33.3 kernel (same as the
>> >> one before, although diff hardware). If you want (and are fine with me
>> >> "destroying" the current state), I can upgrade it to a kernel of your
>> >> choice and repeat the test overnight.
>> >>
>> >> Naturally I didn't have CONFIG_DEBUG_INFO in there... just changed
>> >> that to Y and recompiled. I'm not entirely sure that this preserves
>> >> all the offsets, but at least the BUG-HUNTING doc makes allusions that
>> >> it would.
>> >>
>> >> (gdb) l *( xfs_write+0x2cc)
>> >> 0xffffffff8124342d is in xfs_write (fs/xfs/linux-2.6/xfs_lrw.c:597).
>> >> 592                     if (!need_i_mutex && (mapping->nrpages || pos
>> >> > xip->i_size)) {
>> >> 593                             xfs_iunlock(xip, XFS_ILOCK_EXCL|iolock);
>> >> 594                             iolock = XFS_IOLOCK_EXCL;
>> >> 595                             need_i_mutex = 1;
>> >> 596                             mutex_lock(&inode->i_mutex);
>> >> 597                             xfs_ilock(xip, XFS_ILOCK_EXCL|iolock);
>> >
>> > Make sense. Can you run 'l *(xfs_ilock+0x2c)' as well? I just need to
>> > confirm which lock it has blocked on.
>>
>> (gdb) l *(xfs_ilock+0x2c)
>> 0xffffffff81221001 is in xfs_ilock (fs/xfs/linux-2.6/mrlock.h:48).
>> 43              down_read_nested(&mrp->mr_lock, subclass);
>> 44      }
>> 45
>> 46      static inline void mrupdate_nested(mrlock_t *mrp, int subclass)
>> 47      {
>> 48              down_write_nested(&mrp->mr_lock, subclass);
>> 49      #ifdef DEBUG
>> 50              mrp->mr_writer = 1;
>> 51      #endif
>> 52      }
>
> OK, that doesn't help - it followed into the inline function
> rather than telling me which of the two calls in the function it
> was. I guess I'll need the disassembly output to work it out.
> Can you send the output of "disass xfs_ilock" instead? Thanks.

Looks like it's the first one.

(gdb) disass xfs_ilock
Dump of assembler code for function xfs_ilock:
0xffffffff81220fd5 <xfs_ilock+0>:       push   %rbp
0xffffffff81220fd6 <xfs_ilock+1>:       test   $0x1,%sil
0xffffffff81220fda <xfs_ilock+5>:       mov    %rsp,%rbp
0xffffffff81220fdd <xfs_ilock+8>:       push   %r14
0xffffffff81220fdf <xfs_ilock+10>:      push   %r13
0xffffffff81220fe1 <xfs_ilock+12>:      mov    %rdi,%r13
0xffffffff81220fe4 <xfs_ilock+15>:      push   %r12
0xffffffff81220fe6 <xfs_ilock+17>:      mov    %esi,%r12d
0xffffffff81220fe9 <xfs_ilock+20>:      push   %rbx
0xffffffff81220fea <xfs_ilock+21>:      je     0xffffffff81221003 <xfs_ilock+46>
0xffffffff81220fec <xfs_ilock+23>:      and    $0xff0000,%esi
0xffffffff81220ff2 <xfs_ilock+29>:      lea    0xe8(%rdi),%rdi
0xffffffff81220ff9 <xfs_ilock+36>:      shr    $0x10,%esi
0xffffffff81220ffc <xfs_ilock+39>:      callq  0xffffffff8105b725
<down_write_nested>
0xffffffff81221001 <xfs_ilock+44>:      jmp    0xffffffff8122101e <xfs_ilock+73>
0xffffffff81221003 <xfs_ilock+46>:      test   $0x2,%sil
0xffffffff81221007 <xfs_ilock+50>:      je     0xffffffff8122101e <xfs_ilock+73>
0xffffffff81221009 <xfs_ilock+52>:      and    $0xff0000,%esi
0xffffffff8122100f <xfs_ilock+58>:      lea    0xe8(%rdi),%rdi
0xffffffff81221016 <xfs_ilock+65>:      shr    $0x10,%esi
0xffffffff81221019 <xfs_ilock+68>:      callq  0xffffffff8105b792
<down_read_nested>
0xffffffff8122101e <xfs_ilock+73>:      test   $0x4,%r12b
0xffffffff81221022 <xfs_ilock+77>:      je     0xffffffff81221038 <xfs_ilock+99>
0xffffffff81221024 <xfs_ilock+79>:      mov    %r12d,%esi
0xffffffff81221027 <xfs_ilock+82>:      lea    0x88(%r13),%rdi
0xffffffff8122102e <xfs_ilock+89>:      shr    $0x18,%esi
0xffffffff81221031 <xfs_ilock+92>:      callq  0xffffffff8105b725
<down_write_nested>
0xffffffff81221036 <xfs_ilock+97>:      jmp    0xffffffff81221050
<xfs_ilock+123>
0xffffffff81221038 <xfs_ilock+99>:      test   $0x8,%r12b
0xffffffff8122103c <xfs_ilock+103>:     je     0xffffffff81221050
<xfs_ilock+123>
0xffffffff8122103e <xfs_ilock+105>:     mov    %r12d,%esi
0xffffffff81221041 <xfs_ilock+108>:     lea    0x88(%r13),%rdi
0xffffffff81221048 <xfs_ilock+115>:     shr    $0x18,%esi
0xffffffff8122104b <xfs_ilock+118>:     callq  0xffffffff8105b792
<down_read_nested>
0xffffffff81221050 <xfs_ilock+123>:     cmpl   $0x0,0x863891(%rip)
   # 0xffffffff81a848e8
0xffffffff81221057 <xfs_ilock+130>:     mov    0x8(%rbp),%r14
0xffffffff8122105b <xfs_ilock+134>:     je     0xffffffff8122107e
<xfs_ilock+169>
0xffffffff8122105d <xfs_ilock+136>:     mov    0x86389c(%rip),%rbx
   # 0xffffffff81a84900
0xffffffff81221064 <xfs_ilock+143>:     test   %rbx,%rbx
0xffffffff81221067 <xfs_ilock+146>:     je     0xffffffff8122107e
<xfs_ilock+169>
0xffffffff81221069 <xfs_ilock+148>:     mov    %r14,%rdx
0xffffffff8122106c <xfs_ilock+151>:     mov    %r12d,%esi
0xffffffff8122106f <xfs_ilock+154>:     mov    %r13,%rdi
0xffffffff81221072 <xfs_ilock+157>:     callq  *(%rbx)
0xffffffff81221074 <xfs_ilock+159>:     add    $0x8,%rbx
0xffffffff81221078 <xfs_ilock+163>:     cmpq   $0x0,(%rbx)
0xffffffff8122107c <xfs_ilock+167>:     jmp    0xffffffff81221067
<xfs_ilock+146>
0xffffffff8122107e <xfs_ilock+169>:     pop    %rbx
0xffffffff8122107f <xfs_ilock+170>:     pop    %r12
0xffffffff81221081 <xfs_ilock+172>:     pop    %r13
0xffffffff81221083 <xfs_ilock+174>:     pop    %r14
0xffffffff81221085 <xfs_ilock+176>:     leaveq
0xffffffff81221086 <xfs_ilock+177>:     retq
---Type <return> to continue, or q <return> to quit---
End of assembler dump.

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Dave Chinner
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Dave Chinner
On Wed, Aug 04, 2010 at 11:40:02AM -0400, Ilia Mirkin wrote:

> On Wed, Aug 4, 2010 at 2:53 AM, Dave Chinner <[hidden email]> wrote:
> > On Wed, Aug 04, 2010 at 12:39:08AM -0400, Ilia Mirkin wrote:
> >> On Wed, Aug 4, 2010 at 12:27 AM, Dave Chinner <[hidden email]> wrote:
> >> > On Tue, Aug 03, 2010 at 09:15:53PM -0400, Ilia Mirkin wrote:
> >> >> On Tue, Aug 3, 2010 at 8:47 PM, Dave Chinner <[hidden email]> wrote:
> >> >> > Ilia,
> >> >> >
> >> >> > Can you send me the output of this for your kernel that the
> >> >> > traces came from:
> >> >> >
> >> >> > $ gdb <path/to/vmlinux>
> >> >> > (gdb) l *( xfs_write+0x2cc)
> >> >> >
> >> >> > You can run it against the vmlinux file in the kernel build
> >> >> > directory.  Basically I need to know which xfs_ilock() call in
> >> >> > xfs_write() one of the mysqld-test processes is stuck on.
....
> > OK, that doesn't help - it followed into the inline function
> > rather than telling me which of the two calls in the function it
> > was. I guess I'll need the disassembly output to work it out.
> > Can you send the output of "disass xfs_ilock" instead? Thanks.
>
> Looks like it's the first one.

Yup, that's what it looks like to me, too. That means it's the
XFS_IOLOCK_EXCL it is blocking trying to get, not the
XFS_ILOCK_EXCL. Well, that makes sense, and tells me that the
lockdep "held locks" output at the end  also includes locks we are
trying to obtain. That clears up that confusion.

Ok, I'll work from there. Thanks, Ilia.

Cheers,

Dave.
--
Dave Chinner
[hidden email]

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Dave Chinner
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Dave Chinner
In reply to this post by Ilia Mirkin-2
On Tue, Aug 03, 2010 at 06:30:36PM -0400, Ilia Mirkin wrote:

> On Sun, Jul 18, 2010 at 7:50 PM, Dave Chinner <[hidden email]> wrote:
> > On Sat, Jul 17, 2010 at 09:35:33PM -0400, Ilia Mirkin wrote:
> >> On Sat, Jul 17, 2010 at 9:20 PM, Dave Chinner <[hidden email]> wrote:
> >> > On Sat, Jul 17, 2010 at 12:01:11AM -0400, Ilia Mirkin wrote:
> >> > I can't find a thread that holds the XFS inode lock that everything
> >> > is waiting on. I think it is the ILOCK, but none of the threads in
> >> > this trace should be holding it where they are blocked. IOWs, the
> >> > output does not give me enough information to get to the root cause.
> >>
> >> In case this happens again, was there something more useful I could
> >> have collected? Should I have grabbed all task states?
> >
> > All the task states, including the running tasks, is probably a good
....
> > Though I suspect the only way to get to the bottom of it will
> > be to work out a reproducable test case....
>
> I felt a little bothered by this issue, so I dug in a little further.
> I basically created a sample mysql, tar, and it seemed that having
> memory pressure helped, so I created an allocator as well.

[snip test case]

Nice work.

I think I see the problem, and it's not something that
can be easily fixed. The test-tar process is blocked in
xfs_ilock_map_shared() trying to get the ip->i_lock in exclusive
mode.

You have a couple of IO's waiting to complete in the xfsdatad's
waiting to get the ip->i_Ilock in exclusive mode in a non-blocking
fashiion:

schedule_timeout
schedule_timeout_uninterruptible
xfs_end_io
worker_thread
kthread
kernel_thread_helper

There are test-mysqld processes stuck doing page cache invalidation,
waiting for IO completion to occur (which can't occur because of the
above stuck IOs). These hold the ip->i_iolock in shared mode.

2882, 2887

io_schedule
sync_page
lock_page
invalidate_inode_pages2_range
generic_file_direct_write
xfs_write
xfs_file_aio_write
do_sync_write
vfs_write
sys_write

There's a couple of test-mysqld processes stuck on the
inode->i_mutex here:

2883, 2884, 2886, 2888, 2889, 2891

__mutex_lock_common
mutex_lock_nested
generic_file_llseek
vfs_llseek
sys_lseek

and here:

2890, 2892

__mutex_lock_common
mutex_lock_nested
xfs_write
xfs_file_aio_write
do_sync_write
vfs_write
sys_write

There's a test-mysqld process stuck waiting for either
the ip->i_iolock or ip->i_ilock in exclusive mode:

2885: xfs_write+0x2cc/0x793

rwsem_down_failed_common
rwsem_down_write_failed
call_rwsem_down_write_failed
xfs_ilock
xfs_write
xfs_file_aio_write
do_sync_write
vfs_write
sys_write

And then there is the test-tar process, which is blocked on the
ip->i_ilock trying to get it in _shared_ mode, holding the ip->i_iolock
in shared mode

2897

rwsem_down_failed_common
rwsem_down_read_failed
call_rwsem_down_read_failed
xfs_ilock
xfs_ilock_map_shared
xfs_iomap
__xfs_get_blocks
xfs_get_blocks
do_mpage_readpage
mpage_readpages
xfs_vm_readpages
__do_page_cache_readahead
ra_submit
ondemand_readahead
page_cache_sync_readahead
generic_file_aio_read
xfs_read
xfs_file_aio_read
do_sync_read
vfs_read
sys_read

And a sync process stuck waiting for io completion (no locks held)

xfs_ioend_wait
xfs_sync_inode_data
xfs_inode_ag_walk
xfs_inode_ag_iterator
xfs_sync_data
xfs_quiesce_data
xfs_fs_sync_fs
sync_quota_sb
__sync_filesystem
sync_filesystems
sys_sync

So in summary, we have:

 1) xfsdatad sleeping for a clock tick because it can't get the
   ip->i_lock(EXCL) without blocking
 2) 2882, 2887 waiting for IO completion, holding ip->i_iolock(SHARED)
   (blocked behind #1))
 3) 2885 waiting on an ip->i_iolock(EXCL), holding the i_mutex and
   blocked behind #2) and #5)
 4) 2883, 2884, 2886, 2888, 2889, 2891, 2890, 2892 waiting on
   i_mutex, blocked behind #3)
 5) 2897 doing buffered read, holding ip->i_iolock(SHARED),
  waiting on ip->i_lock(SHARED)

So, this leaves me with the question - what is holding the
ip->i_ilock()? Everything is blocked waiting for it and there are no
reported holders, either through stack trace analysis or from the
lockdep lock holder report.

I said this previously:

>> > I can't find a thread that holds the XFS inode lock that everything
>> > is waiting on. I think it is the ILOCK, but none of the threads in
>> > this trace should be holding it where they are blocked. IOWs, the
>> > output does not give me enough information to get to the root cause.

And my conclusion is the same from this analysis - I cannot see why
everything has stopped waiting on an ILOCK that nobody should hold
and lockdep is saying that nobody does actually hold.

I'm starting to suspect memory corruption or something similar that
is corrupting the inode lock state, but I don't hold much hope of
tracking down something like that if it takes hours to reproduce.

Ilia, while I try to think of the next step to take, can you try
a couple of things to see if the hang can be caused faster? Maybe
trying these different initial conditions:

        - use a smaller test file
        - allocate the file with worst case fragmentation by
          writing it backwards in synchronous 4k blocks before
          running the test
        - preallocating the test file with fallocate()

Also, it woul dbe interesting to find out if a newer kernel also
hangs in a similar manner, say 2.6.35?

Cheers,

Dave.
--
Dave Chinner
[hidden email]

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Ilia Mirkin-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Ilia Mirkin-2
On Thu, Aug 5, 2010 at 2:25 AM, Dave Chinner <[hidden email]> wrote:
> a couple of things to see if the hang can be caused faster? Maybe
> trying these different initial conditions:

Together or independently?

>
>        - use a smaller test file

OK. My theory [based on no evidence] was that it had to be bigger than
RAM size. I'll try less though.

>        - allocate the file with worst case fragmentation by
>          writing it backwards in synchronous 4k blocks before
>          running the test

I created the file with

dd if=/dev/zero of=... bs=1M count=...

How would I "write backwards"? Like truncate and then seek to the end,
write, seek back, write, etc? Is there a utility that might do it
already?

>        - preallocating the test file with fallocate()

Sure.

>
> Also, it woul dbe interesting to find out if a newer kernel also
> hangs in a similar manner, say 2.6.35?

Will do.

--
Ilia Mirkin
[hidden email]

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
Dave Chinner
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

Dave Chinner
On Thu, Aug 05, 2010 at 02:42:38AM -0400, Ilia Mirkin wrote:
> On Thu, Aug 5, 2010 at 2:25 AM, Dave Chinner <[hidden email]> wrote:
> > a couple of things to see if the hang can be caused faster? Maybe
> > trying these different initial conditions:
>
> Together or independently?

independently first - then if that does't work, maybe together?

> >        - allocate the file with worst case fragmentation by
> >          writing it backwards in synchronous 4k blocks before
> >          running the test
>
> I created the file with
>
> dd if=/dev/zero of=... bs=1M count=...
>
> How would I "write backwards"? Like truncate and then seek to the end,
> write, seek back, write, etc? Is there a utility that might do it
> already?

For a 1GB file:

$ for i in `seq 256000 1 0`; do
> dd if=/dev/zero of=testfile bs=4k count=1 seek=$i conv=notrunc,sync
> done

Cheers,

Dave.
--
Dave Chinner
[hidden email]

_______________________________________________
xfs mailing list
[hidden email]
http://oss.sgi.com/mailman/listinfo/xfs
qiyansun
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: XFS hung on 2.6.33.3 kernel

qiyansun
This post has NOT been accepted by the mailing list yet.
Hi,

We ran into a similar situation described in this email thread on the 2.6.32.54 kernel using SLES11SP1. Are there any resolutions or patches available for this issue?

Thanks,
Qiyan
Free forum by Nabble Edit this page