问题描述:
有3个节点做为glusterfs存储集群,通过ganesha对客户端暴露NFS共享卷。客户端通过NFS挂载卷并持续写入大文件。某日现大量Jan 18 05:17:28 node-1 kernel: NMI watchdog: BUG: soft lockup - CPU107 stuck for 22s! [migration/107:551]错误日志(见附件message日志)导致死机。远程SSH以及IPMI均无响应。查sa日志发现相同时间段内的cpu使用率某几个核心使用率为100%(与日志中所报的cou core能对应上。比如:CPU#107代表第107个核心)
附件中包含了sa日志及message日志
另外,这个问题我们在intel平台上没有出现过
=====操作系统:
[root@localhost ~] cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
====内核版本
Linux localhost.localdomain 3.10.0-1062.12.1.el7.x86_64 1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
====CPU
[root@localhost ~] lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Thread(s) per core: 2
Core(s) per socket:
Socket(s): 2
NUMA node(s): 8
Vendor ID: HygonGenuine
CPU family: 24
Model: 2
Model name: Hygon C86 7375 32-core Processor
Stepping: 2
CPU MHz: 1999.774
BogoMIPS: 3999.54
Virtualization: AMD-V
L1d cache: 32K
L1i cache: 64K
L2 cache: 512K
L3 cache: 8192K
NUMA node0 CPU(s): 0-7,64-71
NUMA node1 CPU(s): 8-15,72-79
NUMA node2 CPU(s): 16-23,80-87
NUMA node3 CPU(s): 24-31,88-95
NUMA node4 CPU(s): 32-39,96-103
NUMA node5 CPU(s): 40-47,104-111
NUMA node6 CPU(s): 48-55,112-119
NUMA node7 CPU(s): 56-63,120-127
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate sme retpoline_amd ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
========内存
[root@localhost ~] lsmem
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x000000001fffffff 512M online no 0-3
0x0000000020000000-0x000000002fffffff 256M online yes 4-5
..........
Memory block size: 128M
Total online memory: 256G
Total offline memory: 0B
========glusterFS版本
[root@localhost ~] gluster --version
glusterfs 5.13
========NFS版本
[root@localhost ~] ganesha.nfsd -v
NFS-Ganesha Release = V2.7.6
暂无数据
暂无数据