MongoDB:记一次生产环境中mongo出现的严重出错与排查解决

这篇具有很好参考价值的文章主要介绍了MongoDB:记一次生产环境中mongo出现的严重出错与排查解决。希望对大家有所帮助。如果存在错误或未考虑完全的地方,请大家不吝赐教,您也可以点击"举报违法"按钮提交疑问。

造成此种错误的原因有如下几种常见情况:

* 系统磁盘已满导致mongo无法向文件系统写数据。

* 系统突然死机(或系统重启)造成mongo文件系统被损坏。

* 使用非正当方法强制停止mongo服务,如:kill -1/-9的方式。

* mongo文件系统遭篡改。

还有很多种原因............

启动mongod失败

#mongod -port 27019

2017-05-23T09:12:43.023+0800 I CONTROL  [initandlisten] MongoDB starting : pid=253port=27019 dbpath=/data/db 64-bit host=da6056799173

2017-05-23T09:12:43.023+0800 I CONTROL  [initandlisten] db version v3.2.8

2017-05-23T09:12:43.023+0800 I CONTROL  [initandlisten] git version: ed70e33130c977bda0024c125b56d159573dbaf0

2017-05-23T09:12:43.023+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL1.0.1e-fips 11 Feb 2013

2017-05-23T09:12:43.023+0800 I CONTROL  [initandlisten] allocator: tcmalloc

2017-05-23T09:12:43.023+0800 I CONTROL  [initandlisten] modules: none

2017-05-23T09:12:43.023+0800 I CONTROL  [initandlisten] build environment:

2017-05-23T09:12:43.023+0800 I CONTROL  [initandlisten]     distmod: rhel70

2017-05-23T09:12:43.023+0800 I CONTROL  [initandlisten]     distarch: x86_64

2017-05-23T09:12:43.023+0800 I CONTROL  [initandlisten]     target_arch: x86_64

2017-05-23T09:12:43.023+0800 I CONTROL  [initandlisten] options: { net: { port: 27019}, security: { authorization: "enabled" } }

2017-05-23T09:12:43.045+0800 I -        [initandlisten] Detected data files in/data/db created by the 'wiredTiger' storage engine, so setting the activestorage engine to 'wiredTiger'.

2017-05-23T09:12:43.045+0800 W -        [initandlisten] Detected unclean shutdown -/data/db/mongod.lock is not empty.

2017-05-23T09:12:43.045+0800 W STORAGE  [initandlisten] Recovering data from the lastclean checkpoint.

2017-05-23T09:12:43.045+0800 I STORAGE  [initandlisten] wiredtiger_open config:create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),

2017-05-23T09:12:45.028+0800 E STORAGE  [initandlisten] WiredTiger (0)[1495501965:28163][253:0x7fdf1a440dc0], file:index-59-8962705381866868852.wt,WT_CURSOR.insert: read checksum error for 12288B block at offset 124456960:calculated block checksum of 2057573707 doesn't match expected checksum of1789932834

2017-05-23T09:12:45.028+0800 E STORAGE  [initandlisten] WiredTiger (0)[1495501965:28237][253:0x7fdf1a440dc0], file:index-59-8962705381866868852.wt,WT_CURSOR.insert: index-59-8962705381866868852.wt: encountered an illegal fileformat or internal value

2017-05-23T09:12:45.028+0800 E STORAGE  [initandlisten] WiredTiger (-31804)[1495501965:28249][253:0x7fdf1a440dc0], file:index-59-8962705381866868852.wt,WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTigerlibrary panic

2017-05-23T09:12:45.028+0800 I -        [initandlisten] Fatal Assertion 28558

2017-05-23T09:12:45.028+0800 I -        [initandlisten]

***aborting after fassert() failure

2017-05-23T09:12:45.044+0800 F -        [initandlisten] Got signal: 6(Aborted).

 0x131cfa2 0x131c0f9 0x131c902 0x7fdf190ad1000x7fdf18d115f7 0x7fdf18d12ce8 0x12a68a2 0x10a0a53 0x1a7e43c 0x1a7e8fd 0x1a7ece40x19b2f97 0x19cfcaa 0x19d5280 0x19f5727 0x19c36cf 0x1a11c9e 0x1a8d673 0x1a308d20x1a8de67 0x1a07e57 0x1a00f4a 0x1088aff 0x1084dc3 0xfadd78 0x9b4a2d 0x96e17d0x7fdf18cfdb15 0x9b1169

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"400000","o":"F1CFA2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F1C0F9"},{"b":"400000","o":"F1C902"},{"b":"7FDF1909E000","o":"F100"},{"b":"7FDF18CDC000","o":"355F7","s":"gsignal"},{"b":"7FDF18CDC000","o":"36CE8","s":"abort"},{"b":"400000","o":"EA68A2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CA0A53"},{"b":"400000","o":"167E43C","s":"__wt_eventv"},{"b":"400000","o":"167E8FD","s":"__wt_err"},{"b":"400000","o":"167ECE4","s":"__wt_panic"},{"b":"400000","o":"15B2F97","s":"__wt_bm_read"},{"b":"400000","o":"15CFCAA","s":"__wt_bt_read"},{"b":"400000","o":"15D5280","s":"__wt_page_in_func"},{"b":"400000","o":"15F5727","s":"__wt_row_search"},{"b":"400000","o":"15C36CF","s":"__wt_btcur_insert"},{"b":"400000","o":"1611C9E"},{"b":"400000","o":"168D673"},{"b":"400000","o":"16308D2","s":"__wt_log_scan"},{"b":"400000","o":"168DE67","s":"__wt_txn_recover"},{"b":"400000","o":"1607E57","s":"__wt_connection_workers"},{"b":"400000","o":"1600F4A","s":"wiredtiger_open"},{"b":"400000","o":"C88AFF","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C84DC3"},{"b":"400000","o":"BADD78","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5B4A2D","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"56E17D","s":"main"},{"b":"7FDF18CDC000","o":"21B15","s":"__libc_start_main"},{"b":"400000","o":"5B1169"}],"processInfo":{"mongodbVersion" : "3.2.8", "gitVersion" :"ed70e33130c977bda0024c125b56d159573dbaf0","compiledModules" : [], "uname" : { "sysname" :"Linux", "release" : "3.10.0-514.2.2.el7.x86_64","version" : "#1 SMP Tue Dec 6 23:06:41 UTC 2016","machine" : "x86_64" }, "somap" : [ {"elfType" : 2, "b" : "400000","buildId" : "5F3E5C743BB6FE5AA37A0C943A2741BC6F69AA7E" }, {"b" : "7FFD5F7AD000", "elfType" : 3,"buildId" : "183CE4B56A9471419F233CCEF078E0504837ABF5" }, {"b" : "7FDF19FC6000", "path" : "/lib64/libssl.so.10","elfType" : 3, "buildId" :"478D01A08B923A251D755BB421F3EBAF9F2982C1" }, { "b" :"7FDF19BDE000", "path" :"/lib64/libcrypto.so.10", "elfType" : 3, "buildId": "42AAFD25E9B5F4CE2EFE6309491445B1A92A575D" }, { "b" :"7FDF199D6000", "path" : "/lib64/librt.so.1","elfType" : 3, "buildId" :"CB0D2C9F29DBD13C47E7D2EEFB94B35835698CCA" }, { "b" :"7FDF197D2000", "path" : "/lib64/libdl.so.2","elfType" : 3, "buildId" :"091060A163E7EDA25572F3B1BAF2E8F80209C00E" }, { "b" :"7FDF194D0000", "path" : "/lib64/libm.so.6","elfType" : 3, "buildId" :"F9DF294FB70243549DCB643F1322BB20E70E9FE8" }, { "b" :"7FDF192BA000", "path" : "/lib64/libgcc_s.so.1","elfType" : 3, "buildId" :"6AA1DCC4DE7F1836344949857FC2017278631FFD" }, { "b" :"7FDF1909E000", "path" : "/lib64/libpthread.so.0","elfType" : 3, "buildId" :"723F0AC75EF88E778940AE8A8BC30141D85B116A" }, { "b" :"7FDF18CDC000", "path" : "/lib64/libc.so.6","elfType" : 3, "buildId" :"088D48A9AB5A512D9F75BA3D66B6CF77EB6588F9" }, { "b" :"7FDF1A233000", "path" : "/lib64/ld-linux-x86-64.so.2","elfType" : 3, "buildId" :"09E1BB4D034C7263810A41100647068858A7ECB6" }, { "b" :"7FDF18A90000", "path" :"/lib64/libgssapi_krb5.so.2", "elfType" : 3,"buildId" : "D46A230FFF4A7B808B3CFC213D31FCAC542FB504" }, {"b" : "7FDF187AB000", "path" :"/lib64/libkrb5.so.3", "elfType" : 3, "buildId" :"6D6136A0E795420B05854DEF13A10C226FE9CCB2" }, { "b" :"7FDF185A7000", "path" : "/lib64/libcom_err.so.2","elfType" : 3, "buildId" :"3A1166709F88740C49E060731832E3FAD2DFB66B" }, { "b" :"7FDF18375000", "path" :"/lib64/libk5crypto.so.3", "elfType" : 3,"buildId" : "AA97A848DD7C9E57B06EC913E10D420AEBBCE027" }, {"b" : "7FDF1815F000", "path" : "/lib64/libz.so.1","elfType" : 3, "buildId" :"1982C8CDAE90F898D1AD26DC07E807333B4789D0" }, { "b" :"7FDF17F50000", "path" :"/lib64/libkrb5support.so.0", "elfType" : 3,"buildId" : "AEF6C3D3C5152F339942041519A106FC055DAF71" }, {"b" : "7FDF17D4C000", "path" : "/lib64/libkeyutils.so.1","elfType" : 3, "buildId" :"2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" :"7FDF17B32000", "path" : "/lib64/libresolv.so.2","elfType" : 3, "buildId" :"D02DC134F38F06F3885231FD2486D5EF4796E5F9" }, { "b" :"7FDF1790D000", "path" :"/lib64/libselinux.so.1", "elfType" : 3,"buildId" : "82FF6B18E1E42825CC2D060F969479AD4AF2F62C" }, {"b" : "7FDF176AC000", "path" :"/lib64/libpcre.so.1", "elfType" : 3, "buildId" :"AE64AA461A26E01F60408013D361749D56DD0AE1" }, { "b" :"7FDF17487000", "path" : "/lib64/liblzma.so.5","elfType" : 3, "buildId" :"98131C9354279ABD39FD80D4BE5B3EC5678BD9E0" } ] }}

 mongod(_ZN5mongo15printStackTraceERSo+0x32)[0x131cfa2]

 mongod(+0xF1C0F9) [0x131c0f9]

 mongod(+0xF1C902) [0x131c902]

 libpthread.so.0(+0xF100) [0x7fdf190ad100]

 libc.so.6(gsignal+0x37) [0x7fdf18d115f7]

 libc.so.6(abort+0x148) [0x7fdf18d12ce8]

 mongod(_ZN5mongo13fassertFailedEi+0x82)[0x12a68a2]

 mongod(+0xCA0A53) [0x10a0a53]

 mongod(__wt_eventv+0x42C) [0x1a7e43c]

 mongod(__wt_err+0x8D) [0x1a7e8fd]

 mongod(__wt_panic+0x24) [0x1a7ece4]

 mongod(__wt_bm_read+0x77) [0x19b2f97]

 mongod(__wt_bt_read+0x1EA) [0x19cfcaa]

 mongod(__wt_page_in_func+0x180) [0x19d5280]

 mongod(__wt_row_search+0x677) [0x19f5727]

 mongod(__wt_btcur_insert+0x45F) [0x19c36cf]

 mongod(+0x1611C9E) [0x1a11c9e]

 mongod(+0x168D673) [0x1a8d673]

 mongod(__wt_log_scan+0x9D2) [0x1a308d2]

 mongod(__wt_txn_recover+0x4B7) [0x1a8de67]

 mongod(__wt_connection_workers+0x37)[0x1a07e57]

 mongod(wiredtiger_open+0x155A) [0x1a00f4a]

 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x77F)[0x1088aff]

 mongod(+0xC84DC3) [0x1084dc3]

 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598)[0xfadd78]

 mongod(_ZN5mongo13initAndListenEi+0x3DD)[0x9b4a2d]

 mongod(main+0x15D) [0x96e17d]

 libc.so.6(__libc_start_main+0xF5)[0x7fdf18cfdb15]

 mongod(+0x5B1169) [0x9b1169]

----- END BACKTRACE  -----

Aborted (core dumped)

网上推荐的添加参数 --repair依然无法修复

# mongod --repair

2017-05-23T09:31:20.854+0800 I CONTROL  [initandlisten] MongoDB starting : pid=322port=27017 dbpath=/data/db 64-bit host=da6056799173

2017-05-23T09:31:20.854+0800 I CONTROL  [initandlisten] db version v3.2.8

2017-05-23T09:31:20.854+0800 I CONTROL  [initandlisten] git version:ed70e33130c977bda0024c125b56d159573dbaf0

2017-05-23T09:31:20.854+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL1.0.1e-fips 11 Feb 2013

2017-05-23T09:31:20.854+0800 I CONTROL  [initandlisten] allocator: tcmalloc

2017-05-23T09:31:20.854+0800 I CONTROL  [initandlisten] modules: none

2017-05-23T09:31:20.854+0800 I CONTROL  [initandlisten] build environment:

2017-05-23T09:31:20.854+0800 I CONTROL  [initandlisten]     distmod: rhel70

2017-05-23T09:31:20.854+0800 I CONTROL  [initandlisten]     distarch: x86_64

2017-05-23T09:31:20.854+0800 I CONTROL  [initandlisten]     target_arch: x86_64

2017-05-23T09:31:20.854+0800 I CONTROL  [initandlisten] options: { repair: true }

2017-05-23T09:31:20.876+0800 I -        [initandlisten] Detected data files in/data/db created by the 'wiredTiger' storage engine, so setting the activestorage engine to 'wiredTiger'.

2017-05-23T09:31:20.876+0800 I STORAGE  [initandlisten] Detected WT journalfiles.  Running recovery from lastcheckpoint.

2017-05-23T09:31:20.876+0800 I STORAGE  [initandlisten] journal to nojournaltransition config:create,cache_size=18G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),

2017-05-23T09:31:20.982+0800 E STORAGE  [initandlisten] WiredTiger (-31803)[1495503080:982911][322:0x7f1a1ba1fdc0], txn-recover: Recovery failed:WT_NOTFOUND: item not found

2017-05-23T09:31:20.983+0800 I -        [initandlisten] Assertion:28718:-31803: WT_NOTFOUND: item not found

2017-05-23T09:31:20.984+0800 I STORAGE  [initandlisten] exception in initAndListen:28718 -31803: WT_NOTFOUND: item not found, terminating

2017-05-23T09:31:20.984+0800 I CONTROL  [initandlisten] dbexit:  rc: 100

解决方法将WiredTiger.turtle、WiredTiger.lock、mongod.lock文件进行移除,备份到非mongo dbpath指定的路径下即可

之后mongo服务即可正常启动。文章来源地址https://www.toymoban.com/news/detail-413096.html

到了这里,关于MongoDB:记一次生产环境中mongo出现的严重出错与排查解决的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处: 如若内容造成侵权/违法违规/事实不符,请点击违法举报进行投诉反馈,一经查实,立即删除!

领支付宝红包 赞助服务器费用

相关文章

  • 记录连接mongo突然出现:Cannot connect to MongoDB. No suitable servers founderverSelection TimeoutMS神奇解决方法

    之前使用mongo好好的,今天打开mongo,突然出现了一个异常,怎么也连不上服务器。。。 Cannot connect to MongoDB. No suitable servers found: serverSelectionTimeoutMS expired: [connection timeout calling ismaster on ‘xxxxxxxxxxx:9800’] 排查一:查看mongo服务发现服务正常,排除服务没有出现异常 排查二:查

    2024年02月11日
    浏览(45)
  • 记一次MySQL从节点服务器宕机重启后,从节点出现主键冲突异常的处理

    MySQL 5.7 非GTID模式多线程复制。 某MySQL数据库从节点因故障宕机(因故障直接宕机,非正常关闭),重启之后发现复制状态异常,show slave的结果中Slave_SQL_Running为No,错误代码为1062 error code,从系统表performance_schema.replication_applier_status_by_worker以及error log中显示某条数据因为已

    2024年02月19日
    浏览(35)
  • 记一次nginx反代m3u8资源站出现的问题(upstream和ssl等)

    本人小白,纯粹的给自己记录一下哈,大佬勿喷! 规定一下:资源站的域名 称为目标域名;本地做反向代理的域名 称为反代域名。好分辨 在最近一次本地测试中,因为资源站设定原因,所以需要自行反代网站才能正常使用,那么就只有配置nginx反向代理目标域名,于是就简

    2024年02月20日
    浏览(27)
  • 记一次正式环境升级docker服务基础进行版本异常

    因为服务的httpd和tomcat基础镜像版本比较旧,漏洞多,需要升级至最新版本。在本地环境和测试环境都是直接将dockerfile中的FROM基础镜像升级至最新: httpd:由httpd:2.4.52-alpine升级至httpd:2.4.57 tomcat:由4年前的tomcat:last升级至tomcat:9-jdk21-openjdk-slim 但是在正式环境docker容器都启动失

    2024年02月07日
    浏览(33)
  • 记一次pdjs时安装glob出现,npm ERR! code ETARGET和npm ERR! code ELIFECYCLE

    如往常一样,我使用pdjs来编译proto文件,但出现了以下报错: 大致就是pdjs的util在尝试执行` npm install glob@^7.2.1 escodegen@^1.13.0 `时出错了 尝试手动执行安装,` escodegen `被正确安装,但` glob@^7.2.1 `出错 也就是找不到符合条件的版本为7.2.1的包 执行` npm view glob versions `,出来了很

    2024年02月05日
    浏览(39)
  • 记一次linux系统使用dockerfile编写容器支持中文字体环境

    今天研发开发一个网上受理需要容器支持中文环境否则服务启动起来会出现中文乱码。 解决思路: 1、先把jar把在本地环境直接加参数运行如果运行起来无乱码说明本地的字体文件就可以直接COPY到容器中使用,如果还是乱码就说明本地的字体文件不支持中文环境,就需要下载

    2024年02月09日
    浏览(38)
  • 记一次配置文件spring.profiles.active环境变量不生效的问题

    三个环境的配置文件 分别是grayTg, proTg, testTg application.properties设置环境变量由IDEA统一管理  Maven中配置profile能让你为不同的环境定义不同的构建;包括本地测试环境,灰度环境, 生产环境这三个环境。可以用如下的标签配置在pom.xml文件中 本地Docker打包时, 无法解析环境配置

    2024年01月17日
    浏览(35)
  • 记录一次生产环境Rancher故障

    目录 一、运行环境问题描述 二、问题分析 三、问题处理   一、运行环境问题描述   Rancher版本:2.4.17 操作系统:CentOS 7.2 Docker : 19.03.15   Rancher正常运行突然打不开WEB界面,排查Rancher运行发现无端口 80、443,如下图:  排查Rancher日志一直在报如下错误:

    2024年02月07日
    浏览(30)
  • 【Qt 学习之路】记一次安装 Qt5.12.12 安卓环境的失败案例

    安装的 Qt5.12.12 版本 Qt下载地址: https://download.qt.io/archive/qt/ 安装Qt,可能会碰到“qt.tool.perl”安装程序错误,可以看我的记录解决: Qt开发 之 安装程序错误–安装进程(qt.tool.perl)的解决办法 JDK NDK SDK openssl 注意组合套件的版本和Qt的版本要对应起来!同时,安装路径不可

    2024年02月19日
    浏览(25)
  • 再记一次新电脑(ThinkBook)配置ubuntu环境 SimpleScreenRecorder安装与使用教程(Ubuntu)

    双十一入手新笔记本Thinkbook16+ i5-12500H 核显,安装ubuntu18.04,配置环境 1. 系统安装 2. 外观软件双系统时间同步 3. slam 环境 吐槽一句,配置完环境,发现没办法外接入显示器,笔记本自身的显示器也没识别出来,在设置的显示里面是未知的Display , 在软件和更新里面附加驱动没

    2024年02月14日
    浏览(33)

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

博客赞助

微信扫一扫打赏

请作者喝杯咖啡吧~博客赞助

支付宝扫一扫领取红包,优惠每天领

二维码1

领取红包

二维码2

领红包