2

We have implemented MongoDB master-slave replication and database got deleted.

We restored deleted files from Mongo data directory using extundelete command.

Undelete Blog https://github.com/RIKSOF/development/wiki/Restoring-deleted-files-on-Linux

We wanted to restore at least one collection for which we got the WT (wired timer) file. We are trying to restore that file using the following commands and we are getting the following errors

Recovering a WiredTiger collection from a corrupt MongoDB installation

wget http://source.wiredtiger.com/releases/wiredtiger-2.7.0.tar.bz2 tar xvf wiredtiger-2.7.0.tar.bz2 cd wiredtiger-2.7.0 sudo apt-get install libsnappy-dev build-essential ./configure --enable-snappy make

./wt -v -h ../mongo-bak -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage collection-2657--1723320556100349955.wt

Error -

./wt -v -h ../mongo-bak -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage collection-246--7553069514495955510.wt [1488888117:36780][13536:0x7f7b633cd740], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 1955562709 doesn't match expected checksum of 3146787951 [1488888117:36809][13536:0x7f7b633cd740], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value [1488888117:36817][13536:0x7f7b633cd740], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic lt-wt: WT_PANIC: WiredTiger library panic

Sandip Pingle
  • 647
  • 2
  • 14
  • 37

3 Answers3

5

This could be related to corrupted .wt files (e.g. WiredTiger.wt/WiredTiger.turtle) as per SERVER-31076 bug report.

Try to run repair on all databases by the following command:

mongod --repair --dbpath /path/to/data/db

Also make sure all data/db files have the right read and write permission.

kenorb
  • 155,785
  • 88
  • 678
  • 743
  • I have an issue when i tried to do mongod --dbpath /var/lib/mongodb/ --repair WT_TRY_SALVAGE: database corruption detected Do you have an idea of how to fix it? (mongo 4.0) My mongo is broken since disk space failure ....) – thomas poidevin Jun 03 '19 at 13:06
  • If `mongod --repair` didn't help, I'm not sure, it's better if you can ask another question with the relevant error. – kenorb Jun 03 '19 at 15:50
  • Did the trick for me. The logs contained errors related to WireTiger and checksum mismatch. – Jose Quijada Jan 19 '21 at 21:40
  • it doesn't solve the issue. It just try to run the same files, and obviously the same error is thrown. – asa Apr 22 '21 at 15:56
2

In my case, I had recovered mac from Time Machine. I had mongodb installed via brew install. When I started mongo, the service did not start cleanly. 

vi /usr/local/var/log/mongodb/mongo.log
enter code here
2020-01-30T17:06:10.655-0500 E  STORAGE  [initandlisten] WiredTiger error (0) [1580421970:655376][62181:0x112233dc0], file:WiredTiger.wt, connection: __wt_block_read_off, 283: WiredTiger.wt: read checksum error for 4096B block at offset 12288: block header checksum of 0x325e318 doesn't match expected checksum of 0x65ef0e12 Raw: [1580421970:655376][62181:0x112233dc0], file:WiredTiger.wt, connection: __wt_block_read_off, 283: WiredTiger.wt: read checksum error for 4096B block at offset 12288: block header checksum of 0x325e318 doesn't match expected checksum of 0x65ef0e12

What I saw was that because of a corrupt mongodb collection. So, I ran the cleanup command for the brew install, which has the db in a different location.

mongod --repair --dbpath /usr/local/var/mongodb

    2020-01-30T17:06:11.741-0500 I  STORAGE  [initandlisten] Verify failed on uri table:collection-12--8361386101775862971. Running a salvage operation.
2020-01-30T17:06:11.745-0500 I  STORAGE  [initandlisten] Repairing collection wlchat.jobs
2020-01-30T17:06:11.765-0500 I  STORAGE  [initandlisten] WiredTiger progress WT_SESSION.verify 100
2020-01-30T17:06:11.769-0500 I  STORAGE  [initandlisten] WiredTiger progress WT_SESSION.verify 200
2020-01-30T17:06:11.774-0500 I  STORAGE  [initandlisten] WiredTiger progress WT_SESSION.verify 300
2020-01-30T17:06:11.783-0500 I  STORAGE  [initandlisten] WiredTiger progress WT_SESSION.verify 400
2020-01-30T17:06:11.789-0500 I  STORAGE  [initandlisten] WiredTiger progress WT_SESSION.verify 500
2020-01-30T17:06:11.793-0500 I  STORAGE  [initandlisten] WiredTiger progress WT_SESSION.verify 600
2020-01-30T17:06:11.795-0500 I  STORAGE  [initandlisten] Verify succeeded on uri table:collection-10--8361386101775862971. Not salvaging.
2020-01-30T17:06:11.795-0500 I  STORAGE  [initandlisten] Repairing collection wlchat.SovrenJobs
2020-01-30T17:06:11.801-0500 I  STORAGE  [initandlisten] Verify succeeded on uri table:collection-0-8794251073515214768. Not salvaging.
2020-01-30T17:06:11.814-0500 I  INDEX    [initandlisten] index build: starting on wlchat.users properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "wlchat.users" } using method: Foreground

and restart mongodb-community

brew services restart mongodb-community.
==> Successfully started `mongodb-community` (label: homebrew.mxcl.mongodb-community)

Things are back to normal.

Krishna Vedula
  • 1,643
  • 1
  • 27
  • 31
0

I got this error and trying a lot of way to resolve but not reached to successfully.

I am currently using 5.0.1 mongodb standalone version,

  • removed currently installed mongodb ( 5.0.1 )
  • installed latest version ( 6.0 )

and reapaired all corrupted *.wt file and verified.

mongod --repair --dbpath /path/to/data/db
CMLDMR
  • 334
  • 2
  • 12