Getting performance Improvement with keeping MongoDB Index files in a separate disk partition

When you start mongod with –wiredTigerDirectoryForIndexes, mongod stores indexes and collections in separate subdirectories under the data (i.e. –dbpath) directory. Specifically, mongod stores the indexes in a subdirectory named index and the collection data in a subdirectory named collection.

By using a symbolic link, you can specify a different location for the indexes. Specifically, when mongod instance is not running, move the index subdirectory to the destination and create a symbolic link named index under the data directory to the new destination.

Considering the advantage of paralysation of I/O, keeping huge Index files in a different disk/mount partition will definitely give a better performance in MongoDB when it comes to read/writes to both collection and Index data.

Below illustration will help you on how to enable to this 

[ root @ dbversity : ~ ] mkdir -p /datafiles /dev/shm/indexfiles
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-lv_root
97G 63G 29G 69% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 194M 103M 81M 57% /boot
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] mongod –dbpath /datafiles/ –wiredTigerDirectoryForIndexes –logpath ./mongo.log –port 27017 –fork
about to fork child process, waiting until server is ready for connections.
forked process: 32288
child process started successfully, parent exiting
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] ll -lhtr /datafiles/
total 52K
-rw-r–r– 1 root root 21 Nov 30 05:41 WiredTiger.lock
-rw-r–r– 1 root root 49 Nov 30 05:41 WiredTiger
-rw-r–r– 1 root root 4.0K Nov 30 05:41 WiredTiger.wt
-rw-r–r– 1 root root 842 Nov 30 05:41 WiredTiger.turtle
-rw-r–r– 1 root root 4.0K Nov 30 05:41 WiredTigerLAS.wt
drwxr-xr-x 2 root root 4.0K Nov 30 05:41 journal
-rw-r–r– 1 root root 4.0K Nov 30 05:41 sizeStorer.wt
-rw-r–r– 1 root root 4.0K Nov 30 05:41 _mdb_catalog.wt
-rw-r–r– 1 root root 6 Nov 30 05:41 mongod.lock
-rw-r–r– 1 root root 95 Nov 30 05:41 storage.bson
drwxr-xr-x 2 root root 4.0K Nov 30 05:41 collection
drwxr-xr-x 2 root root 4.0K Nov 30 05:41 index
drwxr-xr-x 2 root root 4.0K Nov 30 05:41 diagnostic.data
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] ll -lhtr /datafiles/collection/
total 8.0K
-rw-r–r– 1 root root 4.0K Nov 30 05:41 0-8110269602054927661.wt
-rw-r–r– 1 root root 4.0K Nov 30 05:41 2-8110269602054927661.wt
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] ll -lhtr /datafiles/index/
total 12K
-rw-r–r– 1 root root 4.0K Nov 30 05:41 1-8110269602054927661.wt
-rw-r–r– 1 root root 4.0K Nov 30 05:41 3-8110269602054927661.wt
-rw-r–r– 1 root root 4.0K Nov 30 05:41 4-8110269602054927661.wt
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] mongo mydb –quiet
MongoDB Enterprise >
MongoDB Enterprise > db.mycol.createIndex({name:1})
{
“createdCollectionAutomatically” : true,
“numIndexesBefore” : 1,
“numIndexesAfter” : 2,
“ok” : 1
}
MongoDB Enterprise >
MongoDB Enterprise > for(var i = 1; i <= 10000 ; i++){db.mycol.insert({ _id : i , name : ‘Separate partition for Index files’ });}

WriteResult({ “nInserted” : 1 })

MongoDB Enterprise >
MongoDB Enterprise >
MongoDB Enterprise > db.mycol.count()
10000
MongoDB Enterprise > use admin
switched to db admin
MongoDB Enterprise >
MongoDB Enterprise > db.shutdownServer()
server should be down…
2018-11-30T05:42:58.209-0500 I NETWORK [thread1] trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2018-11-30T05:42:58.209-0500 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2018-11-30T05:42:58.209-0500 I NETWORK [thread1] reconnect 127.0.0.1:27017 (127.0.0.1) failed failed
MongoDB Enterprise >
MongoDB Enterprise > exit
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] ll -lhtr /datafiles/index/
total 212K
-rw-r–r– 1 root root 44K Nov 30 05:42 7-8110269602054927661.wt
-rw-r–r– 1 root root 120K Nov 30 05:42 6-8110269602054927661.wt
-rw-r–r– 1 root root 16K Nov 30 05:42 4-8110269602054927661.wt
-rw-r–r– 1 root root 16K Nov 30 05:42 3-8110269602054927661.wt
-rw-r–r– 1 root root 16K Nov 30 05:42 1-8110269602054927661.wt
[ root @ dbversity : ~ ]

// Moving index data to new partition 

[ root @ dbversity : ~ ] mv /datafiles/index /dev/shm/indexfiles/
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] ll -lhtr /dev/shm/indexfiles/index/
total 212K
-rw-r–r– 1 root root 44K Nov 30 05:42 7-8110269602054927661.wt
-rw-r–r– 1 root root 120K Nov 30 05:42 6-8110269602054927661.wt
-rw-r–r– 1 root root 16K Nov 30 05:42 4-8110269602054927661.wt
-rw-r–r– 1 root root 16K Nov 30 05:42 3-8110269602054927661.wt
-rw-r–r– 1 root root 16K Nov 30 05:42 1-8110269602054927661.wt
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] ll -lhtr /datafiles/index
ls: cannot access /datafiles/index: No such file or directory
[ root @ dbversity : ~ ]

// Creating a symbolic link to Index folder after moving to new folder 

[ root @ dbversity : ~ ] ln -s /dev/shm/indexfiles/index/ /datafiles/index
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] ls -la /dev/shm/indexfiles/index/ /datafiles/index/
/datafiles/index/:
total 212
drwxr-xr-x 2 root root 140 Nov 30 05:42 .
drwxr-xr-x 3 root root 60 Nov 30 05:43 ..
-rw-r–r– 1 root root 16384 Nov 30 05:42 1-8110269602054927661.wt
-rw-r–r– 1 root root 16384 Nov 30 05:42 3-8110269602054927661.wt
-rw-r–r– 1 root root 16384 Nov 30 05:42 4-8110269602054927661.wt
-rw-r–r– 1 root root 122880 Nov 30 05:42 6-8110269602054927661.wt
-rw-r–r– 1 root root 45056 Nov 30 05:42 7-8110269602054927661.wt

/dev/shm/indexfiles/index/:
total 212
drwxr-xr-x 2 root root 140 Nov 30 05:42 .
drwxr-xr-x 3 root root 60 Nov 30 05:43 ..
-rw-r–r– 1 root root 16384 Nov 30 05:42 1-8110269602054927661.wt
-rw-r–r– 1 root root 16384 Nov 30 05:42 3-8110269602054927661.wt
-rw-r–r– 1 root root 16384 Nov 30 05:42 4-8110269602054927661.wt
-rw-r–r– 1 root root 122880 Nov 30 05:42 6-8110269602054927661.wt
-rw-r–r– 1 root root 45056 Nov 30 05:42 7-8110269602054927661.wt
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] ll -lhtr /datafiles/index
lrwxrwxrwx 1 root root 26 Nov 30 05:44 /datafiles/index -> /dev/shm/indexfiles/index/
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] ll -lhtr /dev/shm/indexfiles/index/
total 212K
-rw-r–r– 1 root root 44K Nov 30 05:42 7-8110269602054927661.wt
-rw-r–r– 1 root root 120K Nov 30 05:42 6-8110269602054927661.wt
-rw-r–r– 1 root root 16K Nov 30 05:42 4-8110269602054927661.wt
-rw-r–r– 1 root root 16K Nov 30 05:42 3-8110269602054927661.wt
-rw-r–r– 1 root root 16K Nov 30 05:42 1-8110269602054927661.wt
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] mongod –dbpath /datafiles/ –wiredTigerDirectoryForIndexes –logpath ./mongo.log –port 27017 –fork
about to fork child process, waiting until server is ready for connections.
forked process: 32509
child process started successfully, parent exiting
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] mongo mydb –quiet
MongoDB Enterprise >
MongoDB Enterprise > for(var i = 10001; i <= 20000 ; i++){db.mycol.insert({ _id : i , name : ‘Separate partition for Index files’ });}

WriteResult({ “nInserted” : 1 })

MongoDB Enterprise >
MongoDB Enterprise >
MongoDB Enterprise > db.mycol.count()
20000
MongoDB Enterprise >
MongoDB Enterprise >
MongoDB Enterprise > use newdb
switched to db newdb
MongoDB Enterprise >
MongoDB Enterprise > for(var i = 1; i <= 10000 ; i++){db.newcol.insert({ _id : i , name : ‘Separate partition for Index files’ });}

WriteResult({ “nInserted” : 1 })

MongoDB Enterprise >
MongoDB Enterprise >
MongoDB Enterprise > exit
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] ll -lhtr /dev/shm/indexfiles/index/
total 544K
-rw-r–r– 1 root root 16K Nov 30 05:42 4-8110269602054927661.wt
-rw-r–r– 1 root root 16K Nov 30 05:46 3-8110269602054927661.wt
-rw-r–r– 1 root root 120K Nov 30 05:47 1-4581787003210364632.wt
-rw-r–r– 1 root root 108K Nov 30 05:47 7-8110269602054927661.wt
-rw-r–r– 1 root root 252K Nov 30 05:47 6-8110269602054927661.wt
-rw-r–r– 1 root root 32K Nov 30 05:47 1-8110269602054927661.wt
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ] ll -lhtr /datafiles/index
lrwxrwxrwx 1 root root 26 Nov 30 05:44 /datafiles/index -> /dev/shm/indexfiles/index/
[ root @ dbversity : ~ ]
[ root @ dbversity : ~ ]