mtools
mtools is a collection of helper scripts to parse and filter MongoDB log files (mongod, mongos), visualize log files and quickly set up complex MongoDB test environments on a local machine.
Installation procedure :-
Step 1: Download all the below prerequisites from the link https://pypi.python.org/simple/<package name without version number>
E.g., : https://pypi.python.org/simple/ordereddict/
Package Names & the required versions.
ordereddict==1.1
argparse==1.2.1
python-dateutil==2.2
matplotlib==1.3.1
nose==1.3.0
numpy==1.8.0
pymongo==2.6.3
psutil==1.2.1
[Lab root @ myhostname /tmp]# ll -lhtr *.tar.gz
-rw-r--r-- 1 oracle dba 1.7M Oct 21 07:11 mtools-1.1.6.tar.gz
-rw-r--r-- 1 oracle dba 2.1K Oct 22 02:38 ordereddict-1.1.tar.gz
-rw-r--r-- 1 oracle dba 68K Oct 22 02:38 argparse-1.2.1.tar.gz
-rw-r--r-- 1 oracle dba 254K Oct 22 02:39 python-dateutil-2.2.tar.gz
-rw-r--r-- 1 oracle dba 396K Oct 22 02:39 nose-1.3.0.tar.gz
-rw-r--r-- 1 oracle dba 41M Oct 22 02:40 matplotlib-1.3.1.tar.gz
-rw-r--r-- 1 oracle dba 3.7M Oct 22 02:40 numpy-1.8.0.tar.gz
-rw-r--r-- 1 oracle dba 317K Oct 22 02:40 pymongo-2.6.3.tar.gz
-rw-r--r-- 1 oracle dba 164K Oct 22 02:40 psutil-1.2.1.tar.gz
-rw-r--r-- 1 oracle dba 613K Oct 22 02:48 distribute-0.6.28.tar.gz
-rw-r--r-- 1 oracle dba 27K Oct 22 03:49 six-1.8.0.tar.gz
[Lab root @ myhostname /tmp]
Step 2: Untar & install each package using the below command by going to the respective folder as below.
E.g.,
[Lab root @ 12d4-dl585-04 /tmp]# tar -zxvf python-dateutil-2.2.tar.gz
python-dateutil-2.2/
python-dateutil-2.2/NEWS
python-dateutil-2.2/LICENSE
python-dateutil-2.2/python_dateutil.egg-info/
python-dateutil-2.2/python_dateutil.egg-info/not-zip-safe
python-dateutil-2.2/python_dateutil.egg-info/dependency_links.txt
python-dateutil-2.2/python_dateutil.egg-info/PKG-INFO
python-dateutil-2.2/python_dateutil.egg-info/SOURCES.txt
python-dateutil-2.2/python_dateutil.egg-info/requires.txt
python-dateutil-2.2/python_dateutil.egg-info/top_level.txt
python-dateutil-2.2/README
python-dateutil-2.2/Makefile
python-dateutil-2.2/test.py
python-dateutil-2.2/MANIFEST.in
python-dateutil-2.2/PKG-INFO
python-dateutil-2.2/setup.py
python-dateutil-2.2/example.py
python-dateutil-2.2/dateutil/
python-dateutil-2.2/dateutil/tzwin.py
python-dateutil-2.2/dateutil/rrule.py
python-dateutil-2.2/dateutil/easter.py
python-dateutil-2.2/dateutil/tz.py
python-dateutil-2.2/dateutil/__init__.py
python-dateutil-2.2/dateutil/zoneinfo/
python-dateutil-2.2/dateutil/zoneinfo/zoneinfo--latest.tar.gz
python-dateutil-2.2/dateutil/zoneinfo/__init__.py
python-dateutil-2.2/dateutil/parser.py
python-dateutil-2.2/dateutil/relativedelta.py
python-dateutil-2.2/setup.cfg
python-dateutil-2.2/sandbox/
python-dateutil-2.2/sandbox/scheduler.py
python-dateutil-2.2/sandbox/rrulewrapper.py
[Lab root @ 12d4-dl585-04 /tmp]#
[Lab root @ 12d4-dl585-04 /tmp]# cd python-dateutil-2.2
[Lab root @ 12d4-dl585-04 /tmp/python-dateutil-2.2]
[Lab root @ myhostname /tmp/mtools-master/python-dateutil-2.2]# python setup.py install
running install
running bdist_egg
running egg_info
writing requirements to python_dateutil.egg-info/requires.txt
writing python_dateutil.egg-info/PKG-INFO
writing top-level names to python_dateutil.egg-info/top_level.txt
writing dependency_links to python_dateutil.egg-info/dependency_links.txt
reading manifest file 'python_dateutil.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'python_dateutil.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
----- blah blah blah -----
[Lab root @ myhostname /tmp/mtools-master/python-dateutil-2.2]#
Testing the mtool :-
The following tools are in the mtools collection:
mlogfilter :-
slices log files by time, merges log files, filters slow queries, finds table scans, shortens log lines, filters by other atributes, convert to JSON
A script to reduce the amount of information from MongoDB log files. mlogfilter takes one or several MongoDB log files as input, together with some filter parameters, parses the contained log lines and outputs the lines that match according to the filter parameters.
If several log files are provided, mlogfilter will merge them by timestamp and insert a marker at the beginning of each line, before applying any of the other filters.
More details at : https://github.com/rueckstiess/mtools/wiki/mlogfilter
[Lab root @ myhostname /opt/mongodb/logs]# ll -lhtr
total 1.9G
-rw-r--r-- 1 root root 96K Oct 17 06:38 shard1_1.log
-rw-r--r-- 1 root root 18M Oct 22 05:31 router.log
-rw-r--r-- 1 root root 257M Oct 22 05:31 config3.log
-rw-r--r-- 1 root root 257M Oct 22 05:31 config2.log
-rw-r--r-- 1 root root 326M Oct 22 05:31 config1.log
-rw-r--r-- 1 root root 142M Oct 22 05:31 arbiter2.log
-rw-r--r-- 1 root root 159M Oct 22 05:31 shard2_2.log
-rw-r--r-- 1 root root 221M Oct 22 05:31 shard2_1.log
-rw-r--r-- 1 root root 382M Oct 22 05:31 shard1_2.log
-rw-r--r-- 1 root root 142M Oct 22 05:31 arbiter1.log
[Lab root @ myhostname /opt/mongodb/logs]#
We have identified the queries which running more than 5000 milli seconds with the below command.
[Lab root @ myhostname /opt/mongodb/logs]# mlogfilter --human --json --slow 5000 shard1_1.log | head -3 {"split_tokens": ["Fri", "Oct", "17", "06:36:03.416", "[conn9]", "getmore", "local.oplog.rs", "query:", "{", "ts:", "{", "$gte:", "Timestamp", "1413542141000|2", "}", "}", "cursorid:285762317417636", "ntoreturn:0", "keyUpdates:0", "locks(micros)", "r:296", "nreturned:0", "reslen:20", "5011ms"], "thread": "conn9", "namespace": "local.oplog.rs", "nreturned": 0, "datetime": "2014-10-17T06:36:03.416000+00:00", "r": 296, "ntoreturn": 0, "duration": 5011, "line_str": "Fri Oct 17 06:36:03.416 [conn9] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1413542141000|2 } } cursorid:285762317417636 ntoreturn:0 keyUpdates:0 locks(micros) r:296 nreturned:0 reslen:20 5011ms", "operation": "getmore"} {"split_tokens": ["Fri", "Oct", "17", "06:36:03.449", "[conn10]", "getmore", "local.oplog.rs", "query:", "{", "ts:", "{", "$gte:", "Timestamp", "1413542141000|2", "}", "}", "cursorid:285870062378187", "ntoreturn:0", "keyUpdates:0", "locks(micros)", "r:266", "nreturned:0", "reslen:20", "5017ms"], "thread": "conn10", "namespace": "local.oplog.rs", "nreturned": 0, "datetime": "2014-10-17T06:36:03.449000+00:00", "r": 266, "ntoreturn": 0, "duration": 5017, "line_str": "Fri Oct 17 06:36:03.449 [conn10] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1413542141000|2 } } cursorid:285870062378187 ntoreturn:0 keyUpdates:0 locks(micros) r:266 nreturned:0 reslen:20 5017ms", "operation": "getmore"} {"split_tokens": ["Fri", "Oct", "17", "06:36:08.427", "[conn9]", "getmore", "local.oplog.rs", "query:", "{", "ts:", "{", "$gte:", "Timestamp", "1413542141000|2", "}", "}", "cursorid:285762317417636", "ntoreturn:0", "keyUpdates:0", "locks(micros)", "r:173", "nreturned:0", "reslen:20", "5011ms"], "thread": "conn9", "namespace": "local.oplog.rs", "nreturned": 0, "datetime": "2014-10-17T06:36:08.427000+00:00", "r": 173, "ntoreturn": 0, "duration": 5011, "line_str": "Fri Oct 17 06:36:08.427 [conn9] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1413542141000|2 } } cursorid:285762317417636 ntoreturn:0 keyUpdates:0 locks(micros) r:173 nreturned:0 reslen:20 5011ms", "operation": "getmore"} [Lab root @ myhostname /opt/mongodb/logs Mloginfo:- Returns info about log file, like start and end time, version, binary, special sections like restarts, connections, distinct view https://github.com/rueckstiess/mtools/wiki/mloginfo [Lab root @ myhostname /opt/mongodb/logs]# mloginfo shard1_1.log source: shard1_1.log host: myhostname:27010 start: 2014 Oct 17 06:34:50.767 end: 2014 Oct 17 06:38:02.394 date format: ctime length: 751 binary: mongod version: 2.4.11 [Lab root @ myhostname /opt/mongodb/logs] mplotqueries :- visualize logfiles with different types of plots (requires matplotlib) https://github.com/rueckstiess/mtools/wiki/mplotqueries (OR) mlogvis creates a self-contained html file that shows an interactive visualization in a web browser (as an alternative to mplotqueries) mlogvis is a script to visualize logfiles interactively in a browser, using the d3.js javascript visualization engine. mlogvis provides an alternative to mplotqueries without the matplotlib dependency, but currently only contains a sub-set of features of mplotqueries. The script will read a logfile, process the data and write a self-contained html file to the current working directory. It will then start open a browser tab to display the file. The html file can also be sent to somebody and openend by any modern browser (optimized for Google Chrome). An internet connection is required for dynamic loading of d3 javascript library. https://github.com/rueckstiess/mtools/wiki/mlogvis [Lab root @ myhostname /opt/mongodb/logs]# mlogvis shard1_1.log copying /usr/lib/python2.6/site-packages/mtools-1.1.6-py2.6.egg/mtools/data/index.html to /opt/mongodb/logs/shard1_1.log.html serving visualization on file:///opt/mongodb/logs/shard1_1.log.html [Lab root @ myhostname /opt/mongodb/logs]# [Lab root @ myhostname /opt/mongodb/logs]# [Lab root @ myhostname /opt/mongodb/logs]# ll -lhtr total 1.9G -rw-r--r-- 1 root root 96K Oct 17 06:38 shard1_1.log -rw-r--r-- 1 root root 124K Oct 22 05:56 shard1_1.log.html -rw-r--r-- 1 root root 18M Oct 22 05:56 router.log -rw-r--r-- 1 root root 258M Oct 22 05:56 config2.log -rw-r--r-- 1 root root 327M Oct 22 05:56 config1.log -rw-r--r-- 1 root root 258M Oct 22 05:56 config3.log -rw-r--r-- 1 root root 221M Oct 22 05:56 shard2_1.log -rw-r--r-- 1 root root 160M Oct 22 05:56 shard2_2.log -rw-r--r-- 1 root root 143M Oct 22 05:56 arbiter1.log -rw-r--r-- 1 root root 384M Oct 22 05:56 shard1_2.log -rw-r--r-- 1 root root 142M Oct 22 05:56 arbiter2.log [Lab root @ myhostname /opt/mongodb/logs]# cp shard1_1.log.html /tmp/ [Lab root @ myhostname /opt/mongodb/logs] mlaunch a script to quickly spin up local test environments, including replica sets and sharded systems (requires pymongo) This tool lets you quickly spin up and monitor MongoDB environments on your local machine. It supports various configurations of stand-alone servers, replica sets and sharded clusters. Individual nodes or groups of nodes can easily be stopped and started again. In addition to all the listed parameters of mlaunch below, you can pass in any arbitrary options that amongos or mongod binary would understand, and mlaunch will pass them on to the correct binary. https://github.com/rueckstiess/mtools/wiki/mlaunch [Lab root @ myhostname /opt/mongodb/logs]# ps -ef | grep mongo root 18984 1 0 Oct17 ? 00:43:21 ./mongod --configsvr --dbpath /opt/mongodb/data/config1 --logpath /opt/mongodb/logs/config1.log --port 39000 --config /etc/mongod.conf root 18987 1 0 Oct17 ? 00:41:05 ./mongod --configsvr --dbpath /opt/mongodb/data/config2 --logpath /opt/mongodb/logs/config2.log --port 39001 --config /etc/mongod.conf root 18989 1 0 Oct17 ? 00:35:15 ./mongod --replSet rs2 --dbpath /opt/mongodb/data/arbiter2 --logpath /opt/mongodb/logs/arbiter2.log --port 27022 --config /etc/mongod.conf root 18995 1 0 Oct17 ? 00:38:17 ./mongod --shardsvr --replSet rs2 --dbpath /opt/mongodb/data/shard2_1 --logpath /opt/mongodb/logs/shard2_1.log --port 27020 --config /etc/mongod.conf root 18997 1 0 Oct17 ? 00:40:54 ./mongod --configsvr --dbpath /opt/mongodb/data/config3 --logpath /opt/mongodb/logs/config3.log --port 39002 --config /etc/mongod.conf root 19000 1 0 Oct17 ? 00:36:20 ./mongod --shardsvr --replSet rs2 --dbpath /opt/mongodb/data/shard2_2 --logpath /opt/mongodb/logs/shard2_2.log --port 27021 --config /etc/mongod.conf root 19311 1 0 Oct17 ? 00:32:55 ./mongos --configdb xx.xx.xx.xx:39000,xx.xx.xx.xx:39001,xx.xx.xx.xx:39002 --logpath /opt/mongodb/logs/router.log --port 10000 root 21872 31501 0 06:33 pts/1 00:00:00 grep mongo [Lab root @ myhostname /opt/mongodb/logs]# [Lab root @ myhostname /opt/mongodb/logs]# Trying to initiate the replica set rs1 with the one of the node port 27010 [Lab root @ myhostname /opt/mongodb/logs]# [Lab root @ myhostname /opt/mongodb/logs]# [Lab root @ myhostname /opt/mongodb/logs]# mlaunch init --replicaset --port 27010 --hostname myhostname --dir /opt/mongodb/data/shard1_1 --binarypath /opt/mongodb/bin/ launching: /opt/mongodb/bin/mongod on port 27010 launching: /opt/mongodb/bin/mongod on port 27011 launching: /opt/mongodb/bin/mongod on port 27012 replica set 'replset' initialized. [Lab root @ myhostname /opt/mongodb/logs]# Starting the nodes [Lab root @ myhostname /opt/mongodb/logs]# [Lab root @ myhostname /opt/mongodb/logs]# mlaunch start --replicaset --port 27010 --hostname myhostname --dir /opt/mongodb/data/shard1_1 --binarypath /opt/mongodb/bin/ no nodes started. [Lab root @ myhostname /opt/mongodb/logs]# To check whether all the 3 nodes started [Lab root @ myhostname /opt/mongodb/logs]# ps -ef | grep mongo | egrep '27010|27011|27012' root 21957 1 0 06:34 ? 00:00:00 /opt/mongodb/bin/mongod --replSet replset --dbpath /opt/mongodb/data/shard1_1/replset/rs1/db --logpath /opt/mongodb/data/shard1_1/replset/rs1/mongod.log --port 27010 --logappend --fork root 22014 1 0 06:34 ? 00:00:00 /opt/mongodb/bin/mongod --replSet replset --dbpath /opt/mongodb/data/shard1_1/replset/rs2/db --logpath /opt/mongodb/data/shard1_1/replset/rs2/mongod.log --port 27011 --logappend --fork root 22060 1 0 06:34 ? 00:00:00 /opt/mongodb/bin/mongod --replSet replset --dbpath /opt/mongodb/data/shard1_1/replset/rs3/db --logpath /opt/mongodb/data/shard1_1/replset/rs3/mongod.log --port 27012 --logappend --fork [Lab root @ myhostname /opt/mongodb/logs] mgenerate generates structured pseudo-random data based on a template for testing and reproduction This is a spec to add a new script mgenerate to the mtools family. It will generate structured, semi-random data according to a template document. The template can be specified directly as a command line argument or it can be a file. The format for the template argument is in JSON. Additional arguments to mgenerate specify how many documents should be inserted. The generated documents are directly inserted into a mongod/s, as specified with--host and --port. The default host is localhost and the default port is 27017. https://github.com/rueckstiess/mtools/wiki/mgenerate Creating a temporary JSON file [Lab root @ myhostname /opt/mongodb/logs]# vi test.json { "glossary": { "title": "example glossary", "GlossDiv": { "title": "S", "GlossList": { "GlossEntry": { "ID": "SGML", "SortAs": "SGML", "GlossTerm": "Standard Generalized Markup Language", "Acronym": "SGML", "Abbrev": "ISO 8879:1986", "GlossDef": { "para": "A meta-markup language, used to create markup languages such as DocBook.", "GlossSeeAlso": ["GML", "XML"] }, "GlossSee": "markup" } } } } } ~ Loading the above JSON file into the test db 100 times using mgenerate [Lab root @ myhostname /opt/mongodb/logs]# mgenerate test.json --num 100 --port 27010 [Lab root @ myhostname /opt/mongodb/bin]# ./mongo --port 27010 MongoDB shell version: 2.4.11 connecting to: 127.0.0.1:27010/test replset:PRIMARY> replset:PRIMARY> replset:PRIMARY> show dbs local 1.078125GB test 0.203125GB replset:PRIMARY> use test switched to db test replset:PRIMARY> show collections mgendata system.indexes replset:PRIMARY> replset:PRIMARY> replset:PRIMARY> db.mgendata.count() 100 replset:PRIMARY> replset:PRIMARY> replset:PRIMARY> db.mgendata.findOne() { "_id" : ObjectId("54478a7f5f513060b6e2ca96"), "glossary" : { "title" : "example glossary", "GlossDiv" : { "GlossList" : { "GlossEntry" : { "GlossDef" : { "GlossSeeAlso" : [ "GML", "XML" ], "para" : "A meta-markup language, used to create markup languages such as DocBook." }, "GlossSee" : "markup", "Acronym" : "SGML", "GlossTerm" : "Standard Generalized Markup Language", "SortAs" : "SGML", "Abbrev" : "ISO 8879:1986", "ID" : "SGML" } }, "title" : "S" } } } replset:PRIMARY> replset:PRIMARY> Source links : https://github.com/rueckstiess/mtools http://rpm.pbone.net/index.php3/stat/4/idpl/24595866/dir/redhat_el_6/com/python-pip-1.3.1-4.el6.noarch.rpm.html https://pypi.python.org/pypi/mtools http://www.mongodb.com/presentations/diagnostics-and-debugging https://github.com/rueckstiess/mtools/blob/master/INSTALL.md https://github.com/rueckstiess/mtools/wiki/mplotqueries
Ask Question
