MongoDB

How to test if MongoDB database exists on command line (bash)

Use this command to test if a given MongoDB database exists:

mongo --quiet --eval 'db.getMongo().getDBNames().indexOf("mydb")'

This will return an index such as 0 or 241 if the database is found. On the other hand, it will return -1 if the database does not exist.

docker-compose version:

docker-compose exec mongodb mongo --quiet --eval 'db.getMongo().getDBNames().indexOf("mydb")'

where mongodb is the name of your container.

Now we can put it together in a bash script to test if the database exists:

# Query if DB exists in MongoDB
mongo_indexof_db=$(mongo --quiet --eval 'db.getMongo().getDBNames().indexOf("mydb")')
if [ $mongo_indexof_db -ne "-1" ]; then
    echo "MongoDB database exists"
else
    echo "MongoDB database does not exist"
fi

 

docker-compose variant:

# Query if DB exists in MongoDB
mongo_indexof_db=$(docker-compose -f inspect.yml exec -T mongodb mongo --quiet --eval 'db.getMongo().getDBNames().indexOf("mydb")')
if [ $mongo_indexof_db -ne "-1" ]; then
    echo "MongoDB database exists"
else
    echo "MongoDB database does not exist"
fi

 

Posted by Uli Köhler in MongoDB, Shell

How to fix Python MongoDB TypeError: Object of type ObjectId is not JSON serializable

Problem:

When trying to export data as JSON that has originally been queried from MongoDB using code like

with open("alle.json", "w") as outfile:
    json.dump(alle, outfile)

you see the following error message:

File /usr/lib/python3.9/json/__init__.py:179, in dump(obj, fp, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    173     iterable = cls(skipkeys=skipkeys, ensure_ascii=ensure_ascii,
    174         check_circular=check_circular, allow_nan=allow_nan, indent=indent,
    175         separators=separators,
    176         default=default, sort_keys=sort_keys, **kw).iterencode(obj)
    177 # could accelerate with writelines in some versions of Python, at
    178 # a debuggability cost
--> 179 for chunk in iterable:
    180     fp.write(chunk)

File /usr/lib/python3.9/json/encoder.py:429, in _make_iterencode.<locals>._iterencode(o, _current_indent_level)
    427     yield _floatstr(o)
    428 elif isinstance(o, (list, tuple)):
--> 429     yield from _iterencode_list(o, _current_indent_level)
    430 elif isinstance(o, dict):
    431     yield from _iterencode_dict(o, _current_indent_level)

File /usr/lib/python3.9/json/encoder.py:325, in _make_iterencode.<locals>._iterencode_list(lst, _current_indent_level)
    323         else:
    324             chunks = _iterencode(value, _current_indent_level)
--> 325         yield from chunks
    326 if newline_indent is not None:
    327     _current_indent_level -= 1

File /usr/lib/python3.9/json/encoder.py:405, in _make_iterencode.<locals>._iterencode_dict(dct, _current_indent_level)
    403         else:
    404             chunks = _iterencode(value, _current_indent_level)
--> 405         yield from chunks
    406 if newline_indent is not None:
    407     _current_indent_level -= 1

File /usr/lib/python3.9/json/encoder.py:438, in _make_iterencode.<locals>._iterencode(o, _current_indent_level)
    436         raise ValueError("Circular reference detected")
    437     markers[markerid] = o
--> 438 o = _default(o)
    439 yield from _iterencode(o, _current_indent_level)
    440 if markers is not None:

File /usr/lib/python3.9/json/encoder.py:179, in JSONEncoder.default(self, o)
    160 def default(self, o):
    161     """Implement this method in a subclass such that it returns
    162     a serializable object for ``o``, or calls the base implementation
    163     (to raise a ``TypeError``).
   (...)
    177 
    178     """
--> 179     raise TypeError(f'Object of type {o.__class__.__name__} '
    180                     f'is not JSON serializable')

TypeError: Object of type ObjectId is not JSON serializable

Solution:

This error occurs because objects queried from PyMongo always contain _id which is of type ObjectId and the normal JSON library (or drop-in replacements like simplejson do not know how to create JSON representations of Objects of type ObjectId).

In order to fix this, use pymongo‘s json_util instead of json. Note that the bson.json_util package contains dumps but does not contain dump, so use the following snippet to write to a file:

 

import bson.json_util as json_util

with open("alle.json", "w") as outfile:
    outfile.write(json_util.dumps(alle))

 

Posted by Uli Köhler in MongoDB, Python

How iterate all documents in MongoDB collection using pymongo

This example will connect to the MongoDB running at localhost (on the default port 27017) without any username or password and open the database named mydb (also see Python MongoDB minimal connect example using pymongo), open the collection mycollectionand iterate all the documents in said collection, printing each document.

from pymongo import MongoClient
client = MongoClient("mongodb://localhost")
db = client["mydb"]
mycollection = db["mycollection"]

for doc in mycollection.find():
    print(doc)

This will print, for example,

{'_id': 123, 'name': 'John', 'phone': '+123456789'}

 

Posted by Uli Köhler in Databases, MongoDB

How to list MongoDB collection names in Python using pymongo

This example will connect to the MongoDB running at localhost (on the default port 27017) without any username or password and open the database named mydb (also see Python MongoDB minimal connect example using pymongo) and list all the collection names in mydb:

from pymongo import MongoClient
client = MongoClient("mongodb://localhost")
db = client["mydb"]

print(db.list_collection_names())

This will print, for example,

['people', 'salaries']

 

Posted by Uli Köhler in Databases, MongoDB

Python MongoDB minimal connect example using pymongo

This example will connect to the MongoDB running at localhost (on the default port 27017) without any username or password and open the database named mydb

from pymongo import MongoClient
client = MongoClient("mongodb://localhost")
db = client["mydb"]

 

Posted by Uli Köhler in Databases, MongoDB

pre and post script for veeam MongoDB backups

Create an pre.sh and post.sh script to be run before and after a veeam backup of your ubuntu machine runs and will shutdown MongoDB and your webservice (myservice) using the MongoDB before the backup and start it again after the backup has finished. For example:

systemctl stop myservice
systemctl stop mongod
exit 0

and

systemctl start mongod
systemctl start myservice
exit 0

I highly recommend proper error handling – here handover error codes from systemctl commands.

Select the path to the shell scripts like this:
veeam job setup enable scripts

Posted by Joshua Simon in Linux, MongoDB, veeam

Best-practice configuration for MongoDB with docker-compose

Create /var/lib/mongodb/docker-compose.yml:

version: '3.1'
services:
  mongo:
    image: mongo
    volumes:
        - ./mongodb_data:/data/db
    ports:
        - 27017:27017

This will store the MongoDB data in /var/lib/mongodb/data. I prefer this variant to using docker volumes since this method keeps all MongoDB-related data in the same directory.

Then create a systemd service using

curl -fsSL https://techoverflow.net/scripts/create-docker-compose-service.sh | sudo bash /dev/stdin

See our post on how to Create a systemd service for your docker-compose project in 10 seconds for more details on this method.

You can access MongoDB at localhost:27017! It will autostart after boot

Restart by

sudo systemctl restart mongodb

Stop by

sudo systemctl stop mongodb

View logs:

sudo journalctl -xfu mongodb

View logs in less:

sudo journalctl -xu mongodb

 

Posted by Uli Köhler in Docker, MongoDB

How to list MongoDB databases on command line

Use this command to list the MongoDB databases on the command line:

echo 'show dbs' | mongo

Example output:

MongoDB shell version v4.0.13
connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("5c1e505e-9b05-4270-ab20-c537c0760481") }
MongoDB server version: 4.0.13
admin       0.000GB
config      0.000GB
drawing     0.001GB
order       0.000GB
production  0.001GB
standards   0.001GB
user        0.000GB
bye

 

Posted by Uli Köhler in MongoDB

How to fix MongoDB ‘exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /var/lib/mongodb, terminating’

Problem:

When trying to start MongoDB e.g. using sudo systemctl start mongod, the process terminates with status code 100 and you see this error message in /var/log/mongodb/mongodb.log:

2019-07-22T17:11:07.858+0200 I STORAGE  [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /var/lib/mongodb, terminating

Solution:

Fix the permissions of /var/lib/mongodb:

sudo chown -R mongodb: /var/lib/mongodb

then restart MongoDB e.g. using

sudo systemctl restart mongod

In case that does not help, check your error message if you are using a data directory different to /var/lib/mongodb. In that case run

sudo chown -R mongodb: <insert your data directory here>

In case that is not the case or doesn’t help either, check if your filesystem is mounted in read-only mode.

Posted by Uli Köhler in MongoDB