Dump MongoDB daily and upload on S3 with AWS CLI


Dump MongoDB daily and upload on S3 with AWS CLI

Install AWS CLI on Ubuntu

$ sudo apt-get update
$ sudo apt-get install awscli

AWS Credentials

To configure AWS Credentials

$ aws configure

Then you can check ~/.aws/credentials and ~/.aws/config to see the content.

Credentials file

$ cat ~/.aws/credentials

[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

Config file

$ cat ~/.aws/config

[default]
output = json
region = eu-central-1

Backup files

First, we need to create a folder and within add two files

Got to the root

$ cd

Create a new folders bin and logs

$ mkdir bin logs

Add a new file to backup the database to S3

$ vi bin/db-backups.sh

And add the content

backup_name=~/db_backups-`date +%Y-%m-%d-%H%M`
mongodump --host localhost --port 27017 --authenticationDatabase admin -u ADMIN_USER -p YOUR_PASSWORD --out $backup_name
tar czf $backup_name.tar.gz $backup_name
aws s3 cp $backup_name.tar.gz s3://YOUR_PATH_HERE
rm -rf $backup_name
rm $backup_name.tar.gz

Export multiple databases with mongodump

But what happens when you need to export multiple databases. The script above it works without any issue, but the real deal goes out when you need to exclude collections with multiple export.

Here is the script

DB_BACKUP_PATH=~/db_backups-`date +%Y-%m-%d-%H%M`

MONGO_HOST=localhost
MONGO_PORT=27017
MONGO_USER=<ADMIN_USER>
MONGO_PASSWORD=<YOUR_PASSWORD>

MONGO_AUTH_STRING="--host $MONGO_HOST --port $MONGO_PORT --authenticationDatabase admin -u $MONGO_USER -p $MONGO_PASSWORD"

dbs=`mongo $MONGO_AUTH_STRING --quiet --eval "db.getMongo().getDBNames()" | tr -d '"' | tr -d '",' | tr -d "[" | tr -d "]"`
for db in $dbs; do
    echo "Dumping database: $db"

    db_export_string="--db=${db}"
    if [ $db == "DATABASE_NAME" ]; then # add the name of your database
        db_export_string+=" --excludeCollection=COLLECTION_NAME" # add the collection name that you want to be excluded
    fi
    mongodump $MONGO_AUTH_STRING $db_export_string --out $DB_BACKUP_PATH
done

tar czf $DB_BACKUP_PATH.tar.gz $DB_BACKUP_PATH
aws s3 cp $backup_name.tar.gz s3://YOUR_PATH_HERE
rm -rf $DB_BACKUP_PATH
rm $DB_BACKUP_PATH.tar.gz

The second file needs to upload the logs to S3

$ vi bin/log-backup.sh 

Add the content

root_folder=/home/ubuntu
backup_name=~/log_backups-`date +%Y-%m-%d-%H%M`
tar czf $backup_name.tar.gz $root_folder/logs
aws s3 cp $backup_name.tar.gz s3://YOUR_PATH_HERE
for f in $root_folder/logs/*.log; do :> $f; done
rm $backup_name.tar.gz

The last step add the following script in crontab

$ crontab -e

At the end of the file add

0 0 * * * /bin/bash ~/bin/db-backups.sh >> ~/logs/db_backups.log 2>&1
0 0 * * * /bin/bash ~/bin/log-backup.sh >> ~/logs/log_backup.log 2>&1

Newsletter


Related Posts

A Week in the Life of an Invoice Wrangler: Navigating Ridesharing and Food Delivery Chaos

As an app founder in the ridesharing and food delivery industry, I found myself knee-deep in invoice reports from companies like Bolt, Uber, Glovo, and Bolt Food

Free HTML templates list for Startups

Free HTML templates list for startup. A complete list with free resources to build your next startup's website and gain the traction to the sky.

Deal with client requests in SaaS

How to deal with client requests in Saas which are seeing only their interests, not the product interest.

The first client of LoyalXpert is not answering anymore

Trying to implement LoyalXpert app, I lost my first customer, he's not answering anymore

Experiments with Tiktok Ads

Recently tried out TikTok ads for the first time and here are some of my learnings and challenges

People don’t care about you, until they know you care about them.

People don’t care about you, until they know you care about them. The same happens in business, you need to take care of your clients.

The One Word That Can Ruin Your SaaS Business Anyone

As a SaaS founder, you probably know how important it is to have a clear and specific target market for your product.

How I got my digital certificate connected it with ANAF

How I got my digital certificate from certSIGN and connected it with ANAF

The Ultimate List of Company Directories to Boost Your Networking

Discover a wide range of company directories to boost your business's visibility and connect with potential clients.

Discover the Best Free AI Art Tools for Your Next Masterpiece

Explore a curated collection of the finest free AI art tools, designed to help you bring your artistic vision to life.