So, do I need to stop the mongod process before start rsync?
To take a file copy backup with rsync or cp you need to quiesce writes by stopping the mongod service or issuing the db.fsyncLock() command in the mongo shell or via a driver. Since the secondary you are backing up from will be intermittently available with this approach, it would be best to make it hidden to avoid drivers trying to read from this member when it is down or fsyncLock'd.
I would make sure you test several full backup and restore cycles. In particular, be wary of rsync options that might invalidate backups (for example: --ignore-existing, --size-only, --partial).
Note: if you are using the fsyncLock() approach there is a db.fsyncUnLock() command to restore normal operation after your backup completes. Another caveat: the WiredTiger storage engine only supports fsyncLock in MongoDB 3.2 or newer.
can I start rsync (dbPath: /var/lib/mongo) without stopping the mongod process.
No. This will result in an inconsistent backup.
what is the recommended backup method other than LVM backup and mongodump?
Supported backup approaches are described in the documentation: MongoDB Backup Methods.
There are less disruptive (and more continuous) backup approaches such as using an agent-based backup service (for example, MongoDB Cloud Manager).
If you want to backup by copying underlying data files, filesystem snapshots (LVM/EBS/...) are generally less disruptive and more recommendable than pausing all writes to a mongod. Filesystem snapshots against a running mongod do have some requirements if you want to capture a consistent backup: all data must be on the same volume, journaling must be enabled, and the filesystem must support point-in-time snapshots. For full details, check out the official documentation and backup tutorials.
