Libsodium is a well regarded library, AES (Advanced Encryption Standard ) is a well secure encryption algorithm, they are not the same kind of thing.
Libsodium provides AES in GCM mode which is quite good as it includes authentication.
The problem with Libsodium is that is has limited algorithms available so interoperability is low.
Safety is largely a matter of how secure your implementation is, any bugs or misuse in encryption break the security.
Define who you are protecting from, ranging from an inquisitive teen to a well funded government and design the security to meet the level you need.
Managing keys is a very difficult problem. On an phone were is usually some form of key repository and that is probably the best you can do. But that relies on the user having a good passcode.
If you want a very secure system pay to have your scheme and code reviewed by a cryptographic domain expert.
Update:
The only secure key management is not to have the key on the device; per my SME (Subject Matter Expert). Require the user enter the passphrase on each invocation of the app and make sure the app is closed after each use. Then you will need to add code to rate limit access attempts with possible exponential back-off delays. Finally the user will have to have a really good passphrase.