It contains some CPU-specific highly-optimized code that is used to improve the performance of some operations, like decoding of certain types of multimedia streams (ffmpeg uses it and I'm not sure if something else does too).
The library is actually a 7zipped pack of several libraries, built for different ARM architectures (ARMv6, ARMv7, etc.), thus its size.
These libraries are extracted to your device the first time you run your app. If you remove them and run your app successfully, it is most likely because they have already been extracted. Clear you app data, and try it again.
If the app doesn't crash when you delete it (and have cleared your app data), it means that you're just not using code that utilizes libarm.so APIs, so, unless you plan to add more multimedia support to your app, it is safe to leave the library out of it.