My guess is that your application is not designed very well for backward compatibility. I suggest reading through Android's developer training on creating your application with backward compatibility.
https://developer.android.com/training/backward-compatible-ui
If you do not want to do this, you might need to restrict the minSdkVersion in your build.gradle file. I don't know what it is set to now for your app but this might help with not having to support earlier versions. Android provides a table for versions and API levels that will help in deciding a minimum.
https://source.android.com/setup/build-numbers
If you want more detailed help, you will need to supply a lot more context to your question.