I found the solution for the problem I was facing.
"An asset pack is eligible for purging when all of the associated tags are no longer retained by any request. The resources associated with a tag may remain on the device for some time before it is purged, including across app launches."
Ultimately, the cache will get automatically purged if there is a memory issue, so it is only a cosmetic issue where users will potentially see your app as being larger than they expected, and then at some point down the line, it will be magically smaller.
So I copied my resources into another directory locally.
Another idea is I zipped my images and added zip file as on-demand resources. So after ODR got installed I extracted and copied my zip files contents locally. (I did this because finding the path for images in asset file was difficult in my case)