Zorgbargle answer is right but there is another consideration when your loading images from resource folder and that's density of the device.
Android scale images base on the device density so you if you only have the image in mdpi folder, you must also divide the points by the density to find the real point on the image:
float[] point = new float[] {event.getX(), event.getY()};
Matrix inverse = new Matrix();
imageView.getImageMatrix().invert(inverse);
inverse.mapPoints(point);
float density = getResources().getDisplayMetrics().density;
point[0] /= density;
point[1] /= density;