I was struggling because of this for a very long time. Answer from adneal didn't get me right colors, most were just white. The trick I did is load icons, convert them to very low resolution bitmaps and then get color that is the most not-greyish (to reduce a chance of picking white/grey/black colors from an icon) out of them.
Here is the code:
fun getBestPrimaryColor(icon:Drawable): Int{
val bitmap: Bitmap = icon.toBitmap(5, 5, Bitmap.Config.RGB_565)
var best_diff_j = 0
var best_diff_k = 0
var best_diff = 0
var current_diff = 0
for(j in 0..4){
for(k in 0..4){
val c = bitmap.getPixel(j,k)
current_diff = max(listOf(c.red,c.green,c.blue)) - min(listOf(c.red,c.green,c.blue))
if(current_diff > best_diff){
best_diff = current_diff
best_diff_j = j
best_diff_k = k
}
}
}
return bitmap.getPixel(best_diff_j, best_diff_k)
}
It turns out like this:

You can change size of the bitmap height and width to get more colors but I don't think it's really necessary.