The Google Dialer is not part of Kitkat. It's not even in the ASOP code base. I guess Google is keeping a tight lid and what exactly they're doing to update the call screen with caller information for now. That said, the Google Dialer is on the Nexus devices and Google Play Experience devices running Android Kitkat.
There are ports to other KitKat devices:
http://www.androidpolice.com/2013/12/18/googles-dialer-with-built-in-search-is-exclusive-to-nexus-and-gpe-devices-but-you-can-install-it-manually-on-other-kitkat-roms-download/
To answer your questions:
Is there any API for this new feature?
The is no documented API as the Google Dialer is still closed source.
If there is an API, does it work for all Android versions, or just
Kitkat?
Just KitKat and intended for Nexus and Google Play Experience devices, but can be ported to other KitKiat devices.
If there is no API, what exactly is shown to the user? where is it
taken from?
http://www.androidpolice.com/2013/11/04/google-does-caller-id-kitkat-4-4-matches-numbers-against-google-places-and-apps-for-business-and-starting-in-2014-google-personal-accounts/
Basically they use your personal information from your Google Account and Google Maps/Places.