OAuth2 and Drive API, can't list/retrieve files

Question

I have a web application that needs to list all files from my Google Drive and then fetch them when clicked. I use OAuth for authenticating and it seems to work (the same code works well with Calendar API). I tried different scopes in serviceAccountScopes with no avail.

Basically authentication is:

credential = new GoogleCredential.Builder().
setTransport(HTTP_TRANSPORT).
setJsonFactory(JSON_FACTORY).
setServiceAccountId(apiEmail).
setServiceAccountScopes(DriveScopes.DRIVE).
setServiceAccountPrivateKeyFromP12File(p12File).build();

credential.refreshToken();

service = new Drive.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential).
setApplicationName("My API").build();

edit: I should Add that credential's accessToken is null before refreshToken() call.

After that I try:

FileList files = service.files().list().execute();

The returned FileList is (should contain items):

{"etag":"\"_U9FTLXcHskmKgrWAZqJlfW8kCo/vyGp6PvFo4RvsFtPoIWeCReyIC8\"","items":[],"kind":"drive#fileList","selfLink":"https://www.googleapis.com/drive/v2/files"}

If I check that selfLink the contents is:

{
 "error": {
  "errors": [
   {
    "domain": "usageLimits",
    "reason": "dailyLimitExceededUnreg",
    "message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup.",
    "extendedHelp": "https://code.google.com/apis/console"
   }
  ],
  "code": 403,
  "message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup."
 }
}

Daily limit is not an issue here. (I guess this has nothing to do with this issue, since: https://stackoverflow.com/a/10639679/2090125). Also, I have enabled Drive and Drive SDK in Console (https://stackoverflow.com/a/10329353/2090125).

When downloading a file this is performed:

File file = service.files().get(fileId).execute();

And it produces this (fileId exists):

An error occured: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "File not found: 0B97KF40kTwrTaTllMnZCTV9ZSnM",
    "reason" : "notFound"
  } ],
  "message" : "File not found: 0B97KF40kTwrTaTllMnZCTV9ZSnM"
}

And again when checking https://www.googleapis.com/drive/v2/files/0B97KF40kTwrTaTllMnZCTV9ZSnM the same "dailyLimitExceededUnreg" is seen.

What is going on here, is there a problem in my authentication? Should I configure Drive Integration in Drive SDK somehow? From Googles documentation I have understood that it's not necessary and the methods I'm using should work without further configuring.