google drive client library invalid grant error on file upload

Question

I always get this error when trying to add files under a folder in google drive, the folder is marked shared with mode "anyone in abc.com can view and edit"

com.google.api.client.auth.oauth2.TokenResponseException: 400 Bad Request
{
  "error" : "invalid_grant"
}

I followed this procedure from Google Developers Guide

Delegating domain-wide authority to the service account

If your application accesses user data, the service account that you created needs to be granted access to the Google Apps domain’s user data that you want to access.

The following steps must be performed by an administrator of the Google Apps domain:

Go to your Google Apps domain’s Admin console.
Select Security from the list of controls. If you don't see Security listed, select More controls from the gray bar at the bottom of the page, then select Security from the list of controls. If you can't see the controls, make sure you're signed in as an administrator for the domain.
Select Show more and then Advanced settings from the list of options.
Select Manage API client access in the Authentication section.
In the Client Name field enter the service account's Client ID.
In the One or More API Scopes field enter the list of scopes that your application should be granted access to. For example, if your application needs domain-wide access to the Google Drive API and the Google Calendar API, enter: https://www.googleapis.com/auth/drive, https://www.googleapis.com/auth/calendar.
Click Authorize.
Your application now has the authority to make API calls as users in your domain (to "impersonate" users). When you prepare to make authorized API calls, you specify the user to impersonate.

added the keys in application.properties

#Google Drive
google.drive.service.account=AAAAAAAAAAAA-i8jfnhhug5uoo9rm8ek2sf06b9452vb9@developer.gserviceaccount.com
google.drive.impersonate.account=my.user@abc.com
google.drive.pkey.file=abcdefgh-0aafb873fcc9.p12
google.drive.application.name=myapp-web

And then created a configuration class for drive

@Configuration
public class GoogleDriveAPIConfiguration {

  @Resource
  private Environment environment;

  private final org.springframework.core.io.Resource pkeyFile = new ClassPathResource("abcdefgh-0aafb873fcc9.p12");

  private static final String GOOGLE_RIVE_SERVICE_ACCOUNT = "google.drive.service.account";

  private static final String GOOGLE_DRIVE_APPLICATION_NAME = "google.drive.application.name";

  private static final String GOOGLE_DRIVE_IMPERSONATE_ACCOUNT = "google.drive.impersonate.account";

  @Bean
  @Autowired
  public GoogleCredential googleCredential(JsonFactory jsonFactory, HttpTransport httpTransport)
      throws GeneralSecurityException, IOException {

    GoogleCredential credential = new GoogleCredential.Builder().setTransport(httpTransport)
        .setJsonFactory(jsonFactory).setServiceAccountId(environment.getProperty(GOOGLE_RIVE_SERVICE_ACCOUNT))
        .setServiceAccountPrivateKeyFromP12File(getPrivateKeyFile())
        .setServiceAccountScopes(Collections.singleton(DriveScopes.DRIVE))
        .setServiceAccountId(environment.getProperty(GOOGLE_DRIVE_IMPERSONATE_ACCOUNT))
        .build();

    return credential;
  }

  @Bean
  public JsonFactory getJsonFactory() {
    JsonFactory JSON_FACTORY = JacksonFactory.getDefaultInstance();
    return JSON_FACTORY;
  }

  @Bean
  public HttpTransport getHttpTransport() throws GeneralSecurityException, IOException {
    HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
    return httpTransport;
  }

  @Bean
  @Autowired
  public Drive getDrive(HttpTransport httpTransport, JsonFactory JSON_FACTORY, GoogleCredential credential) {
    Drive drive = new Drive.Builder(httpTransport, JSON_FACTORY, credential).setApplicationName(
        GOOGLE_DRIVE_APPLICATION_NAME).build();
    return drive;
  }

  private File getPrivateKeyFile() throws IOException {
    return pkeyFile.getFile();
  }
}

The app starts up without any error, here's file upload code (full file in this gist)

  @Autowired
  public void setDrive(Drive drive) {
    this.drive = drive;
  }

  @Async
  public void uploadResume(UploadData data, byte[] uploadFile) throws GeneralSecurityException, IOException {

    LOGGER.info("Uploading file type {} of size {} to Drive",data.getContentType(),uploadFile.length);
    com.google.api.services.drive.model.File file = new com.google.api.services.drive.model.File();

    List<ParentReference> referneces = new ArrayList<ParentReference>();
    referneces.add(buildParentReference(data.getApplication().getRole()));

    file.setTitle(data.getFileName());
    file.setParents(referneces);
    file.setCopyable(true);
    file.setEditable(true);
    file.setWritersCanShare(true);
    file.setTitle(data.getFileName());

    ByteArrayInputStream inputBytes = new ByteArrayInputStream(uploadFile);

    InputStreamContent fileUpload = new InputStreamContent(data.getContentType(), new BufferedInputStream(inputBytes));
    fileUpload.setLength(uploadFile.length);

    Drive.Files.Insert request = drive.files().insert(file, fileUpload);
    request.getMediaHttpUploader().setProgressListener(getProgressListener());
    request.execute();
  }

The error message really doesn't help too, any ideas where I am going wrong?