0

Can you please let me know how to read from Google Storage (CSV) file to Cloud Data Store.

I had done, Java code via App Engine and can able to load one row at a invocation. Same way i would like to get some sample code to load bulk ( read from CSV) to Data Store in a single go.

user12607246
  • 5
  • 3

1 Answers1

0

Can you please let me know how to read from Google Storage (CSV) file to Cloud Data Store.

There are two different workarounds that you could use in order to read from Google Cloud Storage and loaded into your Cloud Datastore project.

Using Apache Beam

As mentioned in this similar post you could use Apache Beam to read the CSV file using the TextIO class.

Next, you will need to apply a transformation that will parse each row in the CSV file and return an Entity object.

In the post you will find an example of how to construct an Entity object based on a CSV file.

Lastly, write the Entity objects to Cloud Datastore.

Using Dataflow

You could use Dataflow. Google provides a set of open-source Dataflow templates that you could use in order to achieve what you are looking for.

As far as the templates is concerned, you could use the Cloud Storage Text to Datastore.

The Cloud Storage Text to Datastore template is a batch pipeline which reads from text files stored in Cloud Storage and writes JSON encoded Entities to Datastore. Each line in the input text files should be in JSON format specified in https://cloud.google.com/datastore/docs/reference/rest/v1/Entity .

I highly recommend going with the first option since it looks good and it got approved.

If it does not work you can always try to use Dataflow templates to read from Google Cloud Storage into Cloud Datastore.

I hope it helps.

Christopher Rodriguez Conde
  • 879
  • 5
  • 7
  • Thanks Christoper for your help. I had seen the first option and it in Python however it will be great if you could provide in Java it will helpful for me to start over. – user12607246 Jan 02 '20 at 08:22
  • The accepted answer in this [post](https://stackoverflow.com/questions/48473024/import-csv-into-google-cloud-datastore) is in Java. Also you can take a look at the method use [here](https://beam.apache.org/releases/javadoc/2.2.0/org/apache/beam/sdk/io/TextIO.html) – Christopher Rodriguez Conde Jan 02 '20 at 11:54