0

I have an on-prem database that has 50GB of JSON data. I'm planning to take the backup of this data to OCI object storage using Java SDK to a single object in a bucket. on-prem database paginates JSON data result. How can I stream/upload each page of data to OCI object storage whenever it arrives? Do I need to use multipart upload? Is there any example that shows the streaming of data to object storage?

The example in SDK shows uploading a file to Objectstorage but my requirement is to stream data without storing it to temp files.

akshay
  • 207
  • 1
  • 2
  • 7
  • 1
    Have you checked the Streaming service? It can write the contents of your stream directly to an Object Storage bucket. See this blogpost for more details: https://blogs.oracle.com/developers/publishing-to-object-storage-from-oracle-streaming-service – lsarecz Apr 17 '20 at 14:13
  • @lsarecz Thanks for the comment. OSS seems overkill for my requirements. I just need to export my database content to Object storage once. – akshay Apr 18 '20 at 08:24

1 Answers1

0

do you want to upload the individual JSON pages, or do you need to combine them back together first into a single JSON object?

I don't think the OCI Java SDK can help you with combining the different pages. For that, you may need a JSON serializer/deserializer, or possibly even work with plain JSON Strings. If the object is large enough, you may not want to actually parse it. Handling Strings is less robust, but possibly faster and more memory-efficient.

To upload, you would either use putObject in the ObjectStorageClient, or use the UploadManager. The latter can upload in multiple parts and is ideal for large objects. Here is an example: https://github.com/oracle/oci-java-sdk/blob/master/bmc-examples/src/main/java/UploadObjectExample.java

Mathias Ricken
  • 156
  • 3