How to Access a Text File from IBM DSX Scala-Spark Notebook

Question

This post is useful, if:

You have created a IBM DSX notebook with “Language=Scala” and “Spark version=2.1 OR 2.0″[did not try with other versions]
AND You are trying to access a text file [did not try other formats] that you have uploaded from your personal computer to the IBM Cloud Object Storage

So basically you decided to cut and paste a blog post to answer a question you conjured up? — Dr G., Feb 04 '18 at 17:52
@A.J.Alger That is my personal website. I wrote that blog post myself. — H.Rehman, Feb 06 '18 at 05:01
Yeah. So you made up a question then answered it yourself. Not exactly SO ettiquite — Dr G., Feb 06 '18 at 05:19
As long as the question is useful, I think there is no harm in that. It seems you have not faced this particular problem yourself. As I have mentioned in my post, I have searched several search engines for hours to find a solution for this problem and even talked to IBM representative. Considering that Stackoverflow is an esteemed platform for such situations, I decided to share my findings. I am really disappointed the way some people taking the whole thing. — H.Rehman, Feb 06 '18 at 05:26

score 0 · Answer 1 · edited Feb 04 '18 at 22:36

In order to access files that are in Cloud Object Storage, you need to provide the path of the file in the form of a CloudObjectStorage instance. Add following lines, after the file credentials, to your code:

var soc = new CloudObjectStorage(sc, credentials, null, "bluemix_cos")
var myFile = sc.textFile(soc.url("your_bucket","data.txt"))

In the second code line above, "your_bucket" is the value of "BUCKET" in the file credentials.

There is one last thing we need to do before we are good to go. The CloudObjectStorage constructor accepts credentials in a way that is for the CSV files. Luckily the main difference between the two formats is the HashMap key names; hence, we can easily manage this problem by changing the above credentials manually, as mentioned below:

"IBM_API_KEY_ID" to "apiKey"

"IAM_SERVICE_ID" to "serviceId"

"ENDPOINT" to "endPoint"

"IBM_AUTH_ENDPOINT" to "iamServiceEndpoint"

For further details, you can visit the link below: http://rehmans.org/2018/01/31/how-to-access-a-text-file-from-ibm-dsx-scala-spark-notebook/

How to Access a Text File from IBM DSX Scala-Spark Notebook

1 Answers1