0

How can I save file to IBM Cloud with using Python script located in a Jupyter Notebook? I have completed the data mining process with my Jupyter Notebook and I want to save the results in .json.

rajah9
  • 11,645
  • 5
  • 44
  • 57
Tungsteniac
  • 1
  • 3

1 Answers1

0

  • Many commands are detailed in this notebook

  • You will need to download ibm-cos-sdk

  • You will also need your Cloud Object Storage credentials (you can found them in cloud.ibm.com, in the interface). I struggled for the mapping between credentials keys thought (ibm_api_key_id VS apikey for example), but the mapping is explained here

Here is an example of what you will get :

from ibm_botocore.client import Config
import ibm_boto3
import json
import pandas as pd

credentials = {
    'IBM_API_KEY_ID': '*******************************',
    'IAM_SERVICE_ID': '*******************************',
    'ENDPOINT': '*******************************',
    'IBM_AUTH_ENDPOINT': '*******************************',
    'BUCKET': '*******************************',
    'FILE': 'wine.csv'
}

cos = ibm_boto3.client(service_name='s3',
    ibm_api_key_id=credentials['IBM_API_KEY_ID'],
    ibm_service_instance_id=credentials['IAM_SERVICE_ID'],
    ibm_auth_endpoint=credentials['IBM_AUTH_ENDPOINT'],
    config=Config(signature_version='oauth'),
    endpoint_url=credentials['ENDPOINT'])

df = pd.DataFrame({'A': [1, 2, 3], 'B': [2, 10, 9]})
data = df.to_json()
type(data)
with open('mydata.json', 'w') as outfile:
    json.dump(data, outfile)

# Upload file mydata.json' from wine folder into project bucket nice_data.json
cos.upload_file(Filename='mydata.json',Bucket=credentials['BUCKET'],Key='nice_data.json')
Phoenixo
  • 2,071
  • 1
  • 6
  • 13
  • I cant imagine how ways to download and to upload help me to save this: bakeries=df_bakeries.to_json(). What's next? What is json object name as a file?. – Tungsteniac Jan 28 '20 at 11:08
  • Where I can look at 'IAM_SERVICE_ID'? – Tungsteniac Jan 28 '20 at 11:31
  • I updated my answer with a full example : a dataframe converted into json and uploaded in your object storage. Another name of 'IAM_SERVICE_ID' can be 'RESOURCE_INSTANCE_ID'. You can find it in ibmcloud, click on your object storage and search in "Service Credentials" – Phoenixo Jan 28 '20 at 13:05
  • S3UploadFailedError: Failed to upload bakeries.json to ...Cinnabon.json: An error occurred (NoSuchKey) when calling the PutObject operation: The specified key does not exist. – Tungsteniac Jan 31 '20 at 07:03
  • Maybe have a look [here](https://github.com/IBM/ibm-cos-sdk-js/issues/26). It's possible that your error comes from the auth-endpoint. Try this : "ibmAuthEndpoint": "https://iam.ng.bluemix.net/oidc/token" – Phoenixo Jan 31 '20 at 13:49
  • `MissingSchema: Invalid URL 'iam.ng.bluemix.net/oidc/token': No schema supplied. Perhaps you meant http://iam.ng.bluemix.net/oidc/token?` – Tungsteniac Jan 31 '20 at 14:00
  • I tried this one that worked : "https://iam.eu-de.bluemix.net/oidc/token" – Phoenixo Jan 31 '20 at 14:06
  • Thanks for help. but it raised `Invalid URL 'iam.eu-de.bluemix.net/oidc/token': No schema supplied. Perhaps you meant http://iam.eu-de.bluemix.net/oidc/token?` again. – Tungsteniac Jan 31 '20 at 14:10
  • `CredentialRetrievalError: Error when retrieving credentials from http://iam.eu-de.bluemix.net/oidc/token?: HttpCode(405) - Retrieval of tokens from server failed.` – Tungsteniac Jan 31 '20 at 14:11
  • Yeah please add **https** in front if this url : `https://iam.eu-de.bluemix.net/oidc/token` – Phoenixo Jan 31 '20 at 14:13
  • S3UploadFailedError: Failed to upload mydata.json to courserads-donotdelete-pr-7suxmds10oqqm9/nice_data.json: An error occurred (404) when calling the PutObject operation: Not Found – Tungsteniac Jan 31 '20 at 14:14
  • Can you check [here](https://stackoverflow.com/questions/46635895/aws-boto3-s3-python-an-error-occurred-404-when-calling-the-headobject-operat) ? It seems that it could be a mispelling error with the '/' for example – Phoenixo Jan 31 '20 at 14:17