1

I'm trying to read a json file stored on my OVH object storage (openstack).

I set everything up :

import pyspark
from pyspark.sql.session import SparkSession
spark = SparkSession.builder.appName("test").getOrCreate()

also the hadoop conf :


sc=spark.sparkContext
hadoopConf=sc._jsc.hadoopConfiguration()
hadoopConf.set("fs.swift.impl","org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem")
hadoopConf.set("fs.swift.service.auth.endpoint.prefix","/AUTH_")
hadoopConf.set("fs.swift.service.abc.http.port","443")
hadoopConf.set("fs.swift.service.abc.auth.url","https://auth.cloud.ovh.net/v2.0/tokens")
hadoopConf.set("fs.swift.service.abc.tenant","MYTENANT")
hadoopConf.set("fs.swift.service.abc.region","MYREG")
hadoopConf.set("fs.swift.service.abc.useApikey","false")
hadoopConf.set("fs.swift.service.abc.username","MYUSER")
hadoopConf.set("fs.swift.service.abc.password","MYPASS")

and then

spark.read.json("swift://mycontainer.abc/yyy.json")

throws the error

org.apache.hadoop.fs.swift.exceptions.SwiftException: Failed to parse Last-Modified: Tue, 21 Apr 2020 20:12:43 GMT
    at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:237)
    at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:182)
    at ...
Caused by: java.text.ParseException: Unparseable date: "Tue, 21 Apr 2020 20:12:43 GMT"

like it is not able to parse the metadata date "Tue, 21 Apr 2020 20:12:43 GMT". I cannot figure out how to solve this problem.

Jean-Luc Bonnaux
  • 11
  • 1
  • Have a look at the class `org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem` on your classpath and check what is the value of PATTERN string. Based on what I see, the date that is being parsed is in the correct format. `https://github.com/DmitryMezhensky/Hadoop-and-Swift-integration/blob/00c2c9cf404f7480ec2c24f918c4b4c055b23800/swift-file-system/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeFileSystemStore.java#L56`. You could debug and find what is the pattern for the date on the class in your classpath – Deepak Apr 22 '20 at 04:46

0 Answers0