Stack Overflow
Stack Overflow

Questions tagged [qubole]

Qubole Data Service (QDS) is cloud Big Data service running on an elastic Hadoop-based cluster

Source Creators of Facebook’s Big Data infrastructure and Apache Hive have leveraged their experience to deliver Qubole Data Service (QDS) – a cloud Big Data service offering the same advanced capabilities used by Big Data savvy organizations.

Minimize operational interaction and provide your data analysts with an easy to use graphical interface, built-in connectors, and seamless, elastic cloud infrastructure.

Your Hadoop cluster is ready within minutes post signup, letting you focus on building sophisticated data pipelines, running queries, scheduling jobs and monetizing your big data.

An auto-scaling cluster, improved I/O optimization, faster queries and support for hybrid pricing - realize cost savings of as much as 50%-60% in total, while accomplishing tasks faster.

87 questions
0
votes
1 answer

Scale plot size of matplotlib plots in Qubole Notebook

Is there a possibility of increasing the size of the plot plotted using z.showplot() in qubole notebooks. import matplotlib as plt plt.figure() plt.bar(pandas_df_hr_sg[:]['hour'],pandas_df_hr_sg[:]['count']) plt.title('Hourly wise user…
Mustajib Mohammed Khan
  • 1
0
votes
2 answers

How do I upgrade a library in Qubole's Jupyter Notebook, using PySpark?

Is there a way to do it right from a cell in the notebook? similar to pip install ... --upgrade I didn't know how to do what's instructed on…
HT.
  • 161
  • 1
  • 7
0
votes
1 answer

How to pass --properties-file to spark-submit in Qubole?

I am using Spark in Qubole by having the clusters created in AWS. In Qubole Workbench, when I execute the below Command Line, it works fine and the command is successful /usr/lib/spark/bin/spark-submit s3://bucket-name/SparkScripts/test.py But,…
Saravanan
  • 49
  • 6
0
votes
2 answers

How to import a .py file to Qubole?

I'm connecting to Azure data lake, and I have the file there, but it's in a different path, and I don't know how to import it. Thank you in advance for your help!
HT.
  • 161
  • 1
  • 7
0
votes
1 answer

In the new Analyze UI, how do I edit the title of my query?

In the new Qubole Analyze UI that came out recently, I cannot seem to find a way to change the title of a command. In the old interface, I could click on the command title and it would become an editable text box.
GreenGiant
  • 4,930
  • 1
  • 46
  • 76
0
votes
1 answer

Performance analysis using Sparklens of Spark Streaming Application

I am trying to get performance analysis of a spark streaming application using sparklens. It is giving results like this Executor count 1 ( 80%) estimated time 01m 29s and estimated cluster utilization 18.55% Executor count 2 (100%)…
Abhay
  • 687
  • 4
  • 13
  • 22
0
votes
0 answers

How to fix 'Malformed class name' error in Spark Scala?

In Qubole notebook I am trying to get certain string from API response. It seems to be working just fine for sample data but fails when I use the full set. Spark version: 2.3.1; Scala version: 2.11; scalaj-http version: 2.4.2 import…
Piotr
  • 1
  • 1
0
votes
1 answer

retrieve size of data copied with hadoop distcp

I am running a hadoop distcp command as below: hadoop distcp src-loc target-loc I want to know the size of the data copied by running this command. I am planning to run the command on Qubole. Any help is appreciated
sneha salvi
  • 57
  • 2
  • 11
0
votes
1 answer

Get correct value from array in Hive QL

I have a Wrapped Array and want to only get the corresponding value struct when I query with LATERAL VIEW EXPLODE. SAMPLE STRUCTURE: COLUMNNAME: theARRAY WrappedArray([null,theVal,valTags,[123,null,null,null,null,null],false],…
noobeerp
  • 417
  • 2
  • 6
  • 11
0
votes
0 answers

Convert column in presto from epoch to date

I tried this but that didn't work. cast(from_unixtime('1532568232662880')) as date Any other ideas?
nak5120
  • 4,089
  • 4
  • 35
  • 94
0
votes
1 answer

Amazon s3Exception bad request and location constraint in hadoop s3a

Does location constraint require extra permission policy for hadoop s3a? I am seeing Exception in thread "main" com.qubole.com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad…
Jal
  • 2,174
  • 1
  • 18
  • 37
0
votes
1 answer

How do I get the value without the square brackets

I have created a dataframe using Scala and Spark SQL. I wanted the first value from the table but I am getting it inside of square brackets []. Can I just get the value without the brackets? Code: val sigh = sqlContext.sql("""SELECT DISTINCT…
noobeerp
  • 417
  • 2
  • 6
  • 11
0
votes
0 answers

select a table from a database in R

I am using dbplyr to select a table from a remote database using Rstudio. I connected with Spark in the server using livy. It shows me the databases I have but when I try to access one of the tables in one of the schemas, it…
Fisseha Berhane
  • 2,533
  • 4
  • 30
  • 48
0
votes
1 answer

Get Not Null Values in Wrapped Array

I have a Wrapped Array and want to only get the Non Null values when I query with LATERAL VIEW EXPLODE. I also tried IS NOT NULL but that does not return anything. SAMPLE STRUCTURE: COLUMNNAME:…
noobeerp
  • 417
  • 2
  • 6
  • 11
0
votes
1 answer

Set partition location in Qubole metastore using Spark

How to set partition location for my Hive table in Qubole metastore? I know that this is MySQL DB, but how to access to it and pass a SQL script with a fix using Spark? UPD: The issue is that ALTER TABLE table_name [PARTITION (partition_spec)] SET…
Vova Lis
  • 21
  • 3
1 2 3 4
5
6