Stack Overflow
Stack Overflow

Questions tagged [spark-thriftserver]

95 questions
0
votes
1 answer

Not able to connect to spark thrift server via jdbc driver in play framework with anorm

I am trying to connect my spark thrift server using play framework via anorm. Play framework version - 2.8.18 Connecting to jdbc as explained here Anorm - 2.7.0 Spark thrift server version - spark 3.2.1, Hadoop 3.2 Scala - 2.13.5 Jdbc driver &…
Bill Goldberg
  • 1,699
  • 5
  • 26
  • 50
0
votes
0 answers

Beeline not working when connecting with thriftserver using !connect jdbc:hive2://localhost:10000

It is showing the following error : 23/04/28 15:35:40 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10000 Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000: Failed to open new session:…
Anshuman Raina
  • 1
0
votes
0 answers

Is it possible to read data from external Spark Thrift server of version 2.4.8 from Spark 3.3.1 application?

I'm trying to read a table from the instance of Spark Thrift server 2.4.8. The connected application is based on Spark 3.3.1 with Hive support of metastore version 3.1.3. Currently I'm receiving such error: org.apache.thrift.TApplicationException:…
Valerii Bosiak
  • 53
  • 7
0
votes
1 answer

Configure Spark 3 thrift server with Apache Ranger

I am trying to configure Spark 3.3.0 Thrift Server with Apache Ranger but I cannot find any resources or information for this setup.Any suggestions on how to implement this? Thanks very much! I already have an STS (kerberos jdbc) turned on and…
adel mejri
  • 13
  • 5
0
votes
0 answers

Spark Thrift Server connection to S3

I'm trying to connect to AWS S3 using Spark Thrift Service. I'm using: spark-defaults.conf spark.sql.warehouse.dir s3://demo-metastore-001/ spark.hadoop.fs.s3.impl …
Luis Estrada
  • 371
  • 7
  • 20
0
votes
1 answer

On kubernetes my spark worker pod is trying to access thrift pod by name

Okay. Where to start? I am deploying a set of Spark applications to a Kubernetes cluster. I have one Spark Master, 2 Spark Workers, MariaDB, a Hive Metastore (that uses MariaDB - and it's not a full Hive install - it's just the Metastore), and a…
Barry
  • 237
  • 1
  • 2
  • 11
0
votes
0 answers

spark thrift server issue: Length field is empty for varchar fields

I am trying to read Data from Spark Thrift Server using SAS. In the table definition through DBeaver, I am seeing that Length field is empty only for fields with VARCHAR data type. I can see the length in the Data Type field as varchar(32). But that…
Ayan Ray
  • 1
0
votes
0 answers

Spark SQL job stuck at the collect stage in the driver

The below is the last message in the logs. I am using spark version 3.1.2 INFO BlockManagerInfo: Removed broadcast_2**_piece0 on *****:32789 in memory I have 500 million strings in a single column in a big table, lets call it big_table and the…
rk1234 afbce
  • 33
  • 5
0
votes
0 answers

Timeout between application master and driver on Thrift Server

we are running a spark thrift server and the configuration as below. thrift drivers and application master are separated with firewall and all the port between these two are opened . Issue is after 2hr 11 mins application server will die because…
SARATH CHANDRAN
  • 307
  • 1
  • 6
  • 16
0
votes
1 answer

Why getKerberosTransport needs 3 part kerborse principle?

Good morning / afternoon / evening! Spark 2.4.x, with Hive 1.2.1 Source code here: https://github.com/apache/spark/blob/master/sql/hive-thriftserver/v1.2/src/main/java/org/apache/hive/service/auth/KerberosSaslHelper.java public static TTransport…
kcode2019
  • 119
  • 1
  • 7
0
votes
1 answer

Control the number of executors when running sql in spark thrift server

I have a spark thrift server,and users can submit spark sql to this server. I would like I can control the number of executors that will be used to run user's sql.That is, I don't want executors will be given as many as possible(later submitted sql…
Tom
  • 5,848
  • 12
  • 44
  • 104
0
votes
1 answer

SYNTHJOIN query from SAP BO towards Spark Thrift Server

Current Architecture: In our current architecture, we are using Hadoop File System to store our reporting data in Parquet file format and we are Spark Thrift Server to expose these data through a SQL interface (JDBC/ODBC). Using this interface, SAP…
Subash Kunjupillai
  • 390
  • 3
  • 12
0
votes
1 answer

Are communications with a Spark Thrift server in binary mode transmitted securely?

In my organisation, we have a Spark Thrift server setup with HTTP & SSL because there is an underlying assumption that the binary mode is not securely encrypted over the wire and thus may reveal credentials or sensitive query data. I have Googled,…
QA Collective
  • 2,222
  • 21
  • 34
0
votes
1 answer

Unable to keep Spark Thrift Server running

I need to expose some temporary tables on spark using Thrift. This is the base code I'm running on my cluster: from pyspark import SparkContext, SparkConf from pyspark.sql import SparkSession from py4j.java_gateway import java_import #…
Bruno Faria
  • 549
  • 1
  • 6
  • 20
0
votes
1 answer

Spark SQL cannot access Spark Thrift Server

I cannot configure Spark SQL so that I could access Hive Table in Spark Thrift Server (without using JDBC, but natively from Spark) I use single configuration file conf/hive-site.xml for both Spark Thrift Server and Spark SQL. I have…
oskarryn
  • 170
  • 2
  • 13
1 2 3 4
5
6 7