Stack Overflow
Stack Overflow

Questions tagged [analytics-engine]

IBM Analytics Engine - a single Hadoop and Spark service providing an environment to develop and deploy advanced analytics applications in minutes.

What is Analytics Engine?

Build and deploy clusters within minutes with simplified user experience, scalability, and reliability. Custom configure the environment. Administer through multiple interfaces. Scale on demand.

31 questions
2
votes
1 answer

Hive can't find partitioned data written by Spark Structured Streaming

I have a spark structured streaming job, writing data to IBM Cloud Object Storage (S3): dataDf. writeStream. format("parquet"). trigger(Trigger.ProcessingTime(trigger_time_ms)). option("checkpointLocation", s"${s3Url}/checkpoint"). …
Chris Snow
  • 23,813
  • 35
  • 144
  • 309
2
votes
1 answer

java.io.FileNotFoundException: Not found cos://mybucket.myservicename/checkpoint/offsets

I'm trying to use Spark Structured Streaming 2.3 to read data from Kafka (IBM Message Hub) and save it into IBM Cloud Object Storage on a 1.1 IBM Analytics Engine Cluster. After creating the cluster, ssh into it: $ ssh…
Chris Snow
  • 23,813
  • 35
  • 144
  • 309
1
vote
2 answers

IBM Cloud: How to disable analytics engine when unused?

I using Analytics Engine on IBM Cloud, but I can't see anywhere how I can stop it when I'm not working. So it takes one day to consumes all my credit as I'm using a Lite account. How to stop it or pause it when I'm not working?
abdoulsn
  • 842
  • 2
  • 16
  • 32
1
vote
1 answer

Could not get a Transport from the Transport Pool for host

I'm trying to write to an IBM Compose Elasticsearch sink from Spark Structured Streaming on IBM Analytics Engine. My spark code: dataDf .writeStream .outputMode(OutputMode.Append) .format("org.elasticsearch.spark.sql") …
Chris Snow
  • 23,813
  • 35
  • 144
  • 309
1
vote
1 answer

WARN Session: Error creating pool to /xxx.xxx.xxx.xxx:28730

I'm trying to connect to a ScyllaDB database running on IBM Cloud from Spark 2.3 running on IBM Analytics Engine. I'm starting the spark shell like so ... $ spark-shell --master local[1] \ --files jaas.conf \ --packages…
Chris Snow
  • 23,813
  • 35
  • 144
  • 309
1
vote
1 answer

How do I configure spark.driver.memory for connecting from Watson Studio (DSX) notebook to IAE?

I need to configure spark.driver.memory for connecting from a Watson Studio (DSX) notebook to IAE? Where/how can I configure this?
Chris Snow
  • 23,813
  • 35
  • 144
  • 309
1
vote
1 answer

Error while creating external Hive table in IBM Analytics Engine

I am creating an external hive table from a csv file located on IBM Cloud Object Storage. I am using beeline client while ssh'd into the cluster with the clsadmin user. I was able to make jdbc connection. Getting the below error while creating the…
Rohan Vaidyanathan
  • 43
  • 7
0
votes
1 answer

How to integrate Watson Studio python notebook with an IAE Spark?

I have an IBM Analytics Engine (IAE) instance, added it to my Watson Studio project as an associated service and created an Environment based on it. Then, I created a python notebook and set its environment to the abovementioned. I ran a simple…
Ronen491
  • 61
  • 6
0
votes
1 answer

Error "Your account has not been authorized to select the Lite Plan" when creating a free IBM Cloud Analytics Engine instance using Lite account

I following one of the hadoop courses from cognitievclass.ai and was attempting to create an Analytics Engine instance as instructed. As mentioned in the IBM Cloud catalog, the Analytics Engine is available for Lite accounts for free but whenever I…
Suchitha Dehigaspitiya
  • 13
  • 3
0
votes
2 answers

IBM Analytics Engine - Cluster creation fails if i pass Ambari configuration as part of advance options

I using Analytics Engine on IBM Cloud and trying to pass Ambari configuration Like below in Advanced provisioning options. { "ambari_config": { "hardware_config": "default", "software_package": "ae-1.2-hive-spark", …
Rachit Arora
  • 1
  • 1
0
votes
0 answers

Missing classes from JAR created by sbt assembly

Here's my build.sbt file: name := "CMDW-Security" version := "19.11.25" scalaVersion := "2.11.12" assemblyJarName in assembly := "CMDW-Security.jar" test in assembly := {} fullClasspath in assembly := (fullClasspath in…
Sparker0i
  • 1,787
  • 4
  • 35
  • 60
0
votes
1 answer

Running same JAR on different machines using spark-submit crashes on one machine

I am trying to run a JAR file using spark-submit on: My local machine (Mac OS) Another local machine (Windows) IBM Cloud Analytics Engine Console PS: All of them run the same Spark and Scala versions (Spark - 2.3.2 and Scala - 2.11.12) This is my…
Sparker0i
  • 1,787
  • 4
  • 35
  • 60
0
votes
1 answer

IBM Cloud: How to open Analytics Engine port 7070?

I want to use big data services on the IBM Cloud, so I found Analytics Engine(AE) and BigInsight, but unfortunately, BigInsight is going to stop, so I can only choose AE. However, IBM AE is different AWS and GCP big data services, AE prohibits users…
Hades Su
  • 25
  • 4
0
votes
1 answer

how to debug hive on IBM Analytics Engine?

I'm trying to debug a Hive job running on IBM Analytics Engine (IAE). I have tried the following instructions: https://community.hortonworks.com/content/supportkb/150214/how-to-enable-debug-hive-cli-logging.html hive -hiveconf…
Chris Snow
  • 23,813
  • 35
  • 144
  • 309
0
votes
1 answer

hive query is returning no data

CREATE EXTERNAL TABLE invoiceitems ( InvoiceNo INT, StockCode INT, Description STRING, Quantity INT, InvoiceDate BIGINT, UnitPrice DOUBLE, CustomerID INT, Country STRING, LineNo INT, InvoiceTime STRING, StoreID INT, …
Chris Snow
  • 23,813
  • 35
  • 144
  • 309
1
2 3