Stack Overflow
Stack Overflow

Questions tagged [cdap]

CDAP exposes developer APIs (Application Programming Interfaces) for creating applications and accessing core CDAP services. CDAP defines and implements a diverse collection of services that support applications and data on existing Hadoop infrastructure such as HBase, HDFS, YARN, MapReduce, Hive, and Spark.

References

138 questions
0
votes
1 answer

How to run cdap datafusion pipelines sequentially

I have a scenario where i have 5 pipelines which i want to run sequentially one after other. Is there any way to do this. I tried reading the documentation but it wasn't clear. Thanks in advance!
code tutorial
  • 554
  • 1
  • 5
  • 17
0
votes
1 answer

GCP Datafusion upload custom plugin is too slow

Currently I am using basic edition of datafusion and i wanted to upload a custom plugin. It is too slow to upload the jar and json. Taking around 10 minutes of time to upload the plugin and entire browser is getting hanged. Did anyone faced this…
code tutorial
  • 554
  • 1
  • 5
  • 17
0
votes
1 answer

Unable to upload CDAP custom plugin in Datafusion

I am trying to upload http-sink plugin from Datafusion upload button. I cloned the http-sink repository and I made few very minimal changes and packaged it as a JAR. Now when I try to upload the JAR and the JSON files using upload plugin button, I…
code tutorial
  • 554
  • 1
  • 5
  • 17
0
votes
1 answer

Implementing SCD type2 in Data Fusion

Am trying to implement type2 in Datafusion.Can someone help in performing insert and updating by using the pipeline transformations/action/conditions to achieve this. I was trying to generate hash using Wrangler for both source and Target and join…
Manasa
  • 31
  • 3
0
votes
1 answer

auto-detect nested json response of HTTP plugin in datafusion

I'm trying to call an HTTP GET API using HTTP batch source plugin in data fusion. The response of the API is a complex dynamic nested json because of which I cannot manually specify the output schema. Is there anyway to overcome this problem. Thanks…
code tutorial
  • 554
  • 1
  • 5
  • 17
0
votes
1 answer

CDAP spark phoenix application connection issue

We have a cdap application to connection to phoenix table from spark using phoenix driver. I have the phoenix version 4.7 in our environment. As per the standard spark2 phoenix connectivity, it requires only the phoenix-spark2 as a dependency and…
ae8
  • 53
  • 1
  • 6
0
votes
1 answer

How can I provide runtime parameter/argument values via a file for CDAP pipleline

How do i add run time parameter/arguments to CDAP pipeline. We can set parameters in all the Transforms, Source and Target nodes and manually enter the values at run time in the Datafusion environments. However in a production environments i would…
Trishit Ghosh
  • 235
  • 3
  • 10
0
votes
0 answers

Pipeline Fail to put into BigQuery Sink MapReduce Program 'phase-1' failed

I am trying to build a simple pipeline that moves data from our Cloud SQL (MySQL) into BigQuery. All the JDBC driver stuff is working fine (if I use the trash can as a sink, I can see the preview data) and the schema propagated. I created the…
Dino
  • 352
  • 2
  • 8
0
votes
1 answer

Trying to upload the latest plugin for Google Cloud to Data Fusion but getting an error while uploading

Reference to this post that I had earlier: Possible to modify or delete rows from a table in BigQuery dataset with a Cloud Data Fusion pipeline? I am trying to do the suggested answer to compile the latest version of Google Cloud Platform plugin and…
Bluescrod
  • 81
  • 1
  • 7
0
votes
1 answer

CDAP ingestion from PubSub

I'm trying to load data from PubSub messages to GCS files. Simple pipeline: PubSub source -> JSON Parser -> GCS sink. Since PubSub only accept the data argument as utf-8, how can I decode it in CDAP? Should I build a custom plugin implementing a…
Luca Natali
  • 414
  • 5
  • 14
0
votes
1 answer

How to remove temporary files from CDAP?

I am running CDAP server for 24x7. But after 5days of continuous running the server gets restarted, due to the low memory. CDAP: sandbox OS: UBUNTU CDAP Version: 5.0.0
muTheTechie
  • 1,443
  • 17
  • 25
0
votes
1 answer

python transform in CDAP Tool

I have trying to create a python transformation in CDAP where I am facing below issue. Request you to kindly check and let me know the update as soon as possible. Please find the below issue,use case for your reference. Issue : Configuring stage…
sai krishna
  • 151
  • 2
  • 11
0
votes
1 answer

CDAP with Azure Data bricks

Has anyone tried using Azure data bricks as the spark cluster for CDAP job processing. CDAP documentation details how to add it to Azure HDInsight, but just wondering is there a way to configure CDAP to point to data bricks spark cluster, is it even…
sp-eagle
  • 1
  • 1
0
votes
1 answer

Cask CDAP services started, but not running during installation

After going through the docs for installing CDAP on MapR system (v6.0) and starting the cdap services, am finding that some CDAP services not running after startup…
lampShadesDrifter
  • 3,925
  • 8
  • 40
  • 102
0
votes
1 answer

cloudera installation process and clustering in local network

How to install cloudera in local system .I'm using centos 6.5.And also I want to do clustering in cloudera .Any one suggest me some documentation to this process properly
Gopirengaraj
  • 43
  • 6
1 2 3
…
9
10