Stack Overflow
Stack Overflow

Questions tagged [azure-synapse]

Questions about Azure Synapse Analytics, bringing together enterprise data warehousing and big data analytics.

Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless or provisioned resources—at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs.

2525 questions
0
votes
1 answer

Apache Spark Pool Mongodb connector

I have been trying to read/write with synapse spark pools into a mongodb atlas server, i have tried PyMongo but im more interested in using the mongodb spark connector but in the install procedure they use this command: ./bin/pyspark --conf…
frammnm
  • 537
  • 1
  • 5
  • 17
0
votes
1 answer

Azure Synapse (Azure Data Factory) REST source pagination with dynamic limit

In Azure Synapse I'm performing a data copy action. It has a REST API as a source and needs to store this data in a on-premise SQL table. The whole setup is configured and the table is filled with 100 records. This is a limit from the API. It…
AntonyJ
  • 29
  • 6
0
votes
1 answer

How to Load data from Azure SQL DB to Synapse Spark Datalake?

I am looking for a Solution through which we can load the data from Azure SQL DB to Azure Synapse Spark Datalake (not in dedicated pool). The Requirements are: we have a csv file in which data is present. Currently we are updating or inserting the…
BigData Lover
  • 135
  • 8
0
votes
2 answers

Synapse/ADF "errorCode": "2200", "message": "Failure happened on 'Source' side. ErrorCode=JsonInvalidDataFormat

I am making a rest call using a copy activity to write data to a datalake storage. Source is a rest dataset and sink is a json. If I use a http binary dataset it works fine, but then I can not perform pagination in an easy way. I am getting the…
Abashirsoldom
  • 11
  • 5
0
votes
0 answers

Creating cetas table returns multiple files with .text.deflate extension

I have created an external data source and CSV file format.I am creating an external table susing cetas script create external table test With (location='test/data/', Data_source=test_datasource, File_format=csv_format) As select * from…
Robs8687
  • 1
  • 1
0
votes
1 answer

How to connect to sap bw (on linux server) from azure data factory or synapse analytics pipeline?

There are many posts and documents on connecting to the SAP BW system by installing the self-hosted integration runtime on a windows server. Still, my problem is the sap BW system is running on a Linux environment. Has anyone successfully created a…
deba prasad nayak
  • 21
  • 6
0
votes
1 answer

Azure Synapse Copy Data Tool conversion overflow error while creating text files in ADLS gen2

I'm trying to fetch data (source) from SQL server to create a text file in ADLS (Sink), but I'm facing conversion overflow error. which I found out is with this value (999999999.00000000000000000000). can someone help me on this. I used Cast…
BM Baloch
  • 3
  • 2
0
votes
1 answer

Performance problem of MERGE statement in Azure Synapse

My DWH is deployed on Azure Synapse SQL pool. I loaded data to DWH by script that consists of update, insert and delete (u-i-d) operations. The duration of full load to target table was 12minutes for near 50million of rows. Recently I tried to use…
niko
  • 11
  • 1
0
votes
1 answer

How can I ensure that only one instance of an Azure Synapse Pipeline modifies a lake database at the same time?

I have created an Azure Synapse Analytics pipeline that is triggered every time a new file is added to a certain directory. It basically obtains the name of the file as input parameter of a notebook, which then reads said file and updates a lake…
MartinMB
  • 13
  • 2
0
votes
1 answer

How to Send a failure email to third party in Synapse if the conditions in our Code is not satisfying the Rules?

I am Currently Looking for a Solution in which we are Doing some transformations in our Synapse Notebook and we should send a mail to third party regarding the failure if the rules are not satisfying . The Scenario here is i am currently writing a…
BigData Lover
  • 135
  • 8
0
votes
1 answer

Azure Data Factory Copy Activity from csv to Azure Synapse is failing with identity columns in synapse tables

I am trying to do a simple copy data from csv in ADLS to Azure Synapse in ADF Pipeline . I have also set pre Copy script to Identity Insert ON but still the Copy activity fails with "An explicit value for the identity column in table can only be…
PAVAN
  • 33
  • 1
  • 4
0
votes
1 answer

Spark throws awaitResult exception while loading Parquet file at readParquetFootersInParallel executor

I am trying to load a parquet file in a MS Synapse Analytics Notebook. When the following error occurs. df1 = spark.read.load(abfss_path+'file_name.parquet', format='parquet') Py4JJavaError: An error occurred while calling o1216.load. :…
Prosuit
  • 1
0
votes
2 answers

How do I calculate the cost of Azure Synapse Analytics?

I need to calculate the cost of the Azure Synapse Analytics. I have used the Azure Pricing Calculator but I could not figure it out. It shows close to USD 2,100. I have the following components as a part of the Azure Synapse Analytics Synapse…
One Developer
  • 99
  • 5
  • 43
  • 103
0
votes
0 answers

Synapse: Integration runtime and Notebooks

I'm trying to load data to a Spark DataFrame from MSSQL/Postgres behind a firewall. When I use pipelines and datasets I can use a Linked service that connects via an integration runtime. How to do it with a notebook and dataframe? Is there a way to…
Robert G
  • 3
  • 2
0
votes
4 answers

Azure Pipeline storage file trigger doesn't fire

I´ve created an Azure Synapse Analytics Pipeline that must be triggered by the creation of a file within a Azure Gen2 storage account. Somehow the blob creation event (i.e. when I upload the file in the corresponding container and folder) doesn´t…
fbglv
  • 31
  • 2
  • 5
1 2 3
…
99
100