Stack Overflow
Stack Overflow

Questions tagged [azure-synapse]

Questions about Azure Synapse Analytics, bringing together enterprise data warehousing and big data analytics.

Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless or provisioned resources—at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs.

2525 questions
3
votes
1 answer

Azure Synapse Analytics (formerly SQL SW) vs Azure Synapse Analytics (workspaces preview)

What are the differences between the following Azure Services? Azure Synapse Analytics (formerly SQL DW) Azure Synapse Analytics (private link hubs preview) Azure Synapse Analytics (workspaces preview) Are these three different products? Or are…
Duane Theriot
  • 2,135
  • 1
  • 13
  • 16
3
votes
1 answer

Cannot Add Managed Identity to Synapse Pool

I am running an Azure Synapse workflow through the Synapse studio and running into this error: { "errorCode": "2200", "message": "ErrorCode=FailedDbOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Please…
C.Nivs
  • 12,353
  • 2
  • 19
  • 44
3
votes
2 answers

Facing Issue in writing PySpark Dataframe to Azure Synapse

I have a PySpark dataframe in Azure Databricks. I want to write into Azure Synapse. But i am getting below error. com.microsoft.sqlserver.jdbc.SQLServerException: The statement failed. Column 'ETL_TableName' has a data type that cannot…
HimanshuSPaul
  • 278
  • 1
  • 4
  • 19
3
votes
1 answer

Databricks version 7.0 not behaving like version 6.3: class java.lang.Long cannot be cast to class java.lang.Integer

I have a working notebook at azure databricks version 6.3 - Spark 2.4.4 This notebook does ingestions into Azure Synapse Analytics using it's connector When I upgraded the notebook to version 7.0 - Spark 3.0.0, the process begun to fail with the…
Flavio Pegas
  • 388
  • 1
  • 9
  • 26
3
votes
2 answers

Azure Synapse - retrieve inserted row identity value

We have an ETL job, which would need to insert a row into Table X, and then retrieve the IDENTITY value for the row inserted into Table X for use in later transformations. IDENTITY is used as a surrogate key, eg like here. In SQL Server it is…
Kristo_R
  • 167
  • 1
  • 13
3
votes
1 answer

Slow Azure Data Factory Pipeline

I am using Azure Data Factory V2 to transfer some csv files from Azure Data Lake to Azure Synapse I have a loop to find all files in special folder on my DataLake. After i have a DataFlow to transfer data from staging to main table. In my for-each…
Ardalan Shahgholi
  • 11,967
  • 21
  • 108
  • 144
3
votes
2 answers

Azure Synapse - String Delimiter

I have a text file with the following format. "01|""sample""|""Test"|""testing""|""01"|"""". I have created an external table in Azure Synapse by setting the format option STRING_DELIMITER to '"'. But while processing the file through an sp, i am…
Sandeep T
  • 421
  • 8
  • 22
3
votes
1 answer

Writing from Databricks to Synapse (Azure DW) very slow

We are using Databricks and its SQL DW connector to load data into Synapse. I have a dataset with 10 000 rows and 40 columns. It takes 7 minutes! Loading same dataset using Data Factory with Polybase and staging option takes 27 seconds. Same with…
Tero Kruth
  • 41
  • 5
3
votes
2 answers

Why Azure Data Factory when we have Polybase

Our requirement is to take data from Blob Storage and convert to other tabular form. This can be achieved with Sql DW using polybase. What really is the role of Azure data factory in such cases? I know the same objective can be met with Azure data…
Blue Clouds
  • 7,295
  • 4
  • 71
  • 112
3
votes
1 answer

Show Disk Space Used for all Tables - Azure SQL Data Warehouse

We are transitioning across to Azure SQL Data Warehouse - and an issue that's been highlighted is the need to change some smaller tables from Round-Robin / Hash-distributed to Replicated to improve performance. MS Design Guidance (See Here) suggests…
Celador
  • 173
  • 1
  • 11
3
votes
2 answers

Cannot find data type 'ntext' error in Azure SQL Data Warehouse

I am running a UPDATE statement against Azure SQL Data Warehouse using pyodbc: cursor.execute( "UPDATE dbo.test SET desc = ? WHERE id = ?", desc, id ) This works fine when desc value is simple. But when the value of desc is more complex (longer…
kee
  • 10,969
  • 24
  • 107
  • 168
3
votes
1 answer

IDENTITY not generating sequential values

I am running the below query to create a dimension table; however the identity column is not yielding sequential values; the values are very random. Any reason for this? I have tried a stored procedure and also a manual insert; but the result is…
Prathamesh
  • 97
  • 2
  • 8
3
votes
1 answer

How can I efficiently prevent duplicated rows in my facts table?

I have built a Data Factory pipeline which ETL the data from a Data Lake into a Datawarehouse. I chose the SCD type 1 for my dimensions. My pipeline contains the following activities: [Stored Procedure] Clear staging tables; [Stored Procedure] Get…
Kzryzstof
  • 7,688
  • 10
  • 61
  • 108
3
votes
1 answer

find the first value based on date and id column

I want to find the values of time taken by a given depot for the stationary. Below is the code for the create table and values. I have also achieved the other requirements for the same table and also have shared the code below. I want to create an…
Raj
  • 221
  • 1
  • 13
3
votes
2 answers

Can I change the datatype of the Spark dataframe columns that is being loaded to SQL Server as a table?

I am trying to read a Parquet file from Azure Data Lake using the following Pyspark code. df= sqlContext.read.format("parquet") .option("header", "true") .option("inferSchema", "true") .load("adl://xyz/abc.parquet") df =…
sri sivani charan
  • 399
  • 1
  • 6
  • 21
1 2 3
…
99 100