Stack Overflow
Stack Overflow

Questions tagged [azure-synapse]

Questions about Azure Synapse Analytics, bringing together enterprise data warehousing and big data analytics.

Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless or provisioned resources—at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs.

2525 questions
0
votes
1 answer

How to copy entire structure between storage accounts in python

my case is the following: Two Azure Storage Accounts (Source/Destination) Source Account may contain multiple containers, folders, blobs, etc. All of the above needs to be copied exactly in the same structure to the DESTINATION account. If any…
Thanatos
  • 5
  • 1
0
votes
1 answer

Azure Synapse Delta Table Creation and Import Data From ADLS delta lake

We have requirement to load the data from ADLS delta data into synapse table. actually, we are writing the delta format data into ADLS gen2 from databricks. now we want to load the data from ADLS gen2(with delta table) to synapse table delta table.…
Developer KE
  • 71
  • 1
  • 2
  • 14
0
votes
1 answer

Synapse polybase data ingestion is not working

I have a task to convert the jobs from synapse bulk insert to synapse polybase pattern. As part of that I see that it doesn't work straight away. It is complaining about some datatypes etc as below.... where as there is no double datatypes sometimes…
user237624
  • 7
  • 2
0
votes
0 answers

SSDT is publishing everything except permissions

I'm using Visual Studio 2022(17.4.1) and I'm importing existing database to a project. With that I can publish it and everything goes well(tables, users, roles, procedures etc.) but I'm experiencing problem that the permissions aren't really being…
tomyki
  • 13
  • 3
0
votes
1 answer

ValueError: can not serialize object larger than 2G - 500 million records

I am reading a json file with 500 million records from a API and writing to blob in Azure. Tried many ways but getting the below error. I am using PySpark notebook in Azure Synapse ValueError Traceback (most recent…
Arun.K
  • 103
  • 2
  • 4
  • 21
0
votes
1 answer

Write a variable's data as it is in ADLS file

I want to write the content of a variable that is created dynamically in the program to a ADLS file. This is how I am getting the data - @dataclass class pipeline_run: id:str group_id:str run_start:str run_end:str …
Bikash
  • 21
  • 4
0
votes
1 answer

Azure Synapse Dedicated SQL Pool Parse Error

When execute the script below on Azure Synapse Dedicated SQL Pool I get the error: Parse error at line: 11, column: 1: Incorrect syntax near 'FOR'. However, I if execute the same code on Azure Synapse Serverless Pool it works fine. DECLARE …
Patterson
  • 1,927
  • 1
  • 19
  • 56
0
votes
0 answers

Ingesting partitioned BigQuery table in Azure Synapse

I am trying to ingest a BigQuery table (from GA) into Azure Synapse but the table is partitioned and every day a new partition is added to it. The table names are like…
Mahdi
  • 3,199
  • 2
  • 25
  • 35
0
votes
1 answer

Azure Synapse .NET C# Sparkpool: Fail to start interpreter

When I am working on a .NET Spark (C#) Notebook in Azure Synapse I always get the following error message: Fail to start interpreter. detail: org.apache.spark.api.dotnet.DotnetBackend. When changing the language from .NET Spark (C#) to Python or…
tomotom12
  • 46
  • 6
0
votes
1 answer

Error Trying to copy a sharepoint file to DL

Hello could someone please help me on this, I am trying to copy sharepoint files to my DL but it's not working I got this error { "errorCode": "2200", "message":…
andy
  • 47
  • 5
0
votes
1 answer

Escape characters in .csv for Azure Synapse seems to dissapear?

I have .csv file that looks like this: "ID", "Name", "Extra Info" "1", "John", "{\"Event\": \"Click\", \"Button Name\": \"Accept\"} "2", "Adam", "{\"Event\": \"Click\", \"Button Name\": \"Accept\"} I'm trying to load this file using this code in…
Mark
  • 75
  • 1
  • 7
0
votes
1 answer

How to increase DWUs on Azure Synapse Dedicated SQL Pool

I inherited an Azure Dedicated SQL Pool at my current firm. The DWU was set at 100, is there a way to increase the DWU's? It would be appear I would have to create a new Dedicated SQL Pool to increaase the DWU's Also, is there a guide showing what…
Patterson
  • 1,927
  • 1
  • 19
  • 56
0
votes
1 answer

Azure synapse linked service for Azure Function in Terraform

I'm writing a terraform script to create a azure synapse workspace. I've created a linked service for Azure Function but I'm unable to use it in pipeline, where it gives me an error of missing function key. This is what i'm using now. I'm sure the…
Ehsan
  • 3
  • 3
0
votes
0 answers

Azure Synapse serveless doesn't read delta

I've around 90 delta views on Synapse serveless, 90% of them works flawless but some of them don't. Databricks and hive shows all results correctly but on Synapse I getting error message when I try to read delta and no rows, if I write same view…
Alvaro
  • 79
  • 1
  • 6
0
votes
1 answer

Mapping data flow allows duplicate records when using UPSERT

Using Synapse pipelines and mapping data flow to process multiple daily files residing in ADLS which represent incremental inserts and updates for any given primary key column. Each daily physical file has ONLY one instance for any given primary…
Geekn
  • 2,650
  • 5
  • 40
  • 80
1 2 3
…
99 100