Stack Overflow
Stack Overflow

Questions tagged [azure-synapse]

Questions about Azure Synapse Analytics, bringing together enterprise data warehousing and big data analytics.

Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless or provisioned resources—at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs.

2525 questions
0
votes
1 answer

Azure Synapse Pipeline Date Expression - Last Monday

I have the following azure function that is supposed to retrieve the date of the previous Monday. It works fine except for if the current date is a monday. I need the function to still retrieve the previous monday date if it is Monday or Tuesday.…
mullinsjo
  • 5
  • 2
0
votes
0 answers

_corrupt_record reading a JSON list in SPARK

I am reading JSON from pages and appending to list using a loop. When tried to read to spark its throwing _corrupt_record error. I have gone through couple of posts regarding this but none of the solution worked. Any suggestions on how to fix…
Arun.K
  • 103
  • 2
  • 4
  • 21
0
votes
1 answer

snowflake schema with number how to use it in Azure data factory to copy data from Synapse to Snowflake

I am trying to copy data from Synapse and load into Snowflake, for this i am using Azure Data Factory and control table having source and target fields names My problem here is the snowflake schema name starts with number for example…
user15488631
  • 85
  • 2
  • 10
0
votes
1 answer

How do I create a DATABASE SCOPED CREDENTIAL in Synapse Serverless?

I have a Serverless SQL pool set up in Azure Synapse Analytics, and I am trying to run this query: CREATE DATABASE SCOPED CREDENTIAL myCredential WITH IDENTITY = 'test', SECRET = 'test2'; When I run the query I get this…
Agneum
  • 727
  • 7
  • 23
0
votes
0 answers

Apache Spark in Azure Synapse Analytics - HTTP request in notebook

I use a Notebook in Synapse where I run my Python code. I would like to make an API request from this Notebook to Microsoft Purview to send the entities. I added the pyapacheatlas library to spark. On my local computer, this code works fine in…
DieX
  • 1
  • 1
0
votes
1 answer

Azure SQL Serverless inbuilt Pool Column/Field Limitations

We have created a SQL Database from our Azure SQL Serverless Pool. We have a table that has over 450 fields. Whenever we try to extract the table with all the fields the query times out and produces the following error: Msg 15884, Level 16, State 1,…
Patterson
  • 1,927
  • 1
  • 19
  • 56
0
votes
0 answers

Cannot write non-nullable data to Azure Synapse - PySpark

The Problem: I am unable to write any dataframe that contains a non-nullable column to Azure Synapse's dedicated SQL pool. Problem details: I have a DataFrame with the following…
SheerKahn
  • 301
  • 2
  • 14
0
votes
2 answers

Azure Synapse Web Activity 'GetFolderByServerRelativeUrl' Unauthorized Access

I'm trying to get a list of files from my sharepoint site in Azure Synapse Pipeline Activity. I'm using 2 web activities, the first one grabs the access token and the second one grabs the relative…
mohamadmaarouf_
  • 27
  • 6
0
votes
1 answer

Need to get proper URL for payloads for Airflow connected with Azure

I have four files main.py, jobs.zip, libs.zip & params.yaml and these I have stored on Azure Storage Account Container. Now I have this code which is making a payload and will try to run a spark job using that payload. And that payload will be…
member2
  • 36
  • 6
0
votes
1 answer

How to insert data into Azure Synapse Database with SQLAlchemy

I'm trying to insert some rows into a Synapse Database using SQLAlchemy. However I'm getting this error when doing db.session.commit(): ERROR - 2023-02-01 15:24:28,200 - Traceback (most recent call last): ERROR - 2023-02-01 15:24:28,209 - File…
Eva Sanchez-Guerrero Ayala
  • 9
  • 2
0
votes
1 answer

User should only be able to run SQL queries in Synapse workspace

We have a special scenario in Azure Synapse Workspace where we want the user to only be able to query the data that in the dedicated SQL pool which is connected to the Synapse workspace. We don't want them to create pipelines, change Synapse…
xmlapi
  • 61
  • 1
0
votes
1 answer

Install Python pycryptodome libraries for Apache Spark in Azure Synapse

I am tying to install pycryptodome to azure synapse notebook. PFB details. scenario - I have created a notebook and Apache spark pool in azure synapse. I used the Below command to list the packages installed on pool. I don't see my required packages…
sac
  • 175
  • 2
  • 14
0
votes
1 answer

Azure Synapse recursive CTE alternative for flatten hierarchy

I am facing following challenge: I would like to flatten a parent child hierarchy in a way that I have per row the child + any parent of the upper levels Source data Child Parent A B B C D E X Y Y Z Whished result Child Any…
Sanchez3005
  • 13
  • 2
0
votes
1 answer

How to increase performance on Azure inbuilt SQL Serverless Pool in Synapse

We are currently extracting multiple tables from Azure SQL Servereless pool in Synapse. Unlike a regular Azure SQL Database it is very easy to increase the performance from Basic all the way through to Premium or Business continuity. Can someone let…
Patterson
  • 1,927
  • 1
  • 19
  • 56
0
votes
0 answers

Azure Synapse Notebook: Data Skew Issue

I need to transform and union 20ish parquet files (with different schema) into one big fat dataframe in Azure Synapse notebook. I didn't run into any issues when querying each individual dataframe. However, after they are unioned into one table,…
Mich
  • 19
  • 1
1 2 3
…
99 100