Stack Overflow
Stack Overflow

Questions tagged [azure-synapse]

Questions about Azure Synapse Analytics, bringing together enterprise data warehousing and big data analytics.

Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless or provisioned resources—at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs.

2525 questions
0
votes
1 answer

Receiving `Login timeout expired` when connecting to a serverless Azure Synapse SQL pool using `ActiveDirectoryPassword` authentication and pyodbc

I'm trying to run Azure Synapse SQL queries on a serverless SQL pool from Python using pyodbc. I'm trying to authenticate with an Azure Active Directory user. Here is my code: import pyodbc driver = '{ODBC Driver 18 for SQL Server}' server =…
alexlod
  • 1,363
  • 2
  • 10
  • 18
0
votes
1 answer

How to dynamically copy multiple datasets from a Google BigQuery project using Azure Synapse Analytics

Is it possible to dynamically copy all datasets from a BigQuery Project to Azure Synapse Analytics, then dynamically copy all tables within each dataset? I know we can dynamically copy all tables within a BigQuery dataset reference to this answered…
dwssc2023
  • 21
  • 6
0
votes
1 answer

How to make Copy Data work faster and have better performance (Azure Synapse)

A bit of context: my Azure Synapse pipeline makes a GET Request to a REST API in order to import data to the Data Lake (ADLSGen2) in parquet file format. I am looking forward to requesting data to the API on an hourly basis in order to get…
Mateo Estrada
  • 15
  • 3
0
votes
0 answers

Synapse Workspace Go Live vs Repository

We have a Synapse Workspace with pipelines and it is connected to our Github Repository. It is not in "Live Mode". Can someone explain to what "Live Mode"? Also, I'm assuming that any changes (ex: to Pipelines) need to be checked in by clicking the…
xmlapi
  • 61
  • 1
0
votes
0 answers

Export html code or string from Azure Synapse to file

I have a PySpark notebook running in Azure Synapse, which creates an interactive visualisation of the data. I want to make this visualisation available to others, but can't manage to export the visualisation as html code. There are manual options to…
IDWR
  • 1
  • 2
0
votes
1 answer

Azure Synapse - Github personal access token is invalid

I have linked a github repo to an Azure synapse account, and all the resources, pipelines, data flows, etc are successfully committed in ARM files in the repo However, when another developer tries to access this synapse workspace he gets the…
jamiemax
  • 179
  • 13
0
votes
1 answer

Unable to Scan Azure SQL Database with Azure Purview

I'm not sure why I'm unable to scan a particular Azure SQL Database with Azure Purview. This is puzzling as I'm able to scan other Azure SQL Databases From the image I have added a new database 'encrichmentdb'. I can scan all of the existing…
Patterson
  • 1,927
  • 1
  • 19
  • 56
0
votes
1 answer

What join columns should I take for ACDOCA and JVSO1 SAP tables?

I have fetched two SAP ERP tables into Azure Synapse: ACDOCA and JVSO1. Now I need to join these two tables. And the column EBELN is required to be in join condition, also both tables have around 40% of EBELN empty. Because of these empty values,…
Robin
  • 85
  • 2
  • 12
0
votes
1 answer

MS spark utilities - file move error using msspark.fs.mv

Am in Synapse notebook, using pyspark to move file using msspark.fs.mv(src, dest, True) Link to ms doc:…
Sreedhar
  • 29,307
  • 34
  • 118
  • 188
0
votes
1 answer

Cloud Computing for Monte Carlo Simulation

I run monte carlo simulation that takes 30 hours for 1000 sims using my desktop. Has anyone used azure, databricks, or any other cloud tools to significantly improve speed/runtime? I'm thinking parrallelization but am struggling to find a library or…
user21067459
  • 1
0
votes
1 answer

ForEach activity to loop through an SQL parameters table?

I'm working on an ETL pipeline in Azure Synapse. In the previous version I used an Array set as a parameter of the pipeline and it contained JSON objects. For example: [{"source":{"table":"Address"},"destination…
Mons
  • 1
0
votes
1 answer

Trying to save an xml string output to an xml file in adls using Azure Synapse Notebooks Pyspark

I am calling an api which sends back an xml string as its response. I am trying to take that xml string and save it as an xml file in ADLS using pyspark in Azure Synapse Notebooks. From there I am then trying to read that xml file and convert it to…
CMc
  • 1
  • 1
0
votes
0 answers

Need help to write a pyarrow table as parquet file in ADLS Gen2 account

I am struggling to write a pyarrow table as parquet file to ADLS Gen2 storage container. I m working in Azure Synapse Analytics using notebook. Here is what I am able to do: Mount ADLS Gen2 account to access files . Spark uses unique syntax to…
Swati Vishwanathan
  • 33
  • 6
0
votes
1 answer

Azure Synapse Pipeline copy data from the BigQuery, where the source schema is hierarchical with nested columns

Please help me with copying data from Google BigQuery to Azure Data Lake Storage Gen2 with Serverless SQL Pool. I am using Azure Synapse's Copy data pipeline. The issue is I cannot figure out how to handle source table from the BigQuery with…
dwssc2023
  • 21
  • 6
0
votes
0 answers

Slow performance writing from pyspark dataframe to Azure Synapse pool

I am writing data from a Spark dataframe in an Azure Databricks notebook into a dedicated Synapse pool. The problem is this takes an extremely long time given the small size of the data involved. Read performance is fine, this syntax will happily…
cddt
  • 539
  • 5
  • 14
1 2 3
…
99 100