Stack Overflow
Stack Overflow

Questions tagged [azure-synapse]

Questions about Azure Synapse Analytics, bringing together enterprise data warehousing and big data analytics.

Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless or provisioned resources—at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs.

2525 questions
2
votes
2 answers

Query on delta lake from external tool using Azure Synapse serverless SQL

We are setting up a delta lake within Azure Synapse Querying the delta lake works within Synapse. For that purpose, we set up a view CREATE VIEW V_Test AS SELECT * FROM OPENROWSET( BULK…
Kermit754
  • 343
  • 5
  • 14
2
votes
1 answer

Error in declare variable with SQL in Azure Synapse

I recently had a first contact with the Azure Synapse Analytics environment, and when implementing some SQL queries, I came across a problem when trying to declare a variable to run a query with WHILE. I am sending below the error messages I am…
Everton Santos
  • 23
  • 3
2
votes
0 answers

external table with partition elimination

I am trying to understand how to create an external table that supports partition elimination. I can create a view with a column derived using the filepath function, but that can't be used by Spark. I can create an external table using create…
Andrew
  • 8,445
  • 3
  • 28
  • 46
2
votes
2 answers

How can I make tables from serverless Azure Synapse views?

I have a view in an on-demand (or "serverless") sql pool. My goal is to over data from the serverless views and materialize them as tables in the dedicated pool. Is this possible?
Agneum
  • 727
  • 7
  • 23
2
votes
2 answers

How to connect to on-premise SQL Server from Azure Synapse

We are currently creating a new Azure Synapse Analytics environment to house our data. For this, we need to connect from Synapse to our on-premise SQL Server databases. These databases are not reachable from the open internet, they are only…
JAD
  • 2,035
  • 4
  • 21
  • 35
2
votes
1 answer

Is possible merge tables between databricks and synapse?

I was working with streaming in delta tables with foreachbatch. spark.readStream.format("delta") \ .option("readChangeFeed", "true").option("startingVersion","latest") \ .load("dbfs:/mnt/cg/file_to_up/table2") \ …
BryC
  • 89
  • 6
2
votes
1 answer

Unable to Install Python Wheel Package to Azure Synapse Apache Spark Pool

I am using the following guide to install Python package https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-manage-python-packages#install-wheel-files I have uploaded the package as seen in the image However, the option to…
Patterson
  • 1,927
  • 1
  • 19
  • 56
2
votes
1 answer

Azure Synapse Serverless CETAS error "External table location is not valid"

I'm using Synapse Serverless Pool and get the following error trying to use CETAS Msg 15860, Level 16, State 5, Line 3 External table location path is not valid. Location provided: 'https://accountName.blob.core.windows.net/ontainerName/test/' My…
CodeMonkey
  • 3,418
  • 4
  • 30
  • 53
2
votes
1 answer

Azure Synapse - How to stop an Apache Spark application / notebook?

When I run (in debug mode) a Spark notebook in Azure Synapse Analytics, it doesn't seem to shutdown as expected. In the last cell I call: mssparkutils.notebook.exit("exiting notebook") But then when I fire off another notebook (again in debug mode,…
Dudeman3000
  • 551
  • 8
  • 21
2
votes
1 answer

AttributeError: 'DataFrame' object has no attribute 'randomSplit'

I am trying to split my data into train and test sets. The data is a Koalas dataframe. However, when I run the below code I am getting the error: AttributeError: 'DataFrame' object has no attribute 'randomSplit' Please find below the code I am…
yocodefreak
  • 43
  • 7
2
votes
1 answer

How to implement if else and else if in ADF dynamic expression

How can I achieve below conditions in ADF dynamic expression: if variable=a then A if variable=b then B else C I'm able to achieve only true or false condition in IF but here I want to verify multiple conditions. No switch case function available…
Krishna Murthy
  • 55
  • 1
  • 2
  • 10
2
votes
0 answers

Making parameters work in Azure .NET C# Spark Notebook

I tried to pass a string parameter into a Spark NoteBook written entirely in .NET Spark C# No matter what I tried it did not work. What finally did work was to Define the notebook as PySharp Define the parameter - PySharp Put the parameter value…
bmukes
  • 119
  • 2
  • 9
2
votes
0 answers

synapseml lightgbm model doesn't converge on Dataproc

I'm trying to deploy a gbdt model with synapseml lightgbm[0.9.5] on google dataproc[2.0-debian10]. I use Spark StringIndexer to index string categorical columns and assemble all columns as a vector. With categorical features setting, I found the…
jiamei shen
  • 21
  • 2
2
votes
2 answers

How to return integer value from notebook in adf pipeline

I have a usecase where I need to return an integer as output from a synapse notebook in pipeline and pass this output in next stage of my pipeline. Currently mssparkutils.notebook.exit() takes only string values. Is there any utility methods…
boom_clap
  • 129
  • 1
  • 12
2
votes
1 answer

Azure synapse deployment failing

I am trying to deploy SQL files to an Azure Synapse Analytics dedicated SQL pools using PowerShell script in an Azure Devops pipeline. I have a folder with SQL files and after defining array of files I am trying to run foreach loop for array and…
MITI SHAH
  • 33
  • 3
1 2 3
…
99 100