Stack Overflow
Stack Overflow

Questions tagged [azure-synapse]

Questions about Azure Synapse Analytics, bringing together enterprise data warehousing and big data analytics.

Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless or provisioned resources—at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs.

2525 questions
0
votes
1 answer

How Openrowset works internally when the data is loaded

I am going through the azure documentation and come across the following phrase OPENROWSET function in Synapse SQL reads the content of the file(s) from a data source. The data source is an Azure storage account and it can be explicitly referenced…
Nag
  • 1,818
  • 3
  • 24
  • 41
0
votes
2 answers

PySpark - Read CSV and ignore file header (not using pandas)

I have a problem that I hope you can help me with. The text file that looks like this: Report Name : column1,column2,column3 this is row 1,this is row 2, this is row 3 I am leveraging Synapse Notebooks to try to read this file into a dataframe. …
data_engineer_eric
  • 1
  • 1
0
votes
1 answer

extract json column using spark.sql in azure synapse notebook

I have a parquet file as source and I loaded that parquet file using PySpark notebook as shown below: df_Employee = spark.read.parquet() df_Employee .createOrReplaceTempView("employee_data") here's how the table looks like Employee…
Swasti
  • 187
  • 1
  • 5
  • 20
0
votes
1 answer

My datasource is Azure Synapse. Do I still need a gateway to be set up in this case?

Our new data source for Power BI reports is Azure Synapse. Do we still need a gateway to be set up in this case? . I think this is the reason that we can’t refresh the data in Power BI. Thanks! I see after clicking on ‘edit credentials’. I tried…
jmartel
  • 3
  • 2
0
votes
1 answer

Group by not detecting duplicates but there are dupes. Strange SQL Server - Azure Synapse database dedicated SQL pool

I have encountered a strange (until I understand the logical reason) behaviour of group by in a SQL Server database. There are many duplicates in a table, for which when I query I get duplicate rows but when I try to find all dupes using group by or…
Aditya Rathi
  • 59
  • 1
  • 6
0
votes
2 answers

Azure: How to provide limited Access Level to a Container in a Storage Account?

Me and my team are using Azure Synapse Analytics to ingest data from a REST API to a Azure Data Lake Storage Gen2, in order to create views automatically. The only way we could manage to do this in our Workspace was by previously changing the Public…
Mateo Estrada
  • 15
  • 3
0
votes
2 answers

Can a stored procedure with input variables display an output table when parameter is selected?

Edit: I think I need to add OUTPUT in my code, but where and how do I change the automated Exec script to return the table? I have created a stored procedure with multiple input parameters, that depending on the input will produce a table output…
dszakris
  • 11
  • 2
0
votes
1 answer

Trouble with Copy Data Mapping: cannot convert data from REST API Get Request

My goal is to get data through a GET Request and, using Copy Data, introduce it in parquet file format in the Data Lake. My pipeline currently works, but I wish not to map manually all the variables and their respective types. I would like to use…
Mateo Estrada
  • 15
  • 3
0
votes
1 answer

Setting up Azure Synapse Analytics Linked Service to ODBC linked server

A vendor provided access to their server via ODBC connector that created a SYSTEM DSN on my local machine. I would like to replicate the data in Azure Synapse data lake for analysis. The Azure Synapse Linked Service configuration does not allow me…
Alen Giliana
  • 2,144
  • 3
  • 17
  • 30
0
votes
0 answers

Which is the most recommended way to pursue integration and system tests por Azure Synapse pipelines?

Good afternoon! Me and my team wish to design and pursue unit testing (integration and system too) on our Azure Synapse pipeline. It works properly, fortunately, and it consists of a Copy Data and a Stored Procedure that creates a view. However, we…
Mateo Estrada
  • 15
  • 3
0
votes
1 answer

spark.read.json() taking extremely long to load data

What I've Tried I have JSON data which comes from an API. I saved all the data into a single directory. Now I am trying to load this data into a spark dataframe, so I can do ETL on it. The API returned fragmented data (per division). Moreover, some…
Psychotechnopath
  • 2,471
  • 5
  • 26
  • 47
0
votes
0 answers

Synapse Apache .NET Spark (C#) parallel execution

I'm using .NET Spark in a Synapse notebook to transform data. The source data consists of multiple parquet files within subfolders in a…
Krumelur
  • 32,180
  • 27
  • 124
  • 263
0
votes
1 answer

Column does not allow DBNull.Value - Synapse

I am running a pipeline that calls a stored procedure and sinks the table to another dedicated pool. I made sure that in the source table I have no NULL VALUES in intercompany table (I have named a field in sink table as Intercompany too), so I'm…
Greencolor
  • 501
  • 1
  • 5
  • 16
0
votes
1 answer

Is there any way to execute the SQL query in Azure Synapse Analystics without creating Pipelines Programatically

Our data is stored in the Blob storage and we are using Azure Synape Analytics to query the data. This works fine with the Azure UI Interface. We are trying to do this programtically via REST APIs. Found this link which discuss on the REST support:…
SNEHAL DANGE
  • 1
0
votes
1 answer

Synapse Notebook Rest API call to Execute a notebook with Parameters

I've created a notebook with one parameter and i'm successfully exceuted the notebook by passing the parameter through notebook activity in pipeline and i'm able to successfully run the notebook without parameters through get but now i'm trying to…
david01
  • 13
  • 4
1 2 3
…
99 100