Stack Overflow
Stack Overflow

Questions tagged [polybase]

PolyBase enables SQL Server 2016 instance to process Transact-SQL queries that read data from Hadoop/Azure Storage. The same query can also access relational tables in your SQL Server. PolyBase enables the same query to also join the data from Hadoop and SQL Server. In SQL Server, an external table or external data source provides the connection to Hadoop.

179 questions
2
votes
0 answers

Loading huge number of small csv files into Azure Synapse using Polybase

I'm currently tasked to load about 3600 small csv files from Azure Data Lake into a single table in Azure Synapse. Each file is less than 10k bytes. I used Polybase to create an external table to point to a folder on Data lake with some wildcards…
Victor Ng
  • 71
  • 3
2
votes
1 answer

SQL Server 2019 Polybase error - cant create External Data Source to SQL Server 2019

I am trying to setup an external data source in SQL Server 2019 to another database on the same server. I am doing this to replicate the SQL Azure setup currently running in production. I tried the following to set it up I installed SQL Server 2019…
Mohamed Cassim
  • 153
  • 1
  • 10
2
votes
1 answer

Azure SQL Data Warehouse (Synapse Analytics) Polybase performances with ORC table

I generate an ORC table (compresssed w/ Snappy) with Spark (Databricks) on an Azure Storage Account (w/ ADLS Gen2 feature). This ORC represent about 12 GB of data (1.2 billions lines). This table has 32 columns. Once it's generated, I load this file…
alxsbn
  • 340
  • 2
  • 14
2
votes
1 answer

Azure SQL Data Warehouse Polybase Query to Azure Data Lake Gen 2 returns zero rows

Why does an Azure SQL Data Warehouse Polybase Query to Azure Data Lake Gen 2 return many rows for a single file source, but zero rows for the parent folder source? I created: Master Key (CREATE MASTER KEY;) Credential (CREATE DATABASE…
Andy Jones
  • 1,472
  • 9
  • 15
2
votes
1 answer

Why is External Table syntax different in SQL Server 2019 vs Azure SQL Database? `

I have to perform cross DB querying in SQL Server 2019 as well as Azure SQL. I am able to create External data source and external tables in both databases, however the syntax seem to be different. Can someone please guide me if there is one…
Shubham Garg
  • 21
  • 1
2
votes
2 answers

Polybase to connect local CSV file

I'm unable to access a local CSV file from SQL Server 2019 Polybase. This is a simple 3-columned text file. I have also created a local system DSN (from ODBC32 UI). I got the sample code from here. However, the driver in the link (cdata) is not…
RaviLobo
  • 447
  • 2
  • 10
  • 29
2
votes
0 answers

Cannot generate SSPI context when creating External Data Source in SQL 2019

I'm trying to create an External Data Source for PolyBase via ODBC connectors in SQL Server 2019, but I'm getting the error: OLE DB provider "MSOLEDBSQL" for linked server "(null)" returned message "Cannot generate SSPI context". Msg -2146893042,…
Cameron Castillo
  • 2,712
  • 10
  • 47
  • 77
2
votes
1 answer

Copy data in Azure SQL DW by using ADF(Staged copy by using Polybase dosen't work according to documentation)

Trying to load csv files in the data lake(gen2) to Azure Synapse by using Azure Data Factory. The source file has "(double quote) as an escape character. This falls outside the data limitations of directly connecting polybase to Data Lake. I setup…
sab
  • 338
  • 1
  • 3
  • 21
2
votes
2 answers

CREATE EXTERNAL TABLE AS SELECT statement failed

When creating an external table, I get the below error: Msg 105005, Level 16, State 1, Line 1 CREATE EXTERNAL TABLE AS SELECT statement failed as the path name 'wasbs://.blob.core.windows.net/folder/Reject_Directory/_rejectedrows/' could not be…
Neil P
  • 2,920
  • 5
  • 33
  • 64
2
votes
1 answer

Access Data from Azure Data Lake Store using Polybase with Azure Data Warehouse

I get a error when create external table https://exoticbaryon.anset.org/2017/06/26/access-data-from-azure-data-lake-store-using-polybase-with-azure-data-warehouse/#comment-157 CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'xxxxx' CREATE DATABASE…
Shuai
  • 21
  • 1
2
votes
0 answers

Azure PolyBase external table from binary blob data?

I have several million image blobs stored in a Azure storage container. The blobs have a predictable name ([unique key].jpg.) Examples: 1234.jpg, 230.jpg & so on. If possible, how can I create an external table using this data which can be…
jujiro
  • 190
  • 1
  • 1
  • 11
2
votes
0 answers

NewLine and Carriage Return in File failing to parse in Polybase

Recently i run into this trouble. I have exported the data from a system using data factory and saved it as txt files in blob storage. We I tread reading from Azure data warehouse using polybase, the reading failing due to new line and carriage…
Anbarasan Dhanushkodi
  • 113
  • 1
  • 8
2
votes
0 answers

Need to enable proxy settings in SQL for creating External table using Azure Polybase for Production environment

The external table creation using polybase was successful in local environment but it was unsuccessful in production where we use proxy servers for internet access. When tried to create external table in production environment I got following…
Dev
  • 31
  • 3
2
votes
1 answer

Best practices to implement incremental data load into azure data lake & azure data warehouse

I am planning to implement azure BI. I need expert advice on how to implement incremental data load using azure data lake, azure sql datawarehouse, azure data factory + poly base. In which format, we should store data in azure data lake etc. Thanks…
npatel
  • 43
  • 4
2
votes
2 answers

Polybase over Data Lake - what is the OAuth_2.0_Token_EndPoint?

I'm trying to connect my Azure SQL Data Warehouse to my Datalake, following the microsoft guide. Microsoft give the following code sample, but do not clarify what the OAuth_2.0_Token_EndPoint is and where to find it in the Azure portal. CREATE…
Neil P
  • 2,920
  • 5
  • 33
  • 64
1
2
3
…
11 12