Stack Overflow
Stack Overflow

Questions tagged [polybase]

PolyBase enables SQL Server 2016 instance to process Transact-SQL queries that read data from Hadoop/Azure Storage. The same query can also access relational tables in your SQL Server. PolyBase enables the same query to also join the data from Hadoop and SQL Server. In SQL Server, an external table or external data source provides the connection to Hadoop.

179 questions
2
votes
1 answer

Azure PolyBase external table from StorageTable

Can somebody explain to me why I can polybase reference an azure BlobStorage as external table but not an azure StorageTable? Did I missed it some where in the documentary? Its some kind of ironic for me if I don't missed it and it is really not…
Steffen Mangold
  • 1,184
  • 1
  • 20
  • 39
2
votes
1 answer

SQL 2016 PolyBase Compute Pushdown to Hadoop HDI that uses WASBS aka Azure Blob

We have an Azure Hadoop HDI system where most of the files are stored in an Azure Storage Account Blob. Accessing the files from Hadoop requires the WASBS:// file system type. I want to configure SQL 2016 Polybase to pushdown compute to the HDI…
Richard Blevins
  • 115
  • 1
  • 9
2
votes
1 answer

Why is Polybase slow for large compressed files that span 1 billion records?

What would cause Polybase performance to degrade when querying larger datasets in order to insert records into Azure Data Warehouse from Blob storage? For example, a few thousand compressed (.gz) CSV files with headers partitioned by a few hours per…
Fastidious
  • 1,249
  • 4
  • 25
  • 43
1
vote
1 answer

CETAS in SQL Server 2022

I have followed the below steps in SQL Server 2022: Step 1: Create a master key Step 2: CREATE DATABASE SCOPED CREDENTIAL [BlobSAS] WITH IDENTITY = 'SHARED ACCESS SIGNATURE', SECRET = '?sv=2021-06-08&ss=bfqt&srt=c<<>>'; Step 3: CREATE…
Nandan
  • 3,939
  • 2
  • 8
  • 21
1
vote
1 answer

Does the Polybase shard range constitute data knowledge?

In the Polybase database, does the client encrypt and then determine in which shard a record belongs? I was trying to understand how do you know the correct location of the data?
Srini
  • 31
  • 1
1
vote
1 answer

Copy data from Parquet file to a table in Azure Synapse by using Polybase T-SQL failing when using a partition column

I tried to copy a parquet file to a table in Azure Synapse by using Polybase T-SQL. Here is an example: data = [["Jean", 15, "Tennis"], ["Jane", 20, "Yoga"], ["Linda", 35, "Yoga"], ["Linda", 35, "Tennis"]] columns = ["Name", "Age", "Sport"] df =…
Quynh-Mai Chu
  • 165
  • 1
  • 9
1
vote
1 answer

How to exceed limit 256 length on Databricks

I am working with databricks. I created a function where I use a try and catch to catch any error messages. Unfortunately with errors with a length larger than 256 characters I cannot write to my target table. def writeIntoSynapseLogTable(df,mode): …
HABLOH
  • 460
  • 2
  • 12
1
vote
1 answer

Keep staging Blobs in Data Lake after copy activity

I've been copying date into Synapse using the copy data functionality in Azure data factory (polybase), with staging enabled to stage the data in our azure data lake. However, once the copy into Synapse is complete the staging files in our azure…
Ben
  • 11
  • 2
1
vote
1 answer

Create external table using Polybase on-premise error

I'm trying to create an external table in SQL Server 2019 on premise ( Polybase has been installed and all services are up and running [Instance and 2 services for polybase] , Hadoop configuration = 7). I want to feed the external table with…
Samayoa
  • 185
  • 1
  • 2
  • 12
1
vote
1 answer

Polybase configuration for SQL Server 2016

According to Microsoft's documentation after installing PolyBase I should run the following code. exec sp_configure @configname = 'polybase enabled', @configvalue = 1; RECONFIGURE; When I try to execute this command it displays an error. How can I…
lukaszFD
  • 99
  • 6
1
vote
1 answer

Unable to query AS400 over ODBC with PolyBase on SQL 2019 Linux

I would like to create external tables on a SQL Server 2019 instance running in an Ubuntu container querying an DB2 database on an AS400. I'm using the ibm iaccess 1.1.0.15 amd64 ODBC driver. When I try to create the external table, I get back the…
Adrian
  • 222
  • 2
  • 11
1
vote
0 answers

SQL Server Polybase - Automatic statistics queries problem

I am trying to create an external table for a table on ClickHouse using ODBC connection. Let's assume the external table code looks like this: CREATE EXTERNAL TABLE [dbo].[externalTable1] ( [col1] [int] NULL, [col2] [int] NULL, …
Sabir
  • 11
  • 1
1
vote
1 answer

Polybase - Error converting data type VARCHAR to DATETIME

I'm trying to create an external table to a CSV file stored in Azure Storage. The CSV data looks like this :- Date Rail Period Calendar Year Calendar Month Calendar Month Name Fiscal Year Fiscal Period Weekday Weekday…
Richard Cooper
  • 23
  • 4
1
vote
0 answers

Bad Connection: Tableau could not connect to the data source. Error Code: A7AED4D2

I am getting this error when I try to drag one of the measure/dimension on to the visual pane. The tableau server is connected with the SSAS database. I have installed the latest ole db drivers for the server. But the Polybase seems not to start.…
milap
  • 11
  • 4
1
vote
1 answer

Cannot ADF polybase file from Azure Storage to Synapse Pool (SQL datawarehouse)

How can I get ADF copy polybase to to simply copy one file from Azure storage to Synapse pool (SQL datawarehouse) I consistently get this error: Operation on target Storage to Synapse failed:…
Sauron
  • 6,399
  • 14
  • 71
  • 136
1 2
3
…
11 12