PolyBase enables SQL Server 2016 instance to process Transact-SQL queries that read data from Hadoop/Azure Storage. The same query can also access relational tables in your SQL Server. PolyBase enables the same query to also join the data from Hadoop and SQL Server. In SQL Server, an external table or external data source provides the connection to Hadoop.
Questions tagged [polybase]
179 questions
1
vote
1 answer
Polybase from Parquet error: Cannot cast Java.lang.Double to
Loading Azure Data Warehouse via Polybase, I'm reading Parquet files that are on Azure Blob.
First I created an external table in SQL to point to the Parquet file, and then loading with CTAS. No matter what data type I use in SQL, it gives me this…
Josh D
- 794
- 4
- 14
- 31
1
vote
1 answer
Update changes in Azure SQL Data Warehouse using polybase
I want help regarding Azure SQL Data Warehouse, I'm using Polybase to ELT data from Azure Data Lake Storage Gen2 to Azure SQL DW. When we load data first time into DW no issues. But when we load data again/incremental load how do we upsert data?…
Soni007
- 103
- 2
- 13
1
vote
2 answers
Polybase to Azure Data Lake Gen 2 Failing
I'm trying to connect from Azure SQLDW via Polybase to a Parquet file in Data Lake Gen 2. This is my code:
CREATE DATABASE SCOPED CREDENTIAL DSC_ServicePrincipal
WITH IDENTITY =…
simon_dmorias
- 2,343
- 3
- 19
- 33
1
vote
2 answers
CREATE EXTERNAL DATA SOURCE from SS2019 CTP2.2 not working
So ... I have 2 SQL Server 2019 instances (CTP2.2) and I have one installed with Polybase in single node config (reference this as SS-A). I have created MASTER KEY in the master of SS-A, and created a DATABASE SCOPED CREDENTIAL in a database on…
Scott Simmons
- 21
- 4
1
vote
2 answers
Polybase: Can't connect to Azure Blob from SQL Server
I am trying out the new Polybase-Feature in SQL-Server by connecting to a CSV. However I do not manage to connect to the Azure Blob Storage:
CREATE EXTERNAL DATA SOURCE AzureBlob WITH (
TYPE = HADOOP,
LOCATION =…
5th
- 2,097
- 3
- 22
- 41
1
vote
1 answer
Create View in Polybase
I am trying to load multiple files via polybase and its throwing me an error while creating view.
select * from dbo.test
fname lname
a.csv null
b.csv null
I am able to successfully create data source and external table without any issue.
Code…
Jo Joseph
- 59
- 10
1
vote
1 answer
Incremental Loads in Polybase
I have created an EXTERNAL TABLE for PolyBase to load data from BLOB storage to Azure SQL Data Warehouse. I have below questions regarding the file load.
1) If there is incremental growth of data then how will the PolyBase work. If there is daily…
Jo Joseph
- 59
- 10
1
vote
1 answer
Creating External Table in Azure SQL DW from Azure BLOB Storage
I am creating a data source in Azure SQL DW in order to create external table using BLOB storage data but I am getting error while creating data source. I am new to this polybase environment so, please share your thought.
I am trying to call the…
Jo Joseph
- 59
- 10
1
vote
1 answer
Configure SQL PolyBase to access external data in MongoDB - problem
Im trying to connect SQL server using Polybase with MongoDB but Im failing to create the external data source for that. Please help ...
Im using the following code:
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'mypassword';
CREATE DATABASE SCOPED…
Amir
- 11
- 2
1
vote
0 answers
create external table in sql server
I created an external table in toad with the following code and it is working.
/*External table query*/
--AUTHID CURRENT_USER is
CREATE DIRECTORY ext1 AS 'E:\outer source'
create table countries_external3
(country_id varchar2(5),
…
zain ul abidin
- 197
- 2
- 13
1
vote
2 answers
Azure SQL DW External File Format treat empty strings as NULL using Polybase
I'm using external tables to load data from csv stored in a blob to a table in Azure SQL Data Warehouse. The csv uses a string delimiter (double quote), empty strings are represented as 2 double quotes ("").
I want the empty columns to be treated…
Ward
- 39
- 7
1
vote
1 answer
Alter Table in Polybase
When I created an External Table and point to a Hadoop HDF, I forgot to add a Reject in the script. Getting error when Alter Table with Reject limit to 100,
ALTER TABLE ad_doc_actg REJECT LIMIT 100;
Msg 102, Level 15, State 1, Line 1
Incorrect…
Jodie tan
- 11
- 2
1
vote
1 answer
Connection refused while connecting from polybase to hadoop
While trying to create external table from sql server 2017 to Hadoop in Ubuntu 16.04 it throws the below error
Msg 105019, Level 16, State 1, Line 1
EXTERNAL TABLE access failed due to internal error: 'Java exception raised on call to…
Mani
- 344
- 4
- 15
1
vote
1 answer
Loading only latest files data to Azure SQL Datawarehouse
Step#1: WE are supposed to copy the CSV Files from On-Premise File Server to Azure Blob Storage (say - 'Staging' Container in Blob Storage).
Step#2: Applying Polybase, we will load these files data to Azure SQL Datawarehouse.
We are maintaining the…
Koushik
- 11
- 3
1
vote
1 answer
Polybase: Loading data from external table to root blob folder for specific date/time range
I have data in a blob storage account, which is organized in the following structure:
/logs/YYYY/mm/dd/HH
Let's say there is a few years of data and I want to pull data from a specific time period (or time periods) into an Azure Data Warehouse…
Shane
- 187
- 2
- 13