0

connecting to sql server using connectorx and polars. everything works correctly locally and not getting any errors. however, when using azure pipelines to run code getting the following error "result = _read_sql(RuntimeError: Server requested a connection to an alternative address":

This doesn't make sense since the exact same details worked locally but does not work in pipeline?

internal files which error is caused:

File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\polars\io.py", line 1095, in read_sql tbl = cx.read_sql( File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\site-packages\connectorx_init_.py", line 257, in read_sql

#code
import polars as pl
import connectorx as cx
from datetime import timedelta

load_dotenv()


username=os.environ["UID"]
password=os.environ["PWD"]
host=os.environ["SERVER"]
port=os.environ["PORT"]
db=os.environ["DATABASE"]
table_name=os.environ["DB_TABLE"]
col_name=os.environ["DB_COL"]
    
datetime_limit= str(datetime.today() - timedelta(days=50))[:-3]
conn=(f"mssql://{username}:{password}@{host}:{port}/{db}?encrypt=true")
query=f"SELECT * from {table_name} WHERE {col_name} > '{datetime_limit}'"
print(query)
start=timer()
data=pl.read_sql(query, conn, partition_on="Partition", partition_num=10)
end = timer()

yaml file

trigger: 
  none

schedules:
- cron: "0 13 * * *"
  displayName: Daily build
  branches:
    include:
    - dev
  always: true

pool:
  vmImage: "windows-latest"

#environment var
variables:
  - group: env-group
    

steps:

  # Set the version of Python which the build agent should use
- task: UsePythonVersion@0
  displayName: Use Python 3.8
  inputs:
    versionSpec: '3.8'
    addToPath: true 

  # Run Pip installer for any prerequisite libraries
- bash: pip3 install -r requirements.txt
  displayName: Install pip package requirements
  workingDirectory: project-requirements

  # Invoke a Python script via Azure CLI in to use it's credentials
- task: AzureCLI@2
  inputs:
    azureSubscription: 'Subscription'
    scriptType: 'batch'
    scriptLocation: 'inlineScript'
    inlineScript: 'python etl.py'
    workingDirectory: wd/scripts
tommyt
  • 309
  • 5
  • 15
  • 1
    Found out the error its because of partitioning. might be a database issue rather than connectorx. partitioning works locally but not in the pipeline am guessing something to do with permissions and threads. #causes error data=pl.read_sql(query, conn, partition_on="Partition", partition_num=10) #no error data=pl.read_sql(query, conn) – tommyt Sep 22 '22 at 10:56

0 Answers0