Stack Overflow
Stack Overflow

Questions tagged [pandas-to-sql]

Pandas DataFrame method that writes the object's records to a SQL database. Be sure to also include the [pandas] tag.

Pandas DataFrame method to_sql can be used to write its records to a SQL database.

The documentation:

144 questions
0
votes
0 answers

Unable to send dataframe to oracle. (Export dataframe to oracle)

Unable to send data from pandas (Dataframe) to oracle database. engine = create_engine('oracle+cx_oracle://username:password@localhost') dataset.to_sql( name='table_name', con=engine, …
user13510399
0
votes
0 answers

Sqlalchemy Teradata ODBC 17.00 to_sql QVCI error CAST CLOB type to DATE

I am trying to use pandas to_sql to an existing table in Teradata database but I got this QVCI error. After some research online, I upgraded my ODBC driver to 17.00 but I still get the same error. Existing Table: I am using sqlalchemy_teradata Here…
lvv
  • 41
  • 6
0
votes
1 answer

how to create to_sql create_engine

I'm trying to add data frame information to PostgreSQL's table. But I don't know how to create an engine. It's very likely related to connection to SQL server, but I could not figure out how to create it. The example I saw on the to_sql document…
Dong-gyun Kim
  • 411
  • 5
  • 23
0
votes
0 answers

Python/SQL loading Error: Exceeded the memory limit of 20 MB per session for prepared statements

I am using Pandas to_sql to load a CSV with just about 6k rows of data. I have a very large Azure Synapse DW with more than enough capacity but i am having a problem that it will load about 1.5k rows and then fail with this out of memory…
m o
  • 21
  • 1
  • 4
0
votes
1 answer

Pandas to_sql ignoring dtype when column contains null values

First SO Question. I hope this is descriptive enough. Pandas 0.25, Oracle 11g I have a dataframe read from a csv. It contains a mix of numeric, string and date data. I force data types within the dataframe using .astype(str), .astype(int) and…
DaveH
  • 31
  • 6
0
votes
0 answers

%b requires a bytes-like object, or an object that implements __bytes__, not 'str' when using to_sql on dataframe

When using this basic read_csv and to_sql script I get the following error code. Can somebody please explain what my system is doing wrong. I used to work but after a while I got these error messages. I have also looked at my MySQL database but it…
Boaz
  • 21
  • 3
0
votes
1 answer

How Append a table from pandas data frame without truncating old data

I have a table rate_card as shown below which I have inserted from data-frame using df5.to_sql('rate_card', cnx, if_exists='replace', schema='asia', index = False,method='multi') Current data inside table ORG_CNTY DEST_CNTY KG Cost …
aeapen
  • 871
  • 1
  • 14
  • 28
0
votes
2 answers

How to speed up pandas to_sql

I am trying to upload data to a MS Azure Sql database using pandas to_sql and it takes very long. I often have to run it before I go to bed and wake up in the morning and it is done but has taken several hours and if there is an error that comes up…
David Eccles
  • 79
  • 6
0
votes
1 answer

Large Dask/Pandas DataFrame (27M rows x 52 columns) .to_csv or .to_sql MemoryError

Long story short, I've been struggling with memory issues (as well as time to execute) dask / pandas to write a large dataframe to_sql or to_csv if anyone can point me in the right direction (more detail below)? I have appended two large .csv files…
David Erickson
  • 16,433
  • 2
  • 19
  • 35
0
votes
0 answers

Populate database with foreign key relationship in Django (Wagtail)

once a day I need to remove all data from 2 tables and upload new data. All goes well with "product" table, but I'm not able to populate "pricelist" table because there is foreign key relationship on the table. model.py: class…
Fortelll
  • 15
  • 6
0
votes
1 answer

Pandas to_sql is rounding datetime column

I'm uploading all the records in a data frame to a SQL Server table but to_sql seems to be rounding datetime values. ie: assignee created_date updated_date my_name 2019-09-16 14:17:23 2019-11-26 14:48:39.261 Both dates are set as…
briba
  • 2,857
  • 2
  • 31
  • 59
0
votes
0 answers

pandas to_sql cannot replace table if domain has brackets [ ]

I'm saving a table into SQL server and need it to replace existing table of the same name. df1.to_sql('customer', schema = r'Marketing\xyz', con = engine, index = False, if_exists = 'replace') worked but adding brackets at the schema will break…
YJZ
  • 3,934
  • 11
  • 43
  • 67
0
votes
3 answers

sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42S22', "[42S22] [Microsoft].....SQL Server]Invalid column name '0'

I am trying to update existing SQL-Data base by using to_sql() Function..... When I am trying to run the code I find this error Traceback (most recent call last): File…
Mahmoud Al-Haroon
  • 2,239
  • 7
  • 36
  • 72
0
votes
0 answers

Pandas to_sql gives ORA-03106: fatal two-task communication protocol error

I try to write pandas dataframe to Oracle and with to_sql method, sometimes writes successfully and sometimes gives "(cx_Oracle.DatabaseError) ORA-03106: fatal two-task communication protocol error" with same dataframe. I checked most of the oracle…
emreb
  • 1
0
votes
0 answers

How to create local database connection using MySQL Connector/Python + SQLAlchemy

I am trying to create a locally DataBase using SQLAlchemy and MySQL Connector/Python .... As I have a huge data I want to work with.... And here's my Connection looks like.... # connect db engine = sa.create_engine('mysql+mysqlconnector://' +…
Mahmoud Al-Haroon
  • 2,239
  • 7
  • 36
  • 72
1 2 3
…
9 10