Stack Overflow
Stack Overflow

Questions tagged [pandas-to-sql]

Pandas DataFrame method that writes the object's records to a SQL database. Be sure to also include the [pandas] tag.

Pandas DataFrame method to_sql can be used to write its records to a SQL database.

The documentation:

144 questions
0
votes
0 answers

Invalid Parameter Type (numpy.int64) using pandas df.to_sql with sqlalchemy and pyodbc with mssql

I have a script where I need to append the contents of a dataframe to a SQL database table I created. I need to do this many times to several tables with several dataframes as sources. I am using Pandas with a sqlalchemy engine on a pyodbc…
John K
  • 3
  • 4
0
votes
1 answer

Python pandas how many rows insert into DB

I use the TO_SQL function from pandas to insert data from DF into a table in Oracle. I would like to check after completing the action the amount of records append to the table. can it be done and how? Thanks
Amir
  • 15
  • 7
0
votes
2 answers

Make pandas.to_sql to create tables with lowercase table name in SAP HANA database

Simple, i just want to create lowercase tables with pandas.to_sql. Tried the following: df.to_sql(name=hdb_table_name.lower(), schema="SCHEMA_NAME", con=hdb_connection, index=True, if_exists='replace') Only when hdb_table_name includes whitespaces,…
sony
  • 35
  • 5
0
votes
1 answer

SQL Server sys.tables not capturing correct modify_date

Context: I have an ETL process setup to populate tables in a datawarehouse using a Python script. The script executes a truncate statement via sqlalchemy to empty each of the tables then uses the pandas .to_sql method to load fresh source data from…
emalcolmb
  • 1,585
  • 4
  • 18
  • 43
0
votes
0 answers

How to know whether a new table was created with pandas to_sql?

I will be inserting new values into existing tables all the time, hence the if_exists will set to append. On some occasions there will be no existing table and I want to preform some additional operation when this is the case. How to I know if a new…
Borut Flis
  • 15,715
  • 30
  • 92
  • 119
0
votes
1 answer

pandas to_sql postgresql with array dtypes

I have a dataframe that holds datetime values and arrays and possibly other datatypes in the future. I wish to to_sql it to PostgreSQL where datetime is a (timestamp without time zone) and arrays are (byte) types, but I have no idea what to put for…
xyiong
  • 363
  • 1
  • 10
0
votes
0 answers

DataFram read from csv and then written to table includes space for cols

there is a case i have whereby a df read from csv and then written to table includes space for certain cols. here is a portion of the csv referenced as 'filename' below: id, l, s 1,a,ay 2,b,bee 3,c,see imports and sqlite connection: import pandas…
etch_45
  • 792
  • 1
  • 6
  • 21
0
votes
0 answers

Loading data to_sql comeup with error "(psycopg2.errors.StringDataRightTruncation) value too long for type character varying(256)" - Redhift/python

I am trying to load a csv file from s3 to redshift table using python. I have used boto3 to pull data from s3. Used pandas to convert data types (timestamp, string and integer) and tried to upload the dataframe to table using to_sql (sqlalchemy). It…
Dinesh
  • 21
  • 5
0
votes
0 answers

Insert dataframe to database table with different column names

I have to insert pandas dataframe into mysql table. Problem here is that dataframe consist different column names that mysql table but matches same number of columns. Is there any workaround to use .to_sql() method? Here is my approach: engine =…
Josef
  • 2,648
  • 5
  • 37
  • 73
0
votes
0 answers

How do I tell if a pandas column is a string or an ordered dict of the collection.OrderedDict variety

Since I can't just do the to_sql directly if a column is an ordered dict, I want to be able to detect it and then turn it into a json.
Henrietta Martingale
  • 771
  • 8
  • 27
0
votes
1 answer

Replace table with dependencies using pandas to_sql's if_exists='replace'

Pandas pd.to_sql() function has the parameter if_exists='replace', which drops the table and inserts the given DataFrame. But the table I'm trying to replace is part of a view. Is there a way to replace the table and keep the view without having to…
Rafael Higa
  • 655
  • 1
  • 8
  • 17
0
votes
1 answer

how to specify Int in dataframe.to_sql?

I am trying to fetch data from my database and trying to write it into another table but for some reason the dataframe.to_sql throws an error: > engine = create_engine("mysql+pymysql://{user}:{pw}@localhost/{db}" …
dhanusha
  • 1
  • 1
0
votes
2 answers

Row size is too large while replacing mysql table with dataframe from python using pymysql

I am writing table to mysql from python using pymysql to_sql function. I am having 1000 rows with 200 columns. Query to connect to mysql is below: engine = create_engine("mysql://hostname:password#@localhostname/dbname") conn =…
Navya
  • 307
  • 3
  • 15
0
votes
1 answer

Pandas to_sql TypeError unsupported operand type

I am doing a database insertion using Pandas to_sql to move millions of rows into sqlalchemy. I've created a small test csv with only 4 rows so that I know exactly what data is in the file. Here is the csv…
Sean Payne
  • 1,625
  • 1
  • 8
  • 20
0
votes
1 answer

Add pandas DataFrame to SQL

I want to add data from a Pandas DataFrame to SQL. I have tried to use .tosql() but if I use if_exists='append', index = True it will add even if a row with same index already exists. Does anyone know how to add the data if the index is not already…
Poorna Chandra R
  • 99
  • 9
1 2 3
…
9 10