Fastest way to populate QTableView from Pandas data frame

Question

I'm very new to PyQt and I am struggling to populate a QTableView control.

My code is the following:

def data_frame_to_ui(self, data_frame):
        """
        Displays a pandas data frame into the GUI
        """
        list_model = QtGui.QStandardItemModel()
        i = 0
        for val in data_frame.columns:
            # for the list model
            if i > 0:
                item = QtGui.QStandardItem(val)
                #item.setCheckable(True)
                item.setEditable(False)
                list_model.appendRow(item)
            i += 1
        self.ui.profilesListView.setModel(list_model)

        # for the table model
        table_model = QtGui.QStandardItemModel()

        # set table headers
        table_model.setColumnCount(data_frame.columns.size)
        table_model.setHorizontalHeaderLabels(data_frame.columns.tolist())
        self.ui.profileTableView.horizontalHeader().setStretchLastSection(True)

        # fill table model data
        for row_idx in range(10): #len(data_frame.values)
            row = list()
            for col_idx in range(data_frame.columns.size):
                val = QtGui.QStandardItem(str(data_frame.values[row_idx][col_idx]))
                row.append(val)
            table_model.appendRow(row)

        # set table model to table object
        self.ui.profileTableView.setModel(table_model)

Actually in the code I succeed to populate a QListView, but the values I set to the QTableView are not displayed, also you can see that I truncated the rows to 10 because it takes forever to display the hundreds of rows of the data frame.

So, What is the fastest way to populate the table model from a pandas data frame?

Thanks in advance.

Did you test the solution given by Wolph to see if it gave better performance? — ekhumoro, Jul 17 '15 at 14:27
Not yet, also I don't fully understand it so it'll take me some time. — Santi Peñate-Vera, Jul 17 '15 at 14:35
I did a little testing. For a table with 25 columns and 10000 rows, the custom model is about 40 times faster (and the performance difference grows geometrically as the number of rows/columns are increased). This was using a simple list-of-lists for the data, so it seems that creating all those instances of `QStandardItem` is the main bottleneck. — ekhumoro, Jul 17 '15 at 16:33
Not sure if this is helpful, but pandas used to have a pyqt model. It looks like it has been split of into another project now, so you might want to check out [pandas-qt](https://github.com/datalyze-solutions/pandas-qt). No idea what the performance is like though. — three_pineapples, Jul 18 '15 at 11:22
@ekhumoro, Do you mind to post your code? with the class as is I get: `return QtCore.QVariant() TypeError: PyQt4.QtCore.QVariant represents a mapped type and cannot be instantiated` — Santi Peñate-Vera, Jul 22 '15 at 08:13
@SantiPeñate-Vera. Are you using python 3? If so, then you can just get rid of all the `QVariant` cruft and return ordinary python types instead. The specific line giivng the error can be omitted altogether (i.e. just allow the method to return `None`). — ekhumoro, Jul 22 '15 at 15:53
I happen to be in Python 3.4. Is `Qvariant()` requited for python 2.7? — Santi Peñate-Vera, Jul 23 '15 at 06:42
@SantiPeñate-Vera. No, but it's the default. See: [Selecting Incompatible APIs](http://pyqt.sourceforge.net/Docs/PyQt4/incompatible_apis.html) in the PyQt docs. — ekhumoro, Jul 26 '15 at 01:52

score 26 · Accepted Answer · edited Mar 04 '21 at 01:04

26

Personally I would just create my own model class to make handling it somewhat easier.

For example:

import sys
from PyQt4 import QtCore, QtGui
Qt = QtCore.Qt

class PandasModel(QtCore.QAbstractTableModel):
    def __init__(self, data, parent=None):
        QtCore.QAbstractTableModel.__init__(self, parent)
        self._data = data

    def rowCount(self, parent=None):
        return len(self._data.values)

    def columnCount(self, parent=None):
        return self._data.columns.size

    def data(self, index, role=Qt.DisplayRole):
        if index.isValid():
            if role == Qt.DisplayRole:
                return QtCore.QVariant(str(
                    self._data.iloc[index.row()][index.column()]))
        return QtCore.QVariant()


if __name__ == '__main__':
    application = QtGui.QApplication(sys.argv)
    view = QtGui.QTableView()
    model = PandasModel(your_pandas_data)
    view.setModel(model)

    view.show()
    sys.exit(application.exec_())

edited Mar 04 '21 at 01:04

K3---rnc

6,717
3
31
46

answered Jul 17 '15 at 13:15

Wolph

78,177
11
137
148

1

Hi, I get the following error: `return QtCore.QVariant() TypeError: PyQt4.QtCore.QVariant represents a mapped type and cannot be instantiated` – Santi Peñate-Vera Jul 22 '15 at 08:16
for new pandas version you have to replace `self._data.iloc[index.row()][index.column()]))` with `self._data.iat[row, col]` – Tuhin Mitra Dec 06 '21 at 12:03
1

if anyone is experiencing slowness or sluggishness when using the `PandasModel`, the issue for me was how `rowCount` is calculated. `self._data.values` returns a `numpy` array, which is a magnitude slower than `len(df.index)`. when running with `timeit` on a df that is (1000,1000): `In [5]: %timeit len(df.values) 6.18 µs ± 5.21 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) In [7]: %timeit len(df.index) 306 ns ± 0.88 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)` – ajoseps Dec 09 '22 at 15:50
`.iat` is also about 20% faster than `.iloc` – misantroop Feb 02 '23 at 16:51

score 22 · Answer 2 · edited Mar 04 '21 at 01:05

This works:

class PandasModel(QtCore.QAbstractTableModel):
    """
    Class to populate a table view with a pandas dataframe
    """
    def __init__(self, data, parent=None):
        QtCore.QAbstractTableModel.__init__(self, parent)
        self._data = data

    def rowCount(self, parent=None):
        return len(self._data.values)

    def columnCount(self, parent=None):
        return self._data.columns.size

    def data(self, index, role=QtCore.Qt.DisplayRole):
        if index.isValid():
            if role == QtCore.Qt.DisplayRole:
                return str(self._data.iloc[index.row()][index.column()])
        return None

    def headerData(self, col, orientation, role):
        if orientation == QtCore.Qt.Horizontal and role == QtCore.Qt.DisplayRole:
            return self._data.columns[col]
        return None

Using it like this:

model = PandasModel(your_pandas_data_frame)
your_tableview.setModel(model)

I read here to avoid QVariant() from PyQT 4.6 on.

Old response but still a good one. If you want the index of your dataframe to appear in the row, you can modify the method headerData as follows: `def headerData(self, rowcol, orientation, role): if orientation == QtCore.Qt.Horizontal and role == QtCore.Qt.DisplayRole: return self._data.columns[rowcol] if orientation == QtCore.Qt.Vertical and role == QtCore.Qt.DisplayRole: return self._data.index[rowcol] return None` — hyamanieu, Jun 28 '16 at 15:38
That works, but how can I make the model editable and then move it back to a dataframe? Currently it cannot even be edited. — Nickpick, Dec 14 '16 at 00:40

score 10 · Answer 3 · answered Mar 22 '17 at 15:17

I've found all of the proposed answers painfully slow for DataFrames with 1000+ rows. What works for me blazingly fast:

class PandasModel(QtCore.QAbstractTableModel):
    """
    Class to populate a table view with a pandas dataframe
    """
    def __init__(self, data, parent=None):
        QtCore.QAbstractTableModel.__init__(self, parent)
        self._data = data

    def rowCount(self, parent=None):
        return self._data.shape[0]

    def columnCount(self, parent=None):
        return self._data.shape[1]

    def data(self, index, role=QtCore.Qt.DisplayRole):
        if index.isValid():
            if role == QtCore.Qt.DisplayRole:
                return str(self._data.iloc[index.row(), index.column()])
        return None

    def headerData(self, col, orientation, role):
        if orientation == QtCore.Qt.Horizontal and role == QtCore.Qt.DisplayRole:
            return self._data.columns[col]
        return None

score 6 · Answer 4 · answered Mar 17 '19 at 15:49

Apart from using QtCore.QAbstractTableModel, one may also inherit from QtGui.QStandardItemModel. I find this way is easier to support handleChanged event emiited from QTableView.

from PyQt5 import QtCore, QtGui

class PandasModel(QtGui.QStandardItemModel):
    def __init__(self, data, parent=None):
        QtGui.QStandardItemModel.__init__(self, parent)
        self._data = data
        for row in data.values.tolist():
            data_row = [ QtGui.QStandardItem("{0:.6f}".format(x)) for x in row ]
            self.appendRow(data_row)
        return

    def rowCount(self, parent=None):
        return len(self._data.values)

    def columnCount(self, parent=None):
        return self._data.columns.size

    def headerData(self, x, orientation, role):
        if orientation == QtCore.Qt.Horizontal and role == QtCore.Qt.DisplayRole:
            return self._data.columns[x]
        if orientation == QtCore.Qt.Vertical and role == QtCore.Qt.DisplayRole:
            return self._data.index[x]
        return None

The above works perfectly on PyQt5. Thanks for the post @Frederick Li - though I did modify the `data_row` line to just input the values as strings, but other than that it cut down my load time from what was possibly a minute or more, down to just a few seconds. — NL23codes, Mar 27 '19 at 03:36

score 6 · Answer 5 · edited Apr 05 '21 at 12:18

Here is fully working copy-paste example for PyQT5 based on @Frederick Li answer with minor modifications.

from PyQt5 import QtGui, QtWidgets
from PyQt5.QtCore import Qt
import sys
import pandas as pd

class MainWindow(QtWidgets.QMainWindow):
    def __init__(self, *args, obj=None, **kwargs):
        super(MainWindow, self).__init__(*args, **kwargs)

        self.centralwidget = QtWidgets.QWidget(self)
        sizePolicy = QtWidgets.QSizePolicy(QtWidgets.QSizePolicy.MinimumExpanding, QtWidgets.QSizePolicy.Fixed)
        self.centralwidget.setSizePolicy(sizePolicy)

        self.pdtable = QtWidgets.QTableView(self.centralwidget)
        sizePolicy = QtWidgets.QSizePolicy(QtWidgets.QSizePolicy.MinimumExpanding, QtWidgets.QSizePolicy.Fixed)
        self.pdtable.setSizePolicy(sizePolicy)

        dataPD = [['tom', 10.0, 180.3], ['nick', 15.0, 175.7], ['juli', 14.0, 160.6]]
        df = pd.DataFrame(dataPD, columns=['Name', 'Age', 'Height'])
        print(df.dtypes)
        self.model = PandasTableModel(df)
        self.pdtable.setModel(self.model)

        self.setCentralWidget(self.centralwidget)


class PandasTableModel(QtGui.QStandardItemModel):
    def __init__(self, data, parent=None):
        QtGui.QStandardItemModel.__init__(self, parent)
        self._data = data
        for col in data.columns:
            data_col = [QtGui.QStandardItem("{}".format(x)) for x in data[col].values]
            self.appendColumn(data_col)
        return

    def rowCount(self, parent=None):
        return len(self._data.values)

    def columnCount(self, parent=None):
        return self._data.columns.size

    def headerData(self, x, orientation, role):
        if orientation == Qt.Horizontal and role == Qt.DisplayRole:
            return self._data.columns[x]
        if orientation == Qt.Vertical and role == Qt.DisplayRole:
            return self._data.index[x]
        return None


if __name__ == "__main__":
    app  = QtWidgets.QApplication(sys.argv)
    app.setStyle("Fusion")
    main = MainWindow()
    main.show()
    main.resize(600, 400)
    sys.exit(app.exec_())

The QT website has a Pandas Simple Example here: https://doc.qt.io/qtforpython/examples/example_external__pandas.html — zeroalpha, Feb 23 '22 at 03:44

score 5 · Answer 6 · answered Sep 28 '16 at 09:50

There is actually some code in pandas supporting integration with Qt.

At the time of writing this answer, the latest pandas version is 0.18.1 and you could do:

from pandas.sandbox.qtpandas import DataFrameModel, DataFrameWidget

That code seems to be coupled to PySide, however it should be relatively trivial to make it work with PyQt. Also, that code has been deprecated and the warning says that the module will be removed in the future.

Luckily they extracted that into a separated project in GitHub called pandas-qt:

https://github.com/datalyze-solutions/pandas-qt

I would try to use that before trying to roll out my own model and view implementation.

Hi I just want to add, pandas-qt doesn't support python3 and it appears like it probably won't. In the mean time you can use [qtpandas](https://github.com/draperjames/qtpandas) `pip install qtpandas` — James Draper, Dec 07 '16 at 18:46

score 4 · Answer 7 · answered Jul 26 '19 at 18:10

Simple and faster way to write a dataframe to QtableWidget

# Takes a df and writes it to a qtable provided. df headers become qtable headers
@staticmethod
def write_df_to_qtable(df,table):
    headers = list(df)
    table.setRowCount(df.shape[0])
    table.setColumnCount(df.shape[1])
    table.setHorizontalHeaderLabels(headers)        

    # getting data from df is computationally costly so convert it to array first
    df_array = df.values
    for row in range(df.shape[0]):
        for col in range(df.shape[1]):
            table.setItem(row, col, QtGui.QTableWidgetItem(str(df_array[row,col])))

Fastest way to populate QTableView from Pandas data frame

7 Answers7

Linked

Related