0

I have a PySide6 application using an MVC structure. It utilizies two threads. The main thread is the Qt Event loop(GUI). The Model utilizes an asyncio event loop which runs in the other thread. The Model handles all the IO for the application. When new data is imported into the application, the Model emits it as a DataFrame via a Signal.

I have a window(AllPurposeTable) which contains an instance of QTableView. This QTableView is connected to an instance of AllPurposeTableModel. AllPurposeTableModel subclasses QAbstractTableModel. It has a Slot that is connected to the Signal in the Model.

After the data is received in on_data_update(), the dataChanged() method must be called to inform the QTableView that the data has changed and to update the view

This entire flow structure works as expected. New data comes in, and it correctly updates the view. However, the memory usage in this Python process continues to rise as long as the AllPurposeTable window is open.

The Slot function reassigns self.df correctly(which is proven with the print(self.df) line. Memory usage stays constant when this update repeatedly happens

When dataChanged() is emitted, the memory usage increases consistently and I can't figure out why. I have attempted multiple implementations of the necessary methods in QAbstractTableModel with no luck. No errors appear during any of this. Where is this excess memory being allocated and how can I prevent it?

import sys
import os
import asyncio
import numpy as np
import pandas as pd
from datetime import datetime
from typing import Any

from PySide6.QtCore import QThread, QObject, Signal, QAbstractTableModel, QModelIndex, Slot
from PySide6.QtWidgets import QApplication, QMainWindow

from PySide6.QtCore import (QCoreApplication, QDate, QDateTime, QLocale,
    QMetaObject, QObject, QPoint, QRect,
    QSize, QTime, QUrl, Qt)
from PySide6.QtGui import (QAction, QBrush, QColor, QConicalGradient,
    QCursor, QFont, QFontDatabase, QGradient,
    QIcon, QImage, QKeySequence, QLinearGradient,
    QPainter, QPalette, QPixmap, QRadialGradient,
    QTransform)
from PySide6.QtWidgets import (QApplication, QGridLayout, QHeaderView, QMainWindow,
    QMenu, QMenuBar, QSizePolicy, QStatusBar,
    QTableView, QWidget)


class Ui_AllPurposeTable(object):
    def setupUi(self, AllPurposeTable):
        if not AllPurposeTable.objectName():
            AllPurposeTable.setObjectName(u"AllPurposeTable")
        AllPurposeTable.resize(581, 348)
        self.actionQuiote = QAction(AllPurposeTable)
        self.actionQuiote.setObjectName(u"actionQuiote")
        self.actionRows = QAction(AllPurposeTable)
        self.actionRows.setObjectName(u"actionRows")
        self.action_edit_columns_showhide = QAction(AllPurposeTable)
        self.action_edit_columns_showhide.setObjectName(u"action_edit_columns_showhide")
        self.action_edit_columns_rename = QAction(AllPurposeTable)
        self.action_edit_columns_rename.setObjectName(u"action_edit_columns_rename")
        self.action_edit_columns_adddelete = QAction(AllPurposeTable)
        self.action_edit_columns_adddelete.setObjectName(u"action_edit_columns_adddelete")
        self.action_loaddata_data = QAction(AllPurposeTable)
        self.action_loaddata_data.setObjectName(u"action_loaddata_data")
        self.actionEdit_Cells = QAction(AllPurposeTable)
        self.actionEdit_Cells.setObjectName(u"actionEdit_Cells")
        self.action_file_settings_open = QAction(AllPurposeTable)
        self.action_file_settings_open.setObjectName(u"action_file_settings_open")
        self.action_file_settings_save = QAction(AllPurposeTable)
        self.action_file_settings_save.setObjectName(u"action_file_settings_save")
        self.action_file_settings_saveas = QAction(AllPurposeTable)
        self.action_file_settings_saveas.setObjectName(u"action_file_settings_saveas")
        self.action_file_data_load = QAction(AllPurposeTable)
        self.action_file_data_load.setObjectName(u"action_file_data_load")
        self.action_file_data_save = QAction(AllPurposeTable)
        self.action_file_data_save.setObjectName(u"action_file_data_save")
        self.action_file_data_saveas = QAction(AllPurposeTable)
        self.action_file_data_saveas.setObjectName(u"action_file_data_saveas")
        self.action_edit_columns_datatypes = QAction(AllPurposeTable)
        self.action_edit_columns_datatypes.setObjectName(u"action_edit_columns_datatypes")
        self.centralwidget = QWidget(AllPurposeTable)
        self.centralwidget.setObjectName(u"centralwidget")
        self.gridLayout_2 = QGridLayout(self.centralwidget)
        self.gridLayout_2.setObjectName(u"gridLayout_2")
        self.gridLayout = QGridLayout()
        self.gridLayout.setObjectName(u"gridLayout")
        self.table_view = QTableView(self.centralwidget)
        self.table_view.setObjectName(u"table_view")

        self.gridLayout.addWidget(self.table_view, 0, 0, 1, 1)


        self.gridLayout_2.addLayout(self.gridLayout, 0, 0, 1, 1)

        AllPurposeTable.setCentralWidget(self.centralwidget)
        self.menubar = QMenuBar(AllPurposeTable)
        self.menubar.setObjectName(u"menubar")
        self.menubar.setGeometry(QRect(0, 0, 581, 21))
        self.menu_file = QMenu(self.menubar)
        self.menu_file.setObjectName(u"menu_file")
        self.menu_file_settings = QMenu(self.menu_file)
        self.menu_file_settings.setObjectName(u"menu_file_settings")
        self.menu_file_data = QMenu(self.menu_file)
        self.menu_file_data.setObjectName(u"menu_file_data")
        self.menu_edit = QMenu(self.menubar)
        self.menu_edit.setObjectName(u"menu_edit")
        self.menu_columns = QMenu(self.menu_edit)
        self.menu_columns.setObjectName(u"menu_columns")
        self.menu_view = QMenu(self.menubar)
        self.menu_view.setObjectName(u"menu_view")
        self.menu_help = QMenu(self.menubar)
        self.menu_help.setObjectName(u"menu_help")
        AllPurposeTable.setMenuBar(self.menubar)
        self.statusbar = QStatusBar(AllPurposeTable)
        self.statusbar.setObjectName(u"statusbar")
        AllPurposeTable.setStatusBar(self.statusbar)

        self.menubar.addAction(self.menu_file.menuAction())
        self.menubar.addAction(self.menu_edit.menuAction())
        self.menubar.addAction(self.menu_view.menuAction())
        self.menubar.addAction(self.menu_help.menuAction())
        self.menu_file.addAction(self.menu_file_data.menuAction())
        self.menu_file.addAction(self.menu_file_settings.menuAction())
        self.menu_file.addAction(self.actionQuiote)
        self.menu_file_settings.addAction(self.action_file_settings_open)
        self.menu_file_settings.addAction(self.action_file_settings_save)
        self.menu_file_settings.addAction(self.action_file_settings_saveas)
        self.menu_file_data.addAction(self.action_file_data_load)
        self.menu_file_data.addAction(self.action_file_data_save)
        self.menu_file_data.addAction(self.action_file_data_saveas)
        self.menu_edit.addAction(self.menu_columns.menuAction())
        self.menu_columns.addAction(self.action_edit_columns_showhide)
        self.menu_columns.addAction(self.action_edit_columns_rename)
        self.menu_columns.addAction(self.action_edit_columns_adddelete)
        self.menu_columns.addAction(self.action_edit_columns_datatypes)

        self.retranslateUi(AllPurposeTable)

        QMetaObject.connectSlotsByName(AllPurposeTable)
    # setupUi

    def retranslateUi(self, AllPurposeTable):
        AllPurposeTable.setWindowTitle(QCoreApplication.translate("AllPurposeTable", u"Table", None))
        self.actionQuiote.setText(QCoreApplication.translate("AllPurposeTable", u"Quit", None))
        self.actionRows.setText(QCoreApplication.translate("AllPurposeTable", u"Rows", None))
        self.action_edit_columns_showhide.setText(QCoreApplication.translate("AllPurposeTable", u"Show/Hide", None))
        self.action_edit_columns_rename.setText(QCoreApplication.translate("AllPurposeTable", u"Rename", None))
        self.action_edit_columns_adddelete.setText(QCoreApplication.translate("AllPurposeTable", u"Add/Delete", None))
        self.action_loaddata_data.setText(QCoreApplication.translate("AllPurposeTable", u"Load Data", None))
        self.actionEdit_Cells.setText(QCoreApplication.translate("AllPurposeTable", u"Edit Cells", None))
        self.action_file_settings_open.setText(QCoreApplication.translate("AllPurposeTable", u"Open", None))
        self.action_file_settings_save.setText(QCoreApplication.translate("AllPurposeTable", u"Save", None))
        self.action_file_settings_saveas.setText(QCoreApplication.translate("AllPurposeTable", u"Save As...", None))
        self.action_file_data_load.setText(QCoreApplication.translate("AllPurposeTable", u"Load", None))
        self.action_file_data_save.setText(QCoreApplication.translate("AllPurposeTable", u"Save", None))
        self.action_file_data_saveas.setText(QCoreApplication.translate("AllPurposeTable", u"Save As...", None))
        self.action_edit_columns_datatypes.setText(QCoreApplication.translate("AllPurposeTable", u"Data Types", None))
        self.menu_file.setTitle(QCoreApplication.translate("AllPurposeTable", u"File", None))
        self.menu_file_settings.setTitle(QCoreApplication.translate("AllPurposeTable", u"Settings", None))
        self.menu_file_data.setTitle(QCoreApplication.translate("AllPurposeTable", u"Data", None))
        self.menu_edit.setTitle(QCoreApplication.translate("AllPurposeTable", u"Edit", None))
        self.menu_columns.setTitle(QCoreApplication.translate("AllPurposeTable", u"Columns", None))
        self.menu_view.setTitle(QCoreApplication.translate("AllPurposeTable", u"View", None))
        self.menu_help.setTitle(QCoreApplication.translate("AllPurposeTable", u"Help", None))
    # retranslateUi


class Ui_MainWindow(object):
    def setupUi(self, MainWindow):
        if not MainWindow.objectName():
            MainWindow.setObjectName(u"MainWindow")
        MainWindow.resize(198, 132)
        self.action_open_allpurposetable = QAction(MainWindow)
        self.action_open_allpurposetable.setObjectName(u"action_open_allpurposetable")
        self.centralwidget = QWidget(MainWindow)
        self.centralwidget.setObjectName(u"centralwidget")
        MainWindow.setCentralWidget(self.centralwidget)
        self.menubar = QMenuBar(MainWindow)
        self.menubar.setObjectName(u"menubar")
        self.menubar.setGeometry(QRect(0, 0, 198, 21))
        self.menuOpen = QMenu(self.menubar)
        self.menuOpen.setObjectName(u"menuOpen")
        MainWindow.setMenuBar(self.menubar)
        self.statusbar = QStatusBar(MainWindow)
        self.statusbar.setObjectName(u"statusbar")
        MainWindow.setStatusBar(self.statusbar)

        self.menubar.addAction(self.menuOpen.menuAction())
        self.menuOpen.addAction(self.action_open_allpurposetable)

        self.retranslateUi(MainWindow)

        QMetaObject.connectSlotsByName(MainWindow)
    # setupUi

    def retranslateUi(self, MainWindow):
        MainWindow.setWindowTitle(QCoreApplication.translate("MainWindow", u"MainWindow", None))
        self.action_open_allpurposetable.setText(QCoreApplication.translate("MainWindow", u"All Purpose Table", None))
        self.menuOpen.setTitle(QCoreApplication.translate("MainWindow", u"Open", None))
    # retranslateUi


class ModelUpdateThread(QThread):
    def __init__(self, model):
        super().__init__()
        self.model = model

    # Override from QThread
    def run(self):
        loop = asyncio.new_event_loop()
        asyncio.set_event_loop(loop)
        task = loop.create_task(self.model.update_model(loop))
        loop.run_until_complete(task)


class Controller(QApplication):
    def __init__(self, sys_argv):
        super().__init__(sys_argv)

        self.model = Model()
        self.model_update_thread = ModelUpdateThread(self.model)
        self.view = View(self, self.model)

        self.model_update_thread.start()


class Model(QObject):
    dataframe_update_signal = Signal(pd.DataFrame)

    def __init__(self):
        super().__init__()
        self.df = None

        self.loop = None

    async def update_data(self):
        """
        Generate DataFrame with random data
        """
        while True:
            data_values = np.random.randint(0, 1000, size=(100, 10))
            df = pd.DataFrame(data_values, columns=list('ABCDEFGHIJ'))
            self.df = df

            self.dataframe_update_signal.emit(self.df)

            await asyncio.sleep(1)

    async def update_model(self, loop):
        """
        This is the main coroutine that is executed in the model update thread
        """
        self.loop = loop

        tasks = [asyncio.create_task(coro()) for coro in (self.update_data,)]

        await asyncio.wait(tasks)


class View(QObject):
    def __init__(self, controller, model):
        super().__init__()
        self.controller = controller
        self.model = model

        self.open_windows = {'main_window': MainWindow(self, self.controller, self.model)}
        self.open_windows['main_window'].show()


class MainWindow(QMainWindow, Ui_MainWindow):
    def __init__(self, parent, controller, model):
        super().__init__()
        self.parent = parent
        self.controller = controller
        self.model = model

        self.setupUi(self)

        self.action_open_allpurposetable.triggered.connect(self.open_allpurposetable)

    def open_allpurposetable(self):
        self.parent.open_windows['demo_table'] = AllPurposeTable(self, self.model, self.parent, self.controller,
                                                                 self.model.df)
        self.parent.open_windows['demo_table'].show()


class AllPurposeTable(QMainWindow, Ui_AllPurposeTable):
    def __init__(self, parent, model, view, controller, df=None):
        super().__init__()
        self.parent = parent
        self.model = model
        self.view = view
        self.controller = controller
        self.df = df

        self.setupUi(self)

        self.table_model = None

        self.initialize_table_model()

    # This initializes the table model which is then presented by the view
    def initialize_table_model(self):
        if self.df is not None:
            self.table_model = AllPurposeTableModel(self, self.model, self.df)
            self.table_view.setModel(self.table_model)


class AllPurposeTableModel(QAbstractTableModel):
    def __init__(self, parent, model, df):
        super().__init__()
        self.parent = parent
        self.model = model
        self.df = df

        self.model.dataframe_update_signal.connect(self.on_data_update)

    @Slot(str)
    def on_data_update(self, df):
        """
        THIS IS WHERE THE PROBLEM IS
        """
        self.df = df

        # The data is correctly updating
        print(self.df)

        # MEMORY LEAK HERE
        self.dataChanged.emit(QModelIndex(), QModelIndex(), Qt.DisplayRole)

    # Override from QAbstractItemModel
    def rowCount(self, parent=QModelIndex()) -> int:
        if parent == QModelIndex():
            return self.df.shape[0]
        return 0

    # Override from QAbstractItemModel
    def columnCount(self, parent=QModelIndex()) -> int:
        if parent == QModelIndex():
            return self.df.shape[1]
        return 0

    # Override from QAbstractItemModel
    def data(self, index: QModelIndex, role=Qt.ItemDataRole) -> Any:
        if index.isValid():
            if role == Qt.DisplayRole:
                value = self.df.iloc[index.row(), index.column()]

                if isinstance(value, (np.bool_, bool)):
                    return str(value)
                if isinstance(value, (int, np.integer)):
                    return str('{:,}'.format(value))
                if isinstance(value, float):
                    return str(round(value, 2))
                if isinstance(value, datetime):
                    return value.strftime('%Y-%m-%d %H:%M:%S')
                if isinstance(value, list):
                    if isinstance(value[0], (int, np.integer)):
                        return ','.join(list(map(str, value)))
                    if isinstance(value[0], str):
                        return ','.join(value)
                if isinstance(value, str):
                    return value

            if role == Qt.BackgroundRole:
                return

            if role == Qt.TextAlignmentRole:
                return

            return None


if __name__ == '__main__':
    print('PID = {}'.format(os.getpid()))
    app = Controller(sys.argv)
    sys.exit(app.exec())
Rob M.
  • 1
  • 3
  • Are you not adding rows or columns to the table view when calling `on_data_update`? It also might just be that the garbage collector hasn't collected the previous dataframe that you are overwriting each time you call `on_data_update`. I am just guessing though – Alexander Feb 28 '23 at 21:45
  • In nearly every instance, the shape of the dataframe is not changing, only the values are updating. Since the memory leak happens when the dataChanged() method is called, I am thinking this is an issue with a C++ object under the hood, which I know very little about – Rob M. Feb 28 '23 at 22:04
  • Creating a QThread to create a loop and use async seems a bit redundant. Besides, the values are not updating: you are creating a new data frame every time, that's not the same. Can you explain why you're using this approach? – musicamante Mar 01 '23 at 00:16
  • I am creating a "dashboard" of sorts for financial analysis. I am utilizing multiple websockets and aiosqlite for I/O, both of which are built on top of asyncio. I did not think it was good practice to be placing Model-orientated tasks on the GUI event loop, so I decided to create a new event loop for the Model to separate the logic. I have had no issues with this idea so far. Reassigning the value of self.df on each iteration of an update does not seem to be increasing the memory(comment out dataChanged() line, print out newly reassigned self.df) – Rob M. Mar 01 '23 at 00:47
  • Then it's possible that the signal indirectly creates a closure to the memory address of the dataframe in the scope of that function. PyQt tries to keep references even when signals are queued (which is your case, since signals emitted in a different thread than their receivers are *queued*), and, normally, the "Qt side" will clear up the memory references on its own: it's possible that the async prevents that (since it's done on the Python side instead). The question remains, though: can't you just *replace* the changed values in the current dataframe instead of completely replacing it? – musicamante Mar 01 '23 at 04:09
  • Remember that a dataframe is a reference to the memory: creating a new one is not the same as changing its contents. Roughly (and imprecisely) speaking: `array = np.array([0, 1]); array[1] = 2` is ***NOT*** the same as `array = np.array([0, 1]); array = np.array([0, 2])`. – musicamante Mar 01 '23 at 04:13
  • Reassigning the DataFrame vs updating its values (pd.DataFrame .update()) has no impact on performance or memory usage. Both methods work, so I suppose I could use the update method... I don't think the issue is with the Signals/Slots. If I get rid of dataChanged(), everything works correctly. The Signal in the Model thread emits the DataFrame, and the Slot correctly receives it and reassigns(or updates). This leads me to believe the issue is with the implementation of QAbstractTableModel (or the Tableview, which I think is less likely). However, this is just a guess, which is why I'm here – Rob M. Mar 01 '23 at 14:39
  • Reassigning *does* have impact if the previous dataframe is not properly collected, which may very well happen since the original signal is queued, while the `dataChanged` is direct (meaning that *that* `self.df`), causing direct calls to everything related to the model, and since you emitted the signal for the *whole* model, that means **a lot** of calls, which may take too much time, possibly causing a closure that would prevent proper garbage collection. I admit that I don't have experience with async, but that seems a logical explanation. Note that this has nothing to do with Qt (not the-> – musicamante Mar 01 '23 at 20:06
  • -> table model, nor the view): it does not depend on how they are implemented, at least, not directly. It *may* be related to the python binding (remember that PySide is a wrapper around Qt objects, and creates C++ object pointers suitable to the python types when passing "through" the C++ side, which is what happens when using signals). The whole mix of python binding and async may very well explain the leak. As said, if you *update* the dataframe (specifically, its numpy array), the problem may not happen at all, but you should also ensure that no race conditions happens in the meantime. – musicamante Mar 01 '23 at 20:10
  • @RobM. Btw, note that in `on_data_update` the `dataChanged` signal has the wrong signature: maybe that's just a typo in the example, but the third argument should be a *list* of roles, not a single one. – musicamante Mar 04 '23 at 18:09
  • I was able to strip down everything and this is an issue with PySide6. Using example code from Qt, website, I was able to recreate the conditions for this memory leak. This leak does not exist in PySide2. I just submitted a bug report https://bugreports.qt.io/browse/PYSIDE-2249 – Rob M. Mar 04 '23 at 19:29
  • 1
    UPDATE: This issue has been fixed. https://codereview.qt-project.org/c/pyside/pyside-setup/+/464629 – Rob M. Mar 06 '23 at 22:31
  • @RobM. Good to know! So, I was both wrong (the dataframe was properly collected, after all) and partially right: it is indeed a PySide issue, but related to enums, not to the wrapping of the signal object. It's interesting to see that even PySide is struggling with enums as PyQt, with the latter taking the more radical path (forcing full namespaces for all flags) and the former providing a "forgiveness mode" (that is still not immune to issues, and a memory leak is a big one). – musicamante Mar 06 '23 at 22:58
  • @RobM. So, if I understand it correctly, if you properly use the full flag (and the proper syntax as mentioned above), the problem could be solved even with the current PySide version? If you can confirm that using `self.dataChanged.emit(QModelIndex(), QModelIndex(), [Qt.ItemDataRole.DisplayRole])` will fix the problem? If so, you could post your own answer for that, linking your report and the related info. Remember that, as said, that argument is optional. – musicamante Mar 06 '23 at 23:01
  • I have updated my initial post to reflect the changes. You are correct in that simply using the full flag syntax for the enumeration fixes the problem. Once this fix gets pushed to production, the short version will not leak anymore either – Rob M. Mar 07 '23 at 15:39
  • @RobM. Please don't use the question to add an answer, use the dedicated answer field instead. – musicamante Mar 07 '23 at 19:01
  • Apologies...I am new to actually interacting on this website. I will make sure to keep good practices going forward. – Rob M. Mar 07 '23 at 21:02

1 Answers1

0

A fix was released yesterday 3/6/2023 which can be found here https://codereview.qt-project.org/c/pyside/pyside-setup/+/464629 ...The issue was related to an enumeration handler in PySide. The "forgiveness mode" was not correctly garbage collecting. To solve this issue immediately, simply change Qt.DisplayRole to Qt.ItemDataRole.DisplayRole(do this for all enumerations in your code). I have tested this on multiple examples and everything looks good now.

Example of the updated data() method...

    def data(self, index: QModelIndex, role=Qt.ItemDataRole) -> Any:
        if index.isValid():

            if role == Qt.ItemDataRole.DisplayRole:
                return

            if role == Qt.ItemDataRole.BackgroundRole:
                return

            if role == Qt.ItemDataRole.TextAlignmentRole:
                return

            return None
Rob M.
  • 1
  • 3