Background Tasks With PyQt

Every library for desktop applications development works with a main loop that manages events such as displaying, moving, or resizing the window, responding to a button or keyboard press, or any other interaction with the user interface. Some of those events might be connected to our own functions; for example, a button1_pressed() method might be invoked by a library when the user presses the button1 widget. When working with Qt, the proper way to respond to those events is by connecting a signal to a slot.

But the problem arises when, while responding to a certain event or during the interface setup, we execute a task whose duration is not negligible (we could say that any task that lasts for more than one second is not negligible). When this happens, the processor is busy executing our heavy task and is not able to execute our application's main loop, thus our interface stops responding: we cannot move it, close it, resize it, nor interact with it in any other way.

Let's see a real case. The following code draws a window with a label (QLabel) and a button (QPushButton) that, when pressed, downloads a file via HTTP using the urllib.request standard module.

from urllib.request import urlopen
from PyQt5.QtWidgets import QApplication, QMainWindow, QLabel, QPushButton
class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("File Download with PyQt")
        self.resize(400, 300)
        self.label = QLabel("Press the button to start the download.",
            self)
        self.label.setGeometry(20, 20, 200, 25)
        self.button = QPushButton("Start download", self)
        self.button.move(20, 60)
        self.button.pressed.connect(self.downloadFile)
    def downloadFile(self):
        self.label.setText("Downloading file...")
        # Disable the button while downloading the file.
        self.button.setEnabled(False)
        url = "https://www.python.org/ftp/python/3.7.2/python-3.7.2.exe"
        filename = "python-3.7.2.exe"
        # Open the URL address.
        with urlopen(url) as r:
            with open(filename, "wb") as f:
                # Read the remote file and write the local one.
                f.write(r.read())
        self.label.setText("¡File downloaded!")
        # Restore the button.
        self.button.setEnabled(True)
if __name__ == "__main__":
    app = QApplication([])
    window = MainWindow()
    window.show()
    app.exec_()
/images/background-tasks-with-pyqt/pyqt-download-file-urllib.png

You will notice that during the file download, which for me takes near to five seconds, the window is totally frozen. We already mention the reason for this behaviour: in this particular code, the line number 29, where r.read() is called (the function that downloads the content of the remote file), blocks Qt's main loop.

We shall consider three different workarounds for this same problem, each one with its own pros and cons.

First workaround. Threads

This solution involves throwing a new thread that executes our heavy task. Since Qt's main loop runs in our application's main thread, and our task runs in a secondary thread, the user interface remains active while the file is downloaded in the background. To achieve this we will employ the QThread class, that provides a cross-platform API to work with threads.

from urllib.request import urlopen
from PyQt5.QtCore import QThread
from PyQt5.QtWidgets import QApplication, QMainWindow, QLabel, QPushButton
class Downloader(QThread):
    def __init__(self, url, filename):
        super().__init__()
        self._url = url
        self._filename = filename
    def run(self):
        # Open the URL address.
        with urlopen(self._url) as r:
            with open(self._filename, "wb") as f:
                # Read the remote file and write the local one.
                f.write(r.read())
class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("Threaded File Download with PyQt")
        self.resize(400, 300)
        self.label = QLabel("Press the button to start the download.",
            self)
        self.label.setGeometry(20, 20, 200, 25)
        self.button = QPushButton("Start download", self)
        self.button.move(20, 60)
        self.button.pressed.connect(self.initDownload)
    def initDownload(self):
        self.label.setText("Downloading file...")
        # Disable the button while downloading the file.
        self.button.setEnabled(False)
        # Execute the download in a new thread.
        self.downloader = Downloader(
            "https://www.python.org/ftp/python/3.7.2/python-3.7.2.exe",
            "python-3.7.2.exe"
        )
        # Qt will invoke the `downloadFinished()` method once the
        # thread has finished.
        self.downloader.finished.connect(self.downloadFinished)
        self.downloader.start()
    def downloadFinished(self):
        self.label.setText("¡File downloaded!")
        # Restore the button.
        self.button.setEnabled(True)
        # Delete the thread when no longer needed.
        del self.downloader
if __name__ == "__main__":
    app = QApplication([])
    window = MainWindow()
    window.show()
    app.exec_()

The main point of this code is the Downloader class that inherits from QThread and re-implements the run() method (line 14), whose content will be executed within a new thread when we create an instance and call the start() method (lines 40 and 47). In the line 46 we connect the finished signal (that is emitted by Qt when the thread finishes its execution) with our downloadFinished() method.

Although this specific example downloads a file, this method allows us to move any heavy task into the new thread: it requires no more than to put it inside the run() method.

However, multi-threading programming must be done extremely carefully. Only the run() method runs in the new thread, while the rest (even the Downloader.__init__() method itself) runs in the main thread. Also, we must make sure to not share objects that might be accessed at the same time from two or more threads.

Second workaround. Events processing

An alternative to throwing a new thread of execution is to make the whole work in the main thread, but letting Qt periodically process the application events so the user interface doesn't freeze. This can be done via the QCoreApplication.processEvents() function.

In our former code, the function that executed the heavy task and blocked the main loop for some seconds was r.read(). Since this method doesn't return any value until the file has been completelly downloaded (i.e., we don't have the control of the code until the function finishes), we need to create our own loop that fetches the remote file by little chunks (128 bytes) and at the same time lets Qt process the application events. We're lucky because read() takes an optional argument that let us indicate how many bytes we want to fetch.

from urllib.request import urlopen
from PyQt5.QtCore import QCoreApplication
from PyQt5.QtWidgets import QApplication, QMainWindow, QLabel, QPushButton
class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("File Download with PyQt")
        self.resize(400, 300)
        self.label = QLabel("Press the button to start the download.",
            self)
        self.label.setGeometry(20, 20, 200, 25)
        self.button = QPushButton("Start download", self)
        self.button.move(20, 60)
        self.button.pressed.connect(self.downloadFile)
    def downloadFile(self):
        self.label.setText("Descargando archivo...")
        # Disable the button while downloading the file.
        self.button.setEnabled(False)
        # Open the URL address.
        url = "https://www.python.org/ftp/python/3.7.2/python-3.7.2.exe"
        filename = "python-3.7.2.exe"
        with urlopen(url) as r:
            with open(filename, "ab") as f:
                while True:
                    # Let Qt process its events so the window
                    # doesn't freeze.
                    QCoreApplication.processEvents()
                    # Read a chunk of the file we are downloading.
                    chunk = r.read(128)
                    # If the result is `None`, that means there
                    # is no downloaded data yet. Just keep
                    # waiting.
                    if chunk is None:
                        continue
                    # If the result is an empty `bytes` instance,
                    # that means the file is complete.
                    elif chunk == b"":
                        break
                    # Write the downladed chunk into the local file.
                    f.write(chunk)
        self.label.setText("¡File downloaded!")
        # Restore the button.
        self.button.setEnabled(True)
if __name__ == "__main__":
    app = QApplication([])
    window = MainWindow()
    window.show()
    app.exec_()

The main point here is between the lines 29 and 45, where we set up the loop that process the application events, fetches a little chunk of data from the network, and throws it in a local file until there's no more data to be received.

Since the whole code runs in a single thread, we don't need to care about the problems concerning accessing and modifying objects, as we did in the previous code. However, r.read(128) is still a blocking call that freezes the code execution, although during a very little (and probably imperceptible) time. Be the internet connection incredibly slow, and even this little amount of bytes might freeze the user interface, so consider changing the chunk size accordingly.

Third workaround. Twisted

The qt5reactor module let us use Twisted and Qt within a single application, which give us access to the rich set of asynchronous functions provided by the networking library. For this third workaround we will also use the treq library (which is similar to Requests, but built on top of Twisted) to connect to the URL address and download its content. So let's first install these tools:

pip install qt5reactor treq

(Twisted will be automatically installed since its required by both modules.)

Once installed, the code is the following:

from PyQt5.QtCore import QCoreApplication
from PyQt5.QtWidgets import QApplication, QMainWindow, QLabel, QPushButton
from twisted.internet.defer import inlineCallbacks
class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("File Download with PyQt & Twisted")
        self.resize(400, 300)
        self.label = QLabel("Press the button to start the download.",
            self)
        self.label.setGeometry(20, 20, 200, 25)
        self.button = QPushButton("Start download", self)
        self.button.move(20, 60)
        self.button.pressed.connect(self.initDownload)
    def initDownload(self):
        self.label.setText("Descargando archivo...")
        # Disable the button while downloading the file.
        self.button.setEnabled(False)
        url = "https://www.python.org/ftp/python/3.7.2/python-3.7.2.exe"
        # The `requestSucceeded()` will be invoked when the connection
        # to the URL address has been successfully established.
        treq.get(url).addCallback(self.requestSucceeded)
    @inlineCallbacks
    def requestSucceeded(self, response):
        # Get the content of the remote file. Note that
        # this operation doesn't block the execution of
        # the code.
        content = yield response.content()
        # We write the content in a local file.
        with open("python-3.7.2.exe", "wb") as f:
            f.write(content)
        self.label.setText("¡File downloaded!")
        # Restore the button.
        self.button.setEnabled(True)
    def closeEvent(self, event):
        QCoreApplication.instance().quit()
if __name__ == "__main__":
    app = QApplication([])
    import qt5reactor
    qt5reactor.install()
    window = MainWindow()
    window.show()
    from twisted.internet import reactor
    import treq
    import os
    import certifi
    # Required for HTTPS connections.
    os.environ["SSL_CERT_FILE"] = certifi.where()
    reactor.run()

Twisted is a right option when our heavy tasks are network-related and its usage is recurrent in the code. treq's HTTP requests are built on top of Twisted's deferreds, which are similar to Qt signals when we need to call a user-defined event. Here we don't have to take care of the problem concerning sharing objects between threads neither, since Twisted is single-threaded.

Those who are well-versed in Twisted will find this workaround very satisfactory. And, indeed, it is: Qt and Twisted work really well together, since they have an alike structure, philosophy and even naming conventions (mixedCase instead of lower_case).

Conclusion

So let's summarize the three workarounds we have seen. The threads approach works for any heavy task, although must be implemented prudently since thread-safety is hard to achieve. If your application makes HTTP requests very often or interacts with other network resources, and if you feel comfortable with Twisted, you will like the qt5reactor and treq solution. Finally, if your heavy task can be decomposed into little steps (such as calling r.read(128) multiple times), then just adding a QCoreApplication.processEvents() call in the right place will prevent your user interface from freezing.