Remote Frontends for Pyside6-based VFX Tooling over QWebSockets

Oh boy is this going to ruffle some feathers... but if it didn't, was it even worth writing about? Qt isn't fun. The signals, the slots, the inherit messiness, the layers upon layers of object-oriented paradigms, QML being awful, how hard it is to make responsive designs. I could go on.

Of course, there is plenty to like: it's cross-platform, has Python bindings that work in tools with Python API's, and it's often superior to the UI Builders for scripts that come with our software (although this last point has been losing ground, but we'll come back to this.)

At this point, you might be thinking, "Well I don't agree, but I wouldn't say my feathers are ruffled..." In that case, I'll proceed: HTML is much better for this task. Please, don't hurt me.

Before you pick up your pitchforks, keep in mind, when I mentioned that Qt made panel building a better experience than using UI builders included in software, this has changed. The thing it's losing ground to is HTML. Adobe was early to this, with Node Webkit support embedding browser panels into their tools and communicating over a bridge. Since then, others like Fusion have introduced web-based panels as well, in Fusion's case via an Electron app.

In this article I would propose an alternative, using Qt itself to open a WebView, and to take it further, that the contents of the client should be hosted remotely.

A screenshot of a Webview based script in Fusion

Benefits to web-based remote frontends

  • Artists can make the tooling their own. Tools like Clutch(full disclosure: I work for them), which we'll use throughout this article make creating web applications a low-code task. While the same could be said about Qt Designer, the need to install it to your machine, the lack of similarities to design tools like Figma, and the limitations of a widget-based system in general, make actual adoption a bigger challenge.
  • Deployment of frontends is one less thing to worry about, the app will be served the latest version, period.
  • A generic messaging API can be determined allowing the same frontends to work in other hosts, emitting the same commands.
  • There is a wealth of tools and components available to make web-based development better. Being able to make changes as the app runs to the frontend in Clutch, while communicating live with the backend, is hard to give up once experienced.

Basic scaffolding

The majority the code comes from this example, the primary differences being there is no Qt UI at all in our example, instead we open a WebView, and additionally, our web source is not a local set of files. I will focus primarily on the differences, but the full code can be explored here, and the Clutch project can be found here.

Opening a WebView

Opening a WebView is straightforward enough:

from PySide6.QtWidgets import QMainWindow
from PySide6.QtWebEngineWidgets import QWebEngineView


class MainWindow(QMainWindow):
    """Create main window and load webapp."""

    def __init__(self):
        """Initialize main window."""
        super().__init__()
        self.webEngineView = QWebEngineView()
        self.setCentralWidget(self.webEngineView)
        self.webEngineView.load(
            QUrl("https://622d303119d5fca85c85049d.clutch.host/"))

QUrl, of course, can be passed any website at this point. The code Qt provides for setting up the sockets I left untouched, the only thing done was to remove Dialog and modify Core to serve as an API layer.

    core = Core()
    channel.registerObject("core", core)

The Core then becomes your "bridge", provided is some example code for Fusion:

class Core(QObject):
    """An instance of this class gets published over the WebChannel."""

    sendText = Signal(str)

    def __init__(self):
        """Initialize the QObject."""
        super().__init__()

    @Slot(str)
    def receiveText(self, text):
        """Slot to interactively send messages from client."""
        data = json.loads(text)
        action = (data["action"])

        if action == "ADD_TOOL":
            tool = data["payload"]["tool"]

            comp = fusion.GetCurrentComp()
            comp.AddTool(tool)
            self.sendText.emit(
                json.dumps({"data": {},
                            "status": 'success',
                            "message": 'Successfully added a tool'}))

The client sends messages as JSON, and receiveText acts as a dispatcher of sorts. For example, the above code is setup to handle the client sending the following JSON:

{
  "action": "ADD_TOOL",
  "payload": {
    "tool":"Blur" /*any valid Fusion tool node  */
  }
}

This of course can expand to any complexity: IMPORT_ALL_PROJECT_ASSETS, CREATE_AND_REPLACE_LORES_BACKPLATES, skies the limit. Also seen above is the ability to emit messages.

The client

The client itself is a React application built in Clutch. At the heart of it is a component, QWebChannel, with a simple interface. It accepts a webSocketAddress, an onMessage event, and passes down to its children the loading state, an error state, the websocket address, the Core object, and a simple function, sendMessage to push messages from the client. As you'll see, sendMessage and onMessage, are the main components to talking between the frontend and PySide application. The whole core is passed down for advanced uses, but sendMessage is surfaced and aliased.

Here's the code to QWebChannel, it can also be seen in Clutch by right clicking the component in the tree and selecting "Edit Definition". Tip: In Clutch the v and b keys toggle between interaction and selection mode, use this to switch from selecting items in the canvas to interacting with them as a client.

import { useState, useLayoutEffect } from 'react';
import { QWebChannel } from 'qwebchannel';

export default function QWebChannelComponent({
  webSocketAddress = 'ws://localhost:12345',
  children,
  onMessage,
}) {
  const [loading, setLoading] = useState(true);
  const [error, setError] = useState(null);
  const [core, setCore] = useState({});

  useLayoutEffect(() => {
    const socket = new WebSocket(webSocketAddress);

    socket.onerror = function (error) {
      setLoading(false);
      setError(error);
    };

    socket.onopen = function () {
      setLoading(false);
      setError(null);

      new QWebChannel(socket, function (channel) {
        setCore(channel.objects.core);

        channel.objects.core.sendText.connect(function (message) {
          onMessage && onMessage(message);
        });
      });
    };

    socket.onclose = function () {
      setLoading(false);
    };
  }, []);

  return typeof children === 'function'
    ? children({
        loading,
        error,
        webSocketAddress,
        core,
        sendMessage: core?.receiveText,
      })
    : null;
}

The onMessage property of QWebChannel is currently set to echo the servers response in the form of a Snackbar, but of course anything can be done here. More complex use cases can be achieved by switching the control to a module (accesible from the more options button on the property label itself), allowing users to pull in third-party libraries.

(json) => {
  const payload = JSON.parse(json);
  enqueueSnackbar
(payload.message, { variant: payload.status})
}

Similarly, the Button component's onClick property shows how a JSON payload can be sent over the wire back to the server.

() => sendMessage
(JSON.stringify({action: "ADD_TOOL", payload: {tool: "Blur"}}))

As mentioned previously, this setup leads to a wonderful development experience, as the frames in Clutch can connect to the running QWebWocketServer.

Clutch pushing commands into Fusion

Comparison to built-in web alternatives

As mentioned, Fusion now includes web-based panels as electron apps. The differences here are subtle but worth pointing out. That Fusion is the host is irrelevant here, so the ability to handle this solution via Python and transport it to Blender, Maya, the countless other VFX tools is valuable in and of itself. Additionally, however, in Fusion, the script is responsible for bringing its client-side file structure, this includes a node_modules folder, an index, etc. This brings into the frontend experience an additional complexity of managing the build systems across potentially many machines and configurations.

Other considerations

At a studio-level there is also the simple fact that finding frontend developers experienced in React / Webdev is easier, and they are already accustomed to working through API's. A frontend/backend model that has served the webspace well could potentially also be applied to the development process of internal tooling. As a single frontend could be created across teams and tools that bridges to different hosts depending on the current context, opportunities arise for a front-end team to exist, to unify the experience of artists across an entire studio regardless of the tools they use.

Resources

©2022