Part.1 How to implement a compositor based on wlroots

Noun Concepts#

Qt#

Qt (/ˈkjuːt/, pronounced like "cute") is a cross-platform C++ application development framework. It is widely used for developing GUI applications, in which case it is referred to as a widget toolkit. It can also be used for developing non-GUI applications, such as console tools and servers.

wlroots#

A modular toolkit for building Wayland compositors, simplifying the development of approximately 60,000 lines of code.

Provides an abstract backend for underlying display and input, supporting KMS/DRM, libinput, Wayland, X11, etc., which can be dynamically created and destroyed.
Implements various Wayland interfaces, supports protocol extensions, and promotes compositor standardization.
Provides general compositor components, such as physical space output management.
Integrates Xwayland abstraction, simplifying X11 window management.
Provides renderer abstraction, supporting simple and custom rendering needs.

seat#

Composed of all hardware devices assigned to a specific working scenario. It contains at least one graphics device, usually along with a keyboard and mouse. Additionally, it may include devices such as cameras and sound cards. A seat is identified by a seat name, which is a short string (no more than 64 characters) that starts with the four characters "seat," followed by at least one character in the range a-zA-Z0-9, or "_" and "-". This naming convention is suitable for use in filenames. Seat names may be stable or unstable; if the seat becomes available again, its name can be reused.

RHI#

RHI stands for Renderer Hardware Interface, which is a set of abstractions for hardware. The upper layer only needs to set parameters, and we do not need to care about whether the lower layer uses OpenGL, Vulkan, DX12, or Metal.

Qt6 provides QRHI, which offers a low-level hardware abstraction for Qt applications, allowing the upper layer QtQuick components to automatically call the corresponding driver interfaces during GPU rendering.

QPA#

Qt Platform Abstraction (QPA) is the core platform abstraction layer in Qt.

The QPA API can be identified by the class prefix "QPlatform*" and is used to implement advanced classes in Qt GUI. For example, QPlatformWindow is used for window system integration, while QPlatformTheme and QStyleHint are used for deep platform themes and integration.

Basic Workflow#

Treeland uses QQuickWindow as the root for rendering, so when developing in Treeland, it is like developing a regular Qt application: first create a Window, then create Qt controls within the Window, and use QEvent to handle various events.

So how does Treeland achieve this?

The private class of QQuickWindow provides an interface for customizing QQuickGraphicsDevice objects, and QQuickGraphicsDevice can create new objects using fromOpenGLContext and fromPhyicalDevice. Therefore, Treeland only needs to inherit QQuickWindow and obtain the OpenGL context and physical device from wlroots, allowing the rendering of Qt QuickWindow to be grafted onto wlroots.

By combining the rendering context of wlroots with the rendering context of Qt, the graphical results rendered by wlroots can be embedded into the rendering process of the Qt application, allowing direct use of graphical resources and device objects provided by wlroots, such as physical devices (phdev), logical devices (dev), and queue families (queue_family), to reduce unnecessary context switches and resource copies. In this way, Qt can leverage the rendering capabilities provided by wlroots while continuing to use Qt's rendering framework and API.

Subsequently, in Qt QPA, screen information and input information are converted into internal Qt objects, thus continuing to process using Qt's own event loop and other mechanisms.

Qt QPA#

QPA provides Qt with cross-platform interface abstraction capabilities, allowing us to provide our own QPA plugins to give Qt applications new capabilities, such as converting wlroots input events into internal Qt events.

Input Event Handling#

The process of Treeland handling low-level events and upper-level events

bool WOutputRenderWindow::event(QEvent *event)
{
    Q_D(WOutputRenderWindow);

    if (event->type() == doRenderEventType) {
        QCoreApplication::removePostedEvents(this, doRenderEventType);
        d_func()->doRender();
        return true;
    }

    if (QW::RenderWindow::beforeDisposeEventFilter(this, event)) {
        event->accept();
        QW::RenderWindow::afterDisposeEventFilter(this, event);
        return true;
    }

    bool isAccepted = QQuickWindow::event(event);
    if (QW::RenderWindow::afterDisposeEventFilter(this, event))
        return true;

    return isAccepted;
}

In the event handling of WOutputRenderWindow, the seat's event filter is additionally called to ensure that the compositor can intercept some events, such as intercepting some key presses and not sending them to the client.

bool QWlrootsRenderWindow::beforeDisposeEventFilter(QEvent *event)
{
    if (event->isInputEvent()) {
        auto ie = static_cast<QInputEvent*>(event);
        auto device = WInputDevice::from(ie->device());
        Q_ASSERT(device);
        Q_ASSERT(device->seat());
        lastActiveCursor = device->seat()->cursor();
        return device->seat()->filterEventBeforeDisposeStage(window(), ie);
    }

    return false;
}

This code demonstrates the functionality of converting input devices, determining the type of input device, and creating the corresponding QInputDevice object.

QPointer<QInputDevice> QWlrootsIntegration::addInputDevice(WInputDevice *device, const QString &seatName)
{
    QPointer<QInputDevice> qtdev;
    auto qwDevice = device->handle();
    const QString name = QString::fromUtf8(qwDevice->handle()->name);
    qint64 systemId = reinterpret_cast<qint64>(device);

    switch (qwDevice->handle()->type) {
    case WLR_INPUT_DEVICE_KEYBOARD: {
        qtdev = new QInputDevice(name, systemId, QInputDevice::DeviceType::Keyboard, seatName);
        break;
    }
    case WLR_INPUT_DEVICE_POINTER: {
        qtdev = new QPointingDevice(name, systemId, QInputDevice::DeviceType::TouchPad, QPointingDevice::PointerType::Generic,
                                    QInputDevice::Capability::Position | QInputDevice::Capability::Hover
                                        | QInputDevice::Capability::Scroll | QInputDevice::Capability::MouseEmulation,
                                    10, 32, seatName, QPointingDeviceUniqueId());
        break;
    }
    case WLR_INPUT_DEVICE_TOUCH: {
        qtdev = new QPointingDevice(name, systemId, QInputDevice::DeviceType::TouchScreen, QPointingDevice::PointerType::Finger,
                                    QInputDevice::Capability::Position | QInputDevice::Capability::Area | QInputDevice::Capability::MouseEmulation,
                                    10, 32, seatName, QPointingDeviceUniqueId());
        break;
    }
    case WLR_INPUT_DEVICE_TABLET_TOOL: {
        qtdev = new QPointingDevice(name, systemId, QInputDevice::DeviceType::Stylus, QPointingDevice::PointerType::Pen,
                                    QInputDevice::Capability::XTilt | QInputDevice::Capability::YTilt | QInputDevice::Capability::Pressure,
                                    1, 32, seatName, QPointingDeviceUniqueId());
        break;
    }
    case WLR_INPUT_DEVICE_TABLET_PAD: {
        auto pad = wlr_tablet_pad_from_input_device(qwDevice->handle());
        qtdev = new QPointingDevice(name, systemId, QInputDevice::DeviceType::TouchPad, QPointingDevice::PointerType::Pen,
                                    QInputDevice::Capability::Position | QInputDevice::Capability::Hover | QInputDevice::Capability::Pressure,
                                    1, pad->button_count, seatName, QPointingDeviceUniqueId());
        break;
    }
    case WLR_INPUT_DEVICE_SWITCH: {
        qtdev = new QInputDevice(name, systemId, QInputDevice::DeviceType::Keyboard, seatName);
        break;
    }
    }

    if (qtdev) {
        device->setQtDevice(qtdev);
        QWindowSystemInterface::registerInputDevice(qtdev);

        if (qtdev->type() == QInputDevice::DeviceType::Mouse || qtdev->type() == QInputDevice::DeviceType::TouchPad) {
            auto primaryQtDevice = QPointingDevice::primaryPointingDevice();
            if (!WInputDevice::from(primaryQtDevice)) {
                // Ensure the primary pointing device is the WInputDevice
                auto pd = const_cast<QPointingDevice*>(primaryQtDevice);
                pd->setParent(nullptr);
                delete pd;
            }
            Q_ASSERT(WInputDevice::from(QPointingDevice::primaryPointingDevice()));
        } else if (qtdev->type() == QInputDevice::DeviceType::Keyboard) {
            auto primaryQtDevice = QInputDevice::primaryKeyboard();
            if (!WInputDevice::from(primaryQtDevice)) {
                // Ensure the primary keyboard device is the WInputDevice
                auto pd = const_cast<QInputDevice*>(primaryQtDevice);
                pd->setParent(nullptr);
                delete pd;
            }
            Q_ASSERT(WInputDevice::from(QInputDevice::primaryKeyboard()));
        }
    }

    return qtdev;
}

Client Events#

In Treeland, there is another type of event that needs to be handled. When a user clicks on a window, the compositor needs to inform the client which coordinates were clicked. Or when input is made using the keyboard, the client needs to be informed of the input content.

First, Treeland marks a window as the active window and sets it to the seat, so that wlroots knows which window currently has focus.

Then, when keyboard input events occur, if Treeland does not filter out key events or allows certain keys, the remaining input events will be sent to the active client in the wseat's sendEvent.

// for keyboard event
inline bool doNotifyKey(WInputDevice *device, uint32_t keycode, uint32_t state, uint32_t timestamp) {
    if (!keyboardFocusSurface())
        return false;

    q_func()->setKeyboard(device);
    /* Send modifiers to the client. */
    this->handle()->keyboard_notify_key(timestamp, keycode, state);
    return true;
}

Screen Information#

In QPA, WOutput is also encapsulated as QWlrootsScreen.

QWlrootsScreen *QWlrootsIntegration::addScreen(WOutput *output)
{
    m_screens << new QWlrootsScreen(output);

    if (isMaster()) {
        QWindowSystemInterface::handleScreenAdded(m_screens.last());

        if (m_placeholderScreen) {
            QWindowSystemInterface::handleScreenRemoved(m_placeholderScreen.release());
        }
    } else {
        Q_UNUSED(new QScreen(m_screens.last()))
    }

    m_screens.last()->initialize();
    output->setScreen(m_screens.last());

    return m_screens.last();
}

QWlrootsScreen inherits from QPlatformScreen and converts some parameters, such as physicalSize, devicePixelRatio, DPI, etc., and then adds the created QWlrootsScreen into Qt using QWindowSystemInterface::handleScreenAdded.

Qt RHI#

Here is an excerpt from the initialization code for Qt RHI in waylib.

bool WOutputRenderWindowPrivate::initRCWithRhi()
{
    W_Q(WOutputRenderWindow);

    QQuickRenderControlPrivate *rcd = QQuickRenderControlPrivate::get(rc());
    QSGRhiSupport *rhiSupport = QSGRhiSupport::instance();

// sanity check for Vulkan
#ifdef ENABLE_VULKAN_RENDER
    if (rhiSupport->rhiBackend() == QRhi::Vulkan) {
        vkInstance.reset(new QVulkanInstance());

        auto phdev = wlr_vk_renderer_get_physical_device(m_renderer->handle());
        auto dev = wlr_vk_renderer_get_device(m_renderer->handle());
        auto queue_family = wlr_vk_renderer_get_queue_family(m_renderer->handle());

#if QT_VERSION > QT_VERSION_CHECK(6, 6, 0)
        auto instance = wlr_vk_renderer_get_instance(m_renderer->handle());
        vkInstance->setVkInstance(instance);
#endif
        //        vkInstance->setExtensions(fromCStyleList(vkRendererAttribs.extension_count, vkRendererAttribs.extensions));
        //        vkInstance->setLayers(fromCStyleList(vkRendererAttribs.layer_count, vkRendererAttribs.layers));
        vkInstance->setApiVersion({1, 1, 0});
        vkInstance->create();
        q->setVulkanInstance(vkInstance.data());

        auto gd = QQuickGraphicsDevice::fromDeviceObjects(phdev, dev, queue_family);
        q->setGraphicsDevice(gd);
    } else
#endif
    if (rhiSupport->rhiBackend() == QRhi::OpenGLES2) {
        Q_ASSERT(wlr_renderer_is_gles2(m_renderer->handle()));
        auto egl = wlr_gles2_renderer_get_egl(m_renderer->handle());
        auto display = wlr_egl_get_display(egl);
        auto context = wlr_egl_get_context(egl);

        this->glContext = new QW::OpenGLContext(display, context, rc());
        bool ok = this->glContext->create();
        if (!ok)
            return false;

        q->setGraphicsDevice(QQuickGraphicsDevice::fromOpenGLContext(this->glContext));
    } else {
        return false;
    }

    QOffscreenSurface *offscreenSurface = new QW::OffscreenSurface(nullptr, q);
    offscreenSurface->create();

    QSGRhiSupport::RhiCreateResult result = rhiSupport->createRhi(q, offscreenSurface);
    if (!result.rhi) {
        qWarning("WOutput::initRhi: Failed to initialize QRhi");
        return false;
    }

    rcd->rhi = result.rhi;
    // Ensure the QQuickRenderControl don't reinit the RHI
    rcd->ownRhi = true;
    if (!rc()->initialize())
        return false;
    rcd->ownRhi = result.own;
    Q_ASSERT(rcd->rhi == result.rhi);
    Q_ASSERT(!swapchain);

    return true;
}

First, obtain QSGRhiSupport and related control objects.

Determine the type of RHI backend, which needs to adapt to vulkan, gles, etc.

Obtain physical device parameters from wlroots, and use QQuickGraphicsDevice::fromDeviceObjects to create Qt's QQuickGraphicsDevice.

The private class of the render window inherits from QQuickWindowPrivate, and it only needs to set the obtained QQuickGraphicsDevice to QQuickWindowPrivate::setGraphicsDevice.

Then create an offscreen rendering surface for RHI initialization.

Qt Viewport#

In Qt, to view or render a component, the Viewport component, commonly known as a camera, is used.

A viewport is an observable polygonal area, and only the content within the Viewport can be displayed on the screen.

The Viewport in wlroots is a concept related to the Wayland display protocol, mainly used to define the display area of rendering output on the screen. It allows for scaling, cropping, or translating the displayed content during rendering to accommodate different resolutions and display requirements.

Treeland uses WOutputViewport to provide viewport functionality, using the screen information in wlroots to perform matrix transformations on the content, which involves parameters such as screen scaling and DPI.

QMatrix4x4 WOutputViewport::renderMatrix() const
{
    QMatrix4x4 renderMatrix;

    if (auto customTransform = viewportTransform()) {
        customTransform->applyTo(&renderMatrix);
    } else if (parentItem() && !ignoreViewport() && input() != this) {
        auto d = QQuickItemPrivate::get(const_cast<WOutputViewport*>(this));
        auto viewportMatrix = d->itemNode()->matrix().inverted();
        if (auto inputItem = input()) {
            QMatrix4x4 matrix = QQuickItemPrivate::get(parentItem())->itemToWindowTransform();
            matrix *= QQuickItemPrivate::get(inputItem)->windowToItemTransform();
            renderMatrix = viewportMatrix * matrix.inverted();
        } else { // the input item is window's contentItem
            auto pd = QQuickItemPrivate::get(parentItem());
            QMatrix4x4 parentMatrix = pd->itemToWindowTransform().inverted();
            renderMatrix = viewportMatrix * parentMatrix;
        }
    }

    return renderMatrix;
}

WOutputViewport provides all the parameters required for the viewport, such as transformation matrices, source geometry size, target geometry size, etc.

In the event of WOutputRenderWindow, it checks if the event is a rendering event, then executes the rendering.

bool WOutputRenderWindow::event(QEvent *event)
{
    Q_D(WOutputRenderWindow);

    if (event->type() == doRenderEventType) {
        QCoreApplication::removePostedEvents(this, doRenderEventType);
        d_func()->doRender();
        return true;
    }

    if (QW::RenderWindow::beforeDisposeEventFilter(this, event)) {
        event->accept();
        QW::RenderWindow::afterDisposeEventFilter(this, event);
        return true;
    }

    bool isAccepted = QQuickWindow::event(event);
    if (QW::RenderWindow::afterDisposeEventFilter(this, event))
        return true;

    return isAccepted;
}

In doRender, it iterates through all Outputs, executes beginRender, and then performs the rendering of the Output.

void WOutputRenderWindowPrivate::doRender(const QList<OutputHelper *> &outputs,
                                          bool forceRender, bool doCommit)
{
    Q_ASSERT(rendererList.isEmpty());
    Q_ASSERT(!inRendering);
    inRendering = true;

    W_Q(WOutputRenderWindow);
    for (OutputLayer *layer : std::as_const(layers)) {
        layer->beforeRender(q);
    }

    rc()->polishItems();

    if (QSGRendererInterface::isApiRhiBased(WRenderHelper::getGraphicsApi()))
        rc()->beginFrame();
    rc()->sync();

    QQuickAnimatorController_advance(animationController.get());
    Q_EMIT q->beforeRendering();
    runAndClearJobs(&beforeRenderingJobs);

    auto needsCommit = doRenderOutputs(outputs, forceRender);

    Q_EMIT q->afterRendering();
    runAndClearJobs(&afterRenderingJobs);

    if (QSGRendererInterface::isApiRhiBased(WRenderHelper::getGraphicsApi()))
        rc()->endFrame();

    if (doCommit) {
        for (auto i : std::as_const(needsCommit)) {
            bool ok = i.first->commit(i.second);

            if (i.second->currentBuffer()) {
                i.second->endRender();
            }

            i.first->resetState(ok);
        }
    }

    resetGlState();

    // On Intel&Nvidia multi-GPU environment, wlroots using Intel card do render for all
    // outputs, and blit nvidia's output buffer in drm_connector_state_update_primary_fb,
    // the 'blit' behavior will make EGL context to Nvidia renderer. So must done current
    // OpenGL context here in order to ensure QtQuick always make EGL context to Intel
    // renderer before next frame.
    if (glContext)
        glContext->doneCurrent();

    inRendering = false;
    Q_EMIT q->renderEnd();
}

qw_buffer *WBufferRenderer::beginRender(const QSize &pixelSize, qreal devicePixelRatio,
                                        uint32_t format, RenderFlags flags)
{
    Q_ASSERT(!state.buffer);
    Q_ASSERT(m_output);

    if (pixelSize.isEmpty())
        return nullptr;

    Q_EMIT beforeRendering();

    m_damageRing.set_bounds(pixelSize.width(), pixelSize.height());

    // configure swapchain
    if (flags.testFlag(RenderFlag::DontConfigureSwapchain)) {
        auto renderFormat = pickFormat(m_output->renderer(), format);
        if (!renderFormat) {
            qWarning("wlr_renderer doesn't support format 0x%s", drmGetFormatName(format));
            return nullptr;
        }

        if (!m_swapchain || QSize(m_swapchain->handle()->width, m_swapchain->handle()->height) != pixelSize
            || m_swapchain->handle()->format.format != renderFormat->format) {
            if (m_swapchain)
                delete m_swapchain;
            m_swapchain = qw_swapchain::create(m_output->allocator()->handle(), pixelSize.width(), pixelSize.height(), renderFormat);
        }
    } else if (flags.testFlag(RenderFlag::UseCursorFormats)) {
        bool ok = m_output->configureCursorSwapchain(pixelSize, format, &m_swapchain);
        if (!ok)
            return nullptr;
    } else {
        bool ok = m_output->configurePrimarySwapchain(pixelSize, format, &m_swapchain,
                                                      !flags.testFlag(DontTestSwapchain));
        if (!ok)
            return nullptr;
    }

    // TODO: Support scanout buffer of wlr_surface(from WSurfaceItem)
    int bufferAge;
    auto wbuffer = m_swapchain->acquire(&bufferAge);
    if (!wbuffer)
        return nullptr;
    auto buffer = qw_buffer::from(wbuffer);

    if (!m_renderHelper)
        m_renderHelper = new WRenderHelper(m_output->renderer());
    m_renderHelper->setSize(pixelSize);

    auto wd = QQuickWindowPrivate::get(window());
    Q_ASSERT(wd->renderControl);
    auto lastRT = m_renderHelper->lastRenderTarget();
    auto rt = m_renderHelper->acquireRenderTarget(wd->renderControl, buffer);
    if (rt.isNull()) {
        buffer->unlock();
        return nullptr;
    }

    auto rtd = QQuickRenderTargetPrivate::get(&rt);
    QSGRenderTarget sgRT;

    if (rtd->type == QQuickRenderTargetPrivate::Type::PaintDevice) {
        sgRT.paintDevice = rtd->u.paintDevice;
    } else {
        Q_ASSERT(rtd->type == QQuickRenderTargetPrivate::Type::RhiRenderTarget);
        sgRT.rt = rtd->u.rhiRt;
        sgRT.cb = wd->redirect.commandBuffer;
        Q_ASSERT(sgRT.cb);
        sgRT.rpDesc = rtd->u.rhiRt->renderPassDescriptor();

#ifndef QT_NO_OPENGL
        if (wd->rhi->backend() == QRhi::OpenGLES2) {
            auto glRT = QRHI_RES(QGles2TextureRenderTarget, rtd->u.rhiRt);
            Q_ASSERT(glRT->framebuffer >= 0);
            auto glContext = QOpenGLContext::currentContext();
            Q_ASSERT(glContext);
            QOpenGLContextPrivate::get(glContext)->defaultFboRedirect = glRT->framebuffer;
        }
#endif
    }

    state.flags = flags;
    state.context = wd->context;
    state.pixelSize = pixelSize;
    state.devicePixelRatio = devicePixelRatio;
    state.bufferAge = bufferAge;
    state.lastRT = lastRT;
    state.buffer = buffer;
    state.renderTarget = rt;
    state.sgRenderTarget = sgRT;

    return buffer;
}

QVector<std::pair<OutputHelper*, WBufferRenderer*>>
WOutputRenderWindowPrivate::doRenderOutputs(const QList<OutputHelper*> &outputs, bool forceRender)
{
    QVector<OutputHelper*> renderResults;
    renderResults.reserve(outputs.size());
    for (OutputHelper *helper : std::as_const(outputs)) {
        if (Q_LIKELY(!forceRender)) {
            if (!helper->renderable()
                || Q_UNLIKELY(!WOutputViewportPrivate::get(helper->output())->renderable())
                || !helper->output()->output()->isEnabled())
                continue;

            if (!helper->contentIsDirty()) {
                if (helper->needsFrame())
                    renderResults.append(helper);
                continue;
            }
        }

        Q_ASSERT(helper->output()->output()->scale() <= helper->output()->devicePixelRatio());

        const auto &format = helper->qwoutput()->handle()->render_format;
        const auto renderMatrix = helper->output()->renderMatrix();

        // maybe using the other WOutputViewport's QSGTextureProvider
        if (!helper->output()->depends().isEmpty())
            updateDirtyNodes();

        qw_buffer *buffer = helper->beginRender(helper->bufferRenderer(), helper->output()->output()->size(), format,
                                                WBufferRenderer::RedirectOpenGLContextDefaultFrameBufferObject);
        Q_ASSERT(buffer == helper->bufferRenderer()->currentBuffer());
        if (buffer) {
            helper->render(helper->bufferRenderer(), 0, renderMatrix,
                           helper->output()->effectiveSourceRect(),
                           helper->output()->targetRect(),
                           helper->output()->preserveColorContents());
        }
        renderResults.append(helper);
    }

    QVector<std::pair<OutputHelper*, WBufferRenderer*>> needsCommit;
    needsCommit.reserve(renderResults.size());
    for (auto helper : std::as_const(renderResults)) {
        auto bufferRenderer = helper->afterRender();
        if (bufferRenderer)
            needsCommit.append({helper, bufferRenderer});
    }

    rendererList.clear();

    return needsCommit;
}

void WBufferRenderer::render(int sourceIndex, const QMatrix4x4 &renderMatrix,
                             const QRectF &sourceRect, const QRectF &targetRect,
                             bool preserveColorContents)
{
    Q_ASSERT(state.buffer);

    const auto &source = m_sourceList.at(sourceIndex);
    QSGRenderer *renderer = ensureRenderer(sourceIndex, state.context);
    auto wd = QQuickWindowPrivate::get(window());

    const qreal devicePixelRatio = state.devicePixelRatio;
    state.renderer = renderer;
    state.worldTransform = renderMatrix;
    renderer->setDevicePixelRatio(devicePixelRatio);
    renderer->setDeviceRect(QRect(QPoint(0, 0), state.pixelSize));
    renderer->setRenderTarget(state.sgRenderTarget);
    const auto viewportRect = scaleToRect(targetRect, devicePixelRatio);

    auto softwareRenderer = dynamic_cast<QSGSoftwareRenderer*>(renderer);
    { // before render
        if (softwareRenderer) {
            // because software renderer don't supports viewportRect,
            // so use transform to simulation.
            const auto mapTransform = inputMapToOutput(sourceRect, targetRect,
                                                       state.pixelSize, state.devicePixelRatio);
            if (!mapTransform.isIdentity())
                state.worldTransform = mapTransform * state.worldTransform;
            state.worldTransform.optimize();
            auto image = getImageFrom(state.renderTarget);
            image->setDevicePixelRatio(devicePixelRatio);

            // TODO: Should set to QSGSoftwareRenderer, but it's not support specify matrix.
            // If transform is changed, it will full repaint.
            if (isRootItem(source.source)) {
                auto rootTransform = QQuickItemPrivate::get(wd->contentItem)->itemNode();
                if (rootTransform->matrix() != state.worldTransform)
                    rootTransform->setMatrix(state.worldTransform);
            } else {
                auto t = state.worldTransform.toTransform();
                if (t.type() > QTransform::TxTranslate) {
                    (image->operator QImage &()).fill(renderer->clearColor());
                    softwareRenderer->markDirty();
                }

                applyTransform(softwareRenderer, t);
            }
        } else {
            state.worldTransform.optimize();

            bool flipY = wd->rhi ? !wd->rhi->isYUpInNDC() : false;
            if (state.renderTarget.mirrorVertically())
                flipY = !flipY;

            if (viewportRect.isValid()) {
                QRect vr = viewportRect;
                if (flipY)
                    vr.moveTop(-vr.y() + state.pixelSize.height() - vr.height());
                renderer->setViewportRect(vr);
            } else {
                renderer->setViewportRect(QRect(QPoint(0, 0), state.pixelSize));
            }

            QRectF rect = sourceRect;
            if (!rect.isValid())
                rect = QRectF(QPointF(0, 0), QSizeF(state.pixelSize) / devicePixelRatio);

            const float left = rect.x();
            const float right = rect.x() + rect.width();
            float bottom = rect.y() + rect.height();
            float top = rect.y();

            if (flipY)
                std::swap(top, bottom);

            QMatrix4x4 matrix;
            matrix.ortho(left, right, bottom, top, 1, -1);

            QMatrix4x4 projectionMatrix, projectionMatrixWithNativeNDC;
            projectionMatrix = matrix * state.worldTransform;

            if (wd->rhi && !wd->rhi->isYUpInNDC()) {
                std::swap(top, bottom);

                matrix.setToIdentity();
                matrix.ortho(left, right, bottom, top, 1, -1);
            }
            projectionMatrixWithNativeNDC = matrix * state.worldTransform;

            renderer->setProjectionMatrix(projectionMatrix);
            renderer->setProjectionMatrixWithNativeNDC(projectionMatrixWithNativeNDC);

            auto textureRT = static_cast<QRhiTextureRenderTarget*>(state.sgRenderTarget.rt);
            if (preserveColorContents) {
                textureRT->setFlags(textureRT->flags() | QRhiTextureRenderTarget::PreserveColorContents);
            } else {
                textureRT->setFlags(textureRT->flags() & ~QRhiTextureRenderTarget::PreserveColorContents);
            }
        }
    }

    state.context->renderNextFrame(renderer);

    { // after render
        if (!softwareRenderer) {
            // TODO: get damage area from QRhi renderer
            m_damageRing.add_whole();
            // ###: maybe Qt bug? Before executing QRhi::endOffscreenFrame, we may
            // use the same QSGRenderer for multiple drawings. This can lead to
            // rendering the same content for different QSGRhiRenderTarget instances
            // when using the RhiGles backend. Additionally, considering that the
            // result of the current drawing may be needed when drawing the next
            // sourceIndex, we should let the RHI (Rendering Hardware Interface)
            // complete the results of this drawing here to ensure the current
            // drawing result is available for use.
            wd->rhi->finish();
        } else {
            auto currentImage = getImageFrom(state.renderTarget);
            Q_ASSERT(currentImage && currentImage == softwareRenderer->m_rt.paintDevice);
            currentImage->setDevicePixelRatio(1.0);
            const auto scaleTF = QTransform::fromScale(devicePixelRatio, devicePixelRatio);
            const auto scaledFlushRegion = scaleTF.map(softwareRenderer->flushRegion());
            PixmanRegion scaledFlushDamage;
            bool ok = WTools::toPixmanRegion(scaledFlushRegion, scaledFlushDamage);
            Q_ASSERT(ok);

            {
                PixmanRegion damage;
                m_damageRing.get_buffer_damage(state.bufferAge, damage);

                if (viewportRect.isValid()) {
                    QRect imageRect = (currentImage->operator const QImage &()).rect();
                    QRegion invalidRegion(imageRect);
                    invalidRegion -= viewportRect;
                    if (!scaledFlushRegion.isEmpty())
                        invalidRegion &= scaledFlushRegion;

                    if (!invalidRegion.isEmpty()) {
                        QPainter pa(currentImage);
                        for (const auto r : std::as_const(invalidRegion))
                            pa.fillRect(r, softwareRenderer->clearColor());
                    }
                }

                if (!damage.isEmpty() && state.lastRT.first != state.buffer && !state.lastRT.second.isNull()) {
                    auto image = getImageFrom(state.lastRT.second);
                    Q_ASSERT(image);
                    Q_ASSERT(image->size() == state.pixelSize);

                    // TODO: Don't use the previous render target, we can get the damage region of QtQuick
                    // before QQuickRenderControl::render for qw_damage_ring, and add dirty region to
                    // QSGAbstractSoftwareRenderer to force repaint the damage region of current render target.
                    QPainter pa(currentImage);

                    PixmanRegion remainderDamage;
                    ok = pixman_region32_subtract(remainderDamage, damage, scaledFlushDamage);
                    Q_ASSERT(ok);

                    int count = 0;
                    auto rects = pixman_region32_rectangles(remainderDamage, &count);
                    for (int i = 0; i < count; ++i) {
                        auto r = rects[i];
                        pa.drawImage(r.x1, r.y1, *image, r.x1, r.y1, r.x2 - r.x1, r.y2 - r.y1);
                    }
                }
            }

            if (!isRootItem(source.source))
                applyTransform(softwareRenderer, state.worldTransform.inverted().toTransform());
            m_damageRing.add(scaledFlushDamage);
        }
    }

    if (auto dr = qobject_cast<QSGDefaultRenderContext*>(state.context)) {
        QRhiResourceUpdateBatch *resourceUpdates = wd->rhi->nextResourceUpdateBatch();
        dr->currentFrameCommandBuffer()->resourceUpdate(resourceUpdates);
    }

    if (shouldCacheBuffer())
        wTextureProvider()->setBuffer(state.buffer);
}

After processing the image, if the screen needs to be updated, the commit is called to send the image to the screen.

bool OutputHelper::commit(WBufferRenderer *buffer)
{
    if (output()->offscreen())
        return true;

    if (!buffer || !buffer->currentBuffer()) {
        Q_ASSERT(!this->buffer());
        return WOutputHelper::commit();
    }

    setBuffer(buffer->currentBuffer());

    if (m_lastCommitBuffer == buffer) {
        if (pixman_region32_not_empty(&buffer->damageRing()->handle()->current))
            setDamage(&buffer->damageRing()->handle()->current);
    }

    m_lastCommitBuffer = buffer;

    return WOutputHelper::commit();
}

It will also check for hardware acceleration (GPU) and prioritize using hardware to accelerate the computation process.

} else {
    state.worldTransform.optimize();

    bool flipY = wd->rhi ? !wd->rhi->isYUpInNDC() : false;
    if (state.renderTarget.mirrorVertically())
        flipY = !flipY;

    if (viewportRect.isValid()) {
        QRect vr = viewportRect;
        if (flipY)
            vr.moveTop(-vr.y() + state.pixelSize.height() - vr.height());
        renderer->setViewportRect(vr);
    } else {
        renderer->setViewportRect(QRect(QPoint(0, 0), state.pixelSize));
    }

    QRectF rect = sourceRect;
    if (!rect.isValid())
        rect = QRectF(QPointF(0, 0), QSizeF(state.pixelSize) / devicePixelRatio);

    const float left = rect.x();
    const float right = rect.x() + rect.width();
    float bottom = rect.y() + rect.height();
    float top = rect.y();

    if (flipY)
        std::swap(top, bottom);

    QMatrix4x4 matrix;
    matrix.ortho(left, right, bottom, top, 1, -1);

    QMatrix4x4 projectionMatrix, projectionMatrixWithNativeNDC;
    projectionMatrix = matrix * state.worldTransform;

    if (wd->rhi && !wd->rhi->isYUpInNDC()) {
        std::swap(top, bottom);

        matrix.setToIdentity();
        matrix.ortho(left, right, bottom, top, 1, -1);
    }
    projectionMatrixWithNativeNDC = matrix * state.worldTransform;

    renderer->setProjectionMatrix(projectionMatrix);
    renderer->setProjectionMatrixWithNativeNDC(projectionMatrixWithNativeNDC);

    auto textureRT = static_cast<QRhiTextureRenderTarget*>(state.sgRenderTarget.rt);
    if (preserveColorContents) {
        textureRT->setFlags(textureRT->flags() | QRhiTextureRenderTarget::PreserveColorContents);
    } else {
        textureRT->setFlags(textureRT->flags() & ~QRhiTextureRenderTarget::PreserveColorContents);
    }
}

Surface Rendering#

In Treeland, WSurfaceItem is created for the Surface to represent a window, and WSurfaceContent is created as the delegate of WSurfaceItem.

void WSurfaceItemPrivate::initForDelegate()
{
    Q_Q(WSurfaceItem);

    std::unique_ptr<QQuickItem> newContentContainer;

    if (!delegate) {
        if (getItemContent()) {
            Q_ASSERT(!delegateIsDirty);
            return;
        }

        delegateIsDirty = false;
        auto contentItem = new WSurfaceItemContent(q);
        if (surface)
            contentItem->setSurface(surface);
        contentItem->setCacheLastBuffer(!surfaceFlags.testFlag(WSurfaceItem::DontCacheLastBuffer));
        contentItem->setSmooth(q->smooth());
        contentItem->setLive(!q->flags().testFlag(WSurfaceItem::NonLive));
        QObject::connect(q, &WSurfaceItem::smoothChanged, contentItem, &WSurfaceItemContent::setSmooth);
        newContentContainer.reset(contentItem);
    } else if (delegateIsDirty) {
        auto obj = delegate->createWithInitialProperties({{"surface", QVariant::fromValue(q)}}, qmlContext(q));
        if (!obj) {
            qWarning() << "Failed on create surface item from delegate, error message:"
                       << delegate->errorString();
            return;
        }

        delegateIsDirty = false;
        auto contentItem = qobject_cast<QQuickItem*>(obj);
        if (!contentItem)
            qFatal() << "SurfaceItem's delegate must be an Item";

        newContentContainer.reset(new QQuickItem(q));
        QQmlEngine::setObjectOwnership(contentItem, QQmlEngine::CppOwnership);
        contentItem->setParent(newContentContainer.get());
        contentItem->setParentItem(newContentContainer.get());
    }

    if (!newContentContainer)
        return;

    newContentContainer->setZ(qreal(WSurfaceItem::ZOrder::ContentItem));

    if (contentContainer) {
        newContentContainer->setPosition(contentContainer->position());
        newContentContainer->setSize(contentContainer->size());
        newContentContainer->setTransformOrigin(contentContainer->transformOrigin());
        newContentContainer->setScale(contentContainer->scale());

        contentContainer->disconnect(q);
        contentContainer->deleteLater();
    }
    contentContainer = newContentContainer.release();
    updateEventItem(false);
    updateBoundingRect();
    if (eventItem)
        updateEventItemGeometry();

    Q_EMIT q->contentItemChanged();
}

Then, when WSurfaceItem needs to update the display, it can call updatePaintNode to refresh the rendering.

QSGNode *WSurfaceItemContent::updatePaintNode(QSGNode *oldNode, UpdatePaintNodeData *)
{
    W_D(WSurfaceItemContent);

    auto tp = wTextureProvider();
    if (d->live || !tp->texture()) {
        auto texture = d->surface ? d->surface->handle()->get_texture() : nullptr;
        if (texture) {
            tp->setTexture(qw_texture::from(texture), d->buffer.get());
        } else {
            tp->setBuffer(d->buffer.get());
        }
    }

    if (!tp->texture() || width() <= 0 || height() <= 0) {
        delete oldNode;
        return nullptr;
    }

    auto node = static_cast<QSGImageNode*>(oldNode);
    if (Q_UNLIKELY(!node)) {
        node = window()->createImageNode();
        node->setOwnsTexture(false);
        QSGNode *fpnode = new WSGRenderFootprintNode(this);
        node->appendChildNode(fpnode);
    }

    auto texture = tp->texture();
    node->setTexture(texture);
    const QRectF textureGeometry = d->bufferSourceBox;
    node->setSourceRect(textureGeometry);
    const QRectF targetGeometry(d->ignoreBufferOffset ? QPointF() : d->bufferOffset, size());
    node->setRect(targetGeometry);
    node->setFiltering(smooth() ? QSGTexture::Linear : QSGTexture::Nearest);

    return node;
}

The purpose of using a delegate is to allow multiple WSurfaceItem instances to share the same window content, such as when temporarily creating a clone of a window, window switching lists, multitasking views, etc.

QSGTextureProvider *WSurfaceItemContent::textureProvider() const
{
    if (QQuickItem::isTextureProvider())
        return QQuickItem::textureProvider();

    return wTextureProvider();
}

WSGTextureProvider *WSurfaceItemContent::wTextureProvider() const
{
    W_DC(WSurfaceItemContent);

    auto w = qobject_cast<WOutputRenderWindow*>(d->window);
    if (!w || !d->sceneGraphRenderContext() || QThread::currentThread() != d->sceneGraphRenderContext()->thread()) {
        qWarning("WQuickCursor::textureProvider: can only be queried on the rendering thread of an WOutputRenderWindow");
        return nullptr;
    }

    if (!d->textureProvider) {
        d->textureProvider = new WSGTextureProvider(w);
        if (d->surface) {
            if (auto texture = d->surface->handle()->get_texture()) {
                d->textureProvider->setTexture(qw_texture::from(texture), d->buffer.get());
            } else {
                d->textureProvider->setBuffer(d->buffer.get());
            }
        }
    }
    return d->textureProvider;
}

Treeland uses WQuickTextureProxy to create a proxy display for the window, which retrieves the textureProvider of WSurfaceItem.

QSGTextureProvider *WQuickTextureProxy::textureProvider() const
{
    if (QQuickItem::isTextureProvider())
        return QQuickItem::textureProvider();

    W_DC(WQuickTextureProxy);
    if (!d->sourceItem)
        return nullptr;

    return d->sourceItem->textureProvider();
}

This way, multiple proxies can display the content of the same window, which is more efficient than QML's ShaderEffectSource.

Conclusion#

The above is just a part of the process of stitching Qt and wlroots together in Treeland. In reality, event handling is quite complex, involving not only keyboard input but also cursor, touch, and other devices. Additionally, cursor rendering needs to distinguish between hard and soft cursors, as well as hardware acceleration and software implementations during rendering.

In the future, I plan to write about cursor-related processing and how Treeland's rendering is done.