Syphon for Python

⚠️ This library is still in development.

Python wrapper for the Syphon GPU texture sharing framework. This library was created to support both the Metal backend and the deprecated OpenGL backend. It requires macOS 11 or above.

The implementation is based on PyObjC to wrap the Syphon framework directly from Python. This approach eliminates native wrapper and allows Python developers to extend the library as needed.

State of Development

• Syphon Server Discovery
• Metal Server
• Metal Client
• OpenGL Server
• OpenGL Client
• Syphon Client On Frame Callback

Usage

To install syphon-python it is recommended to use a prebuilt binary from PyPi:

pip install syphon-python

To run all the examples, please also install Numpy and OpenCV:

pip install numpy opencv-python

The following code snippet is a basic example showing how to share numpy images as MTLTexture with a SyphonMetalServer. There are more examples in examples.

import time

import numpy as np

import syphon
from numpy import copy_image_to_mtl_texture
from raw import create_mtl_texture

# create server and texture
server = syphon.SyphonMetalServer("Demo")
texture = create_mtl_texture(server.device, 512, 512)

# create texture data
texture_data = np.zeros((512, 512, 4), dtype=np.uint8)
texture_data[:, :, 0] = 255  # fill red
texture_data[:, :, 3] = 255  # fill alpha

while True:
    # copy texture data to texture and publish frame
    copy_image_to_mtl_texture(texture_data, texture)
    server.publish_frame_texture(texture)
    time.sleep(1)

server.stop()

Development

To develop or manually install the library, use the following commands to set up the local repository.

Installation

# clone the repository and it's submodules
git clone --recurse-submodules https://github.com/cansik/syphon-python.git

# install dependencies
pip install -r dev-requirements.txt
pip install -r requirements.txt

# for some examples the following dependencies are needed
pip install numpy
pip install opencv-python

Build

Build the Syphon framework on your machine:

python setup.py build

Distribute

Create a wheel package (also runs build automatically)

python setup.py bdist_wheel

Generate Documentation

# create documentation into "./docs
python setup.py doc

# launch pdoc webserver
python setup.py doc --launch

About

MIT License - Copyright (c) 2023 Florian Bruggisser

Documentation

The syphon-python library is a wrapper for the Syphon framework for the Python language. It exposes the Objective-C API to the Python world and adds helper methods to easily interoperate with Syphon. Syphon is an open source Mac OS X technology that allows applications to share video and still images with each other in real time.

Why a new library?

There are already wrappers like syphonpy that do the same thing as syphon-python. There are two main reasons why syphon-python was implemented:

Modern Graphics Pipeline

Most existing Python wrappers for Syphon only support the OpenGL framework. Even though OpenGL is still available in modern MacOS versions, as of MacOS Mojave 10.14, the framework is marked as deprecated and should be avoided. And while many applications are switching to the Metal graphics backend, Syphon needs to do the same.

The Syphon framework already supports the Metal graphics backend, but the wrappers usually do not. Syphon-python adds support for Metal and retains OpenGL.

Objective-C to Python

To add support for the new Metal graphics backend, the existing wrappers could be extended. However, many of them use an intermediate wrapper in C to expose the Objective-C API to Python. For Python developers it can be difficult to extend existing C code, so syphon-python uses PyObjC, a Python to Objective-C bridge.

Syphon Package

The main package of syphon-python is called syphon and contains all the necessary objects and classes of the library. To use syphon-python in a project, start with the following import statement.

import syphon

For each Metal class there is also an OpenGL counterpart. It is worth noting that Syphon supports interopability between Metal and OpenGL. This means that it is possible to run a Metal-based Syphon server and receive it in an OpenGL client and vice versa.

Syphon Server

To share graphic textures with other applications, a syphon server (sender) has to be created. All server implementations are based on the BaseSyphonServer and share the same interface, except the constructor. The following code example creates either a Metal oder OpenGL based server, using the app name Demo and using the default device or context.

# create a Metal based server
server = syphon.SyphonMetalServer("Demo")

# create an OpenGL based server
server = syphon.SyphonOpenGLServer("Demo")

To publish a texture, the method publish_frame_texture() can be used. We assume that the corresponding texture has already been created and is available in the texture variable. To see how to create and fill a MTLTexture or glTexture, please have a look at the examples.

texture = ... # MTLTexture or glTexture
server.publish_frame_texture(texture)

The publish_frame_texture() contains a flag called flip to flip the texture horizontally. This can be set to True if the image is upside down on the receiver side.

It is also possible to check if a server has connected clients with the has_clients property.

To clean up and release allocated resources, a server should be stopped with the stop() method.

if not server.has_clients:
    server.stop()

Metal Server

On initialisation, the SyphonMetalServer creates a new system default Metal device as well as a new command queue. It is possible to override which MTLDevice the Syphon server is running on or which type of command queue is used. This can be done by using the additional parameters of the SyphonMetalServer.

import Metal

# overwrite the mtl_device
mtl_device = Metal.MTLCreateSystemDefaultDevice()
server = syphon.SyphonMetalServer("Demo", device=mtl_device)

# or also create custom command queue
mtl_command_queue = mtl_device.newCommandQueue()
server = syphon.SyphonMetalServer("Demo", device=mtl_device, command_queue=mtl_command_queue)

OpenGL Server

On initialisation, the SyphonOpenGLServer tries to find the current cglContextObj using the current NSOpenGLContext. It is possible to override the automatic lookup by passing a valid cglContextObj as a parameter to the SyphonOpenGLServer.

import AppKit

ns_ctx = AppKit.NSOpenGLContext.currentContext()
cgl_context = ns_ctx.CGLContextObj()

server = syphon.SyphonOpenGLServer("Demo", cgl_context_obj=cgl_context)

For example, if glfw is used to create an OpenGL window, it is enough to set the current context through glfw and the SyphonOpenGLServer will be able to find this context on its own.

glfw.make_context_current(window)
server = syphon.SyphonOpenGLServer("Demo")

Shared Directory

To get a list of active Syphon servers on the system, the SyphonServerDirectory can be used. The resulting list of objects is of type SyphonServerDescription.

directory = syphon.SyphonServerDirectory()
servers = directory.servers

for server in servers:
    print(f"{server.app_name} ({server.uuid})")

It is also possible to listen for events when a server changes its status. However, it is important to update the NSRunLoop to receive messages. This can be done by repeatedly calling directory.update_run_loop().

def handler(event):
    print("A new server has been announced.")


directory.add_observer(syphon.SyphonServerNotification.Announce, handler)

while True:
    directory.update_run_loop()
    time.sleep(1.0)

Syphon Client

To receive graphic textures from other applications, a syphon client (receiver) must be created. All client implementations are based on BaseSyphonClient and share the same interface except for the constructor. The following code example creates either a Metal or OpenGL based client, using the first found server description and the default device or context.

# receive the first server description
directory = syphon.SyphonServerDirectory()
server_info = directory.servers[0]

# create a Metal client
client = syphon.SyphonMetalClient(server_info)

# create an OpenGL client
client = syphon.SyphonOpenGLClient(server_info)

To get textures, it is possible to first check if the server has provided a new texture using the has_new_frame property, and then read the new frame image using the new_frame_image property.

if client.has_new_frame:
    texture = client.new_frame_image # either MTLTexture or glTexture

To stop the client and disconnect from the server, the stop() method can be used.

client.stop()

Metal Client

As with the metal server, it is possible to overwrite the device which the metal client is running on. This can be done by using the additional parameters of the SyphonMetalClient.

import Metal

# overwrite the mtl_device
mtl_device = Metal.MTLCreateSystemDefaultDevice()
client = syphon.SyphonMetalClient(server_info, device=mtl_device)

OpenGL Client

As with the opengl server, it is possible to override the automatic lookup by passing a valid cglContextObj as a parameter to the SyphonOpenGLClient.

import AppKit

ns_ctx = AppKit.NSOpenGLContext.currentContext()
cgl_context = ns_ctx.CGLContextObj()

client = syphon.SyphonOpenGLClient(server_info, cgl_context_obj=cgl_context)

Utilities

To make sharing graphic textures as easy as possible, the library provides some utility methods to manipulate texture data.

Raw

The raw module contains methods to create and manipulate textures with a raw bytes array.

Create MTLTexture

To create an MTLTexture the method create_mtl_texture can be used. It is possible to create your own default device or use a server's device property to get the current device.

import Metal
from raw import create_mtl_texture

mtl_device = Metal.MTLCreateSystemDefaultDevice()
texture = create_mtl_texture(mtl_device, 512, 512)

Manipulate MTLTexture

To write bytes to an MTLTexture the method copy_bytes_to_mtl_texture() can be used.

from raw import copy_bytes_to_mtl_texture

data = ...  # bytes() based buffer
texture = ... # MLTTexture object

copy_bytes_to_mtl_texture(data, texture)

To read bytes from an MTLTexture the method copy_mtl_texture_to_bytes() can be used.

from raw import copy_mtl_texture_to_bytes

texture = ... # MLTTexture object

data = copy_mtl_texture_to_bytes(texture) # returns bytes

Numpy

If you are working with Numpy arrays, the numpy package contains helper methods for reading and writing numpy images to and from MTLTexture.

It is important to note that the numpy package is not installed by default, it must be installed using pip install numpy.

To write a numpy image to a MTLTexture, the copy_image_to_mtl_texture() method can be used.

import numpy as np
from numpy import copy_image_to_mtl_texture

texture = ... # MLTTexture object

# create RGBA image
texture_data = np.zeros((512, 512, 4), dtype=np.uint8)

# copy image to texture
copy_image_to_mtl_texture(texture_data, texture)

To read a numpy image from a MTLTexture, the copy_mtl_texture_to_image() method can be used.

import numpy as np
from numpy import copy_mtl_texture_to_image

texture = ... # MLTTexture object

texture_data = copy_mtl_texture_to_image(texture) # returns numpy array

Python Binding

As described in the Objective-C to Python chapter, the syphon-python library is based on the PyObjC Python to Objective-C bridge. This means that there is no intermediate wrapper between Python and Objective-C, and it is possible to access and call Objective-C objects directly from Python. This can be useful if a method of the original Syphon framework has not yet been exposed by the wrapper.

Access Objective-C Objects

To access the raw Objective-C object of a SyphonMetalServer, it is possible to access the context variable. To get a list of methods that can be called, the dir() method can be used.

server = syphon.SyphonMetalServer("Demo")
objc_syphon_metal_server = server.context

print(dir(objc_syphon_metal_server))

Raw Pointers to Python Objective-C Objects

The framework expects PyObjC pointers to be passed to the methods. Sometimes only raw ctype pointers are available. This example shows how to cast a nanogui MTLTexture pointer to a PyObjC object.

import ctypes
from typing import Any

import objc
from nanogui import Screen


def get_mtl_texture(texture: Any) -> Any:
    ctypes.pythonapi.PyCapsule_GetName.restype = ctypes.c_char_p
    ctypes.pythonapi.PyCapsule_GetName.argtypes = [ctypes.py_object]
    capsule_name = ctypes.pythonapi.PyCapsule_GetName(texture)

    ctypes.pythonapi.PyCapsule_GetPointer.restype = ctypes.c_void_p
    ctypes.pythonapi.PyCapsule_GetPointer.argtypes = [ctypes.py_object, ctypes.c_char_p]
    result = ctypes.pythonapi.PyCapsule_GetPointer(texture, capsule_name)

    mtl_texture = objc.objc_object(c_void_p=result)
    return mtl_texture


class SimpleServerScreen(Screen):
    ...

    def send(self):
        texture = self.metal_texture()  # of type PyCapsule
        texture_pointer = get_mtl_texture(texture)
        self.syphon_server.publish_frame_texture(texture_pointer, is_flipped=True)