Radxa's AICore SG2300x Module Brings 32 TOPS to Bear on Edge AI, On-Device Generative AI

Embedded computing specialist Radxa is aiming to power future edge artificial intelligence (edge AI) projects with the AICore SG2300x, a system-on-module (SOM) offering a claimed 32 tera-operations per second (TOPS) of compute performance — enough to run large language models (LLMs) and other generative AI models on device.

Built around the SOPHON SG2300x system-on-chip, the Radxa AICore SG2300x features eight Arm Cortex-A53 processor cores running at up to 2.3GHz and a dedicated tensor processing unit (TPU) coprocessor offering a claimed 32 tera-operations per second (TOPS) at INT8 precision, dropping to 16 TOPS in FP16/BF16 precision and 2 TOPS at FP32 precision. There's a generous 16GB of LPDDR4X RAM on-board along with a 64GB eMMC storage module, with an SDMMC interface for expansion.

According to the company's pre-launch announcement, brought to our attention by Linux Gizmos, the module provides the power to run popular generative AI models entirely on device — including generative pre-trained transformers (GPTs) and large language models (LLMs), the Stable Diffusion image generation model, and the document-querying ChatDOC model. In cases where 32 TOPS at INT8 isn't enough, Radxa says two modules can be "cascaded" to provide up to 64 TOPS.

Radxa promises support for a range of machine-learning frameworks, including TensorFlow and PyTorch, through the BMNNSDK software development kit. "The easy-to-use and convenient toolkit covers the model optimization, efficient runtime support, and other capabilities required for neural network inference," the company claims, and will be offered alongside a model zoo with ready-to-run models including YoloV8, ResNet, PP-OCR, DeepSORT, and OpenPose.

For peripheral connectivity the module includes one PCI Express 3.0 four-lane root complex and another endpoint, up to three UART and three I2C buses, and 32 general-purpose input/output (GPIO) pins with two pulse-width modulation (PWM) channels. There's a pair of gigabit Ethernet PHYs and hardware decoding for 32 channels of H.264/H.265 video at 1080p25 plus encoding for 12 channels of H.264/H.265 at the same resolution and refresh rate.

More information on the module is available on the Radxa website, but the company has yet to disclose pricing and general availability; it has, however, confirmed plans to release an open-source carrier board design along with a design guide document for those looking to build their own.