Multi-core boot process on the RK3568 development board

Let's talk about the multi-core boot process on the RK3568 development board.

The YY3568 motherboard is based on the Rockchip RK3568 chip platform, featuring a quad-core 64-bit Cortex-A55 processor with a maximum frequency of up to 2GHz. It integrates a dual-core architecture GPU and high-performance NPU, delivering excellent chip performance. The development board features rich interface functions and robust multimedia performance, making it suitable for unique advantages in IoT, industrial control, smart transportation, and lightweight AI fields.

It comes with 2xDSI, 1 HDMI, and 1 edp display interfaces, supporting dual-screen independent display and 4K resolution. With powerful display capabilities and compatibility with youyeetoo proprietary 7-inch MIPI screen and edp screen, it offers significant advantages and lower costs in applications like multi-screen advertising machines, electronic signage, self-service kiosks, and industrial HMIs.

The board features 2 Gigabit Ethernet ports for accessing and transferring data between internal and external networks. Equipped with WIFI/BT (M.2 interface), PCIe 3.0 interface, and SIM slot, it can connect to 4G communication modules to improve network transmission efficiency.

With 5 serial ports onboard, communication costs are significantly reduced. 2 I2C ports support multiple I2C devices, while 1 CAN port meets automotive electronic requirements.

The board also includes PCIe 3.0 and SATA interfaces, supporting M.2 SSDs and SATA hard drives for expandable storage.

YY3568 Multi-Core Boot Scheme

Rockchip provides 4 multi-core software schemes, but the boot process is generally consistent. The scheme is as follows:

Multi-core Boot Analysis of YY3568
The solution we validated on the YY3568 development board at youyeetoo Tech: 3 kernel (SMP) + 1 RT-Thread.
Boot Configuration
The multi-core boot configuration file path: device/rockchip/rk3568/rk3568_amp_linux.its.
The configuration file rk3568_amp_linux.its is in the format of device tree. Therefore, methods for operating its contents are similar to operating a device tree.
Multi-core boot mainly consists of two core nodes:
The conf node description: Describes which sub-cores need to be started (node: loadables), Linux kernel boot parameters (node: linux).
The images node description: Describes the parameters needed to start sub-cores, such as architecture, instruction set, partition start address, core boot delay, etc.

/{
description="FITsourcefileforrockchipAMP";
#address-cells=<1>;
images{
amp3{
description="bare-mental-core3";
data=/incbin/("cpu3.bin");//打包前的固件位置，一般不需要
type="firmware";
compression="none";
arch="arm";//固件的指令架构，当前只支持arm
cpu=<0x300>;//mpidr
thumb=<0>;//0:armorthumb2;1:thumb
hyp=<0>;//0:el1/svc;1:el2/hyp
load=<0x02800000>;//内存分区起始地址
udelay=<10000>;//启动下一个核心的延迟时间
hash{
algo="sha256";
};
};
};
configurations{
default="conf";
conf{
description="RockchipAMPimages";
rollback-index=<0x0>;
loadables="amp3";
signature{
algo="sha256,rsa2048";
padding="pss";
key-name-hint="dev";
sign-images="loadables";
};
/*-runlinuxoncpu0
*-itisbroughtupbyamp(thatrunonU-Boot)
*-itisbootentrydependsonU-Boot
*/
linux{
description="linux-os";
arch="arm64";
cpu=<0x000>;
thumb=<0>;
hyp=<0>;
udelay=<0>;
};
};
};
};

Memory Partition

Describes the starting address and size of the memory partition for each core.

In the solution we adopted: 3 kernel (SMP) + 1 RT-Thread, so the memory location for RT-Thread is: CPU3_MEM_BASE=0x02800000.

# Linux + HAL/RTT Example of memory resource partition in formal format：
CPU0_MEM_BASE=0x03000000
CPU1_MEM_BASE=0x01800000
CPU2_MEM_BASE=0x02000000
CPU3_MEM_BASE=0x02800000
CPU0_MEM_SIZE=0x00800000
CPU1_MEM_SIZE=0x00800000
CPU2_MEM_SIZE=0x00800000
CPU3_MEM_SIZE=0x00800000

AMP Firmware Packaging

The AMP firmware for RK3568 contains: boot configuration information (rk3568_amp_linux.its) + code for the secondary core.

It is packaged together using mkimage, located at: device/rockchip/common/mkimage.

Packaging command: mkimage -f amp.its -E -p 0xe00 amp.img, where:

0xe00: This is the size of the its in the firmware. If the size of the its file is less than the corresponding size, it is padded with zeros.

The code for the secondary core is appended after it.

Source Code Analysis

Kernel Boot Process - Preparation

The multi-core boot process of RK3568 is managed by uboot, so we mainly analyze the source code of uboot.

The code path for the multi-core boot process is: uboot/drivers/cpu/rockchip_amp.c.

The entry function for multi-core boot is: int amp_cpus_on(void).

To start the cores, four preparations need to be made in advance:

• 1. Obtain the boot device for the device. For YY3568, we currently use EMMC, so this part describes the EMMC device; then obtain the AMP partition from the boot device.
• 2. Allocate space to store the header information, and obtain the header information (i.e., the multi-core boot configuration information: rk3568_amp_linux.its) from the AMP partition; check the validity of its and obtain its size.
• 3. Allocate memory for the firmware, and obtain the content of the secondary cores from the AMP partition.
• 4. Parse the loadable items based on the header information. Then call brought_up_all_amp() to start all cores.

intamp_cpus_on(void)
{
....省略
dev_desc=rockchip_get_bootdev();
....省略
if(part_get_info_by_name(dev_desc,AMP_PART,&part)< 0)       // ①
....省略
hdr = memalign(ARCH_DMA_MINALIGN, FIT_HEADER_SIZE);
....省略
/* get totalsize */
offset = part.start;
cnt = DIV_ROUND_UP(FIT_HEADER_SIZE, part.blksz);
if (blk_dread(dev_desc, offset, cnt, hdr) != cnt) {
....省略
if (fdt_check_header(hdr)) {
....省略
if (fit_get_totalsize(hdr, &totalsize)) {           // ②
....省略
/* load image */
fit = memalign(ARCH_DMA_MINALIGN, ALIGN(totalsize, part.blksz));
....省略
offset += cnt;
cnt = DIV_ROUND_UP(totalsize, part.blksz) - cnt;
if (blk_dread(dev_desc, offset, cnt, fit + FIT_HEADER_SIZE) != cnt) {   // ③
....省略
ret = parse_os_amp_dispatcher();
....省略
/* Load loadables */
memset(&images, 0, sizeof(images));
images.fit_uname_cfg = "conf";
images.fit_hdr_os = fit;
images.verify = 1;
ret = boot_get_loadable(0, NULL, &images, IH_ARCH_DEFAULT, NULL, NULL);     // ④
....省略
/* Wakeup */
ret = brought_up_all_amp(images.fit_hdr_os, images.fit_uname_cfg); //⑤
....省略
}

Kernel Boot Process - Loadable Items Retrieval

The core boot process is divided into two parts: the Linux kernel boot part and the non-Linux kernel (RT-Thread) boot part.

For the Linux kernel boot part: Retrieve the Linux node from the rk3568_amp_linux.its configuration. Then call brought_up_amp() to start the kernel.

Non-Linux Kernel (RT-Thread) Boot Process: Retrieve the loadables node from the rk3568_amp_linux.its configuration. Traverse the node members, obtain the configuration information of the corresponding loadable items, and then call brought_up_amp() to start the kernel.

staticintbrought_up_all_amp(void*fit,constchar*fit_uname_cfg)
{
....omit
g_bootcpu.boot_on=1;
linux_noffset=fdt_subnode_offset(fit,conf_noffset,"linux");//①
if(linux_noffset>0){
ret=brought_up_amp(fit,linux_noffset,&g_bootcpu,1);
if(ret)
returnret;
}
for(loadables_index=0;//②
uname=fdt_stringlist_get(fit,conf_noffset,
FIT_LOADABLE_PROP,loadables_index,NULL),uname;
loadables_index++){
cpu_noffset=fit_image_get_node(fit,uname);
if(cpu_noffset< 0)
return cpu_noffset;
ret = brought_up_amp(fit, cpu_noffset, &g_bootcpu, 0);
if (ret)
return ret;
}
....omit
return 0;
}

Kernel Boot Process - Retrieving Core Configuration Parameters

Once we've obtained the loadable item nodes, we extract their loading parameters from the nodes, using the same method as retrieving data from a device tree.

We then initiate core startup using smc_cpu_on().

staticintbrought_up_amp(void*fit,intnoffset,
boot_cpu_t*bootcpu,intis_linux)
{
....omission
desc=fdt_getprop(fit,noffset,"description",NULL);
cpu=fit_get_u32_default(fit,noffset,"cpu",-ENODATA);
hyp=fit_get_u32_default(fit,noffset,"hyp",0);
thumb=fit_get_u32_default(fit,noffset,"thumb",0);
entry=load=fit_get_u32_default(fit,noffset,"load",-ENODATA);
us=fit_get_u32_default(fit,noffset,"udelay",0);
boot_on=fit_get_u32_default(fit,noffset,"boot-on",1);
fit_image_get_arch(fit,noffset,&arch);
fit_image_get_type(fit,noffset,&type);
fit_image_get_data_size(fit,noffset,&data_size);
memset(&args,0,sizeof(args));
….omission
/*bootnow*/
ret=smc_cpu_on(cpu,pe_state,entry,&args,is_linux);
if(ret)
returnret;
exit:
if(us)
udelay(us);
return0;
}

Kernel Boot Process - Kernel Startup

• Check the PE (Processing Element) status. If the target PE status is the default "arch" status, power on the CPU directly.
• If it's a non-Linux system, jump to the "finish" step to start the kernel directly.
• If it's a Linux system, set the boot parameters, then start the kernel.

staticintsmc_cpu_on(u32cpu,u32pe_state,u32entry,
boot_args_t*args,boolis_linux)
{
....omit
/*iftargetpestateisdefaultarchstate,powerupcpudirectly*/
if(is_default_pe_state(pe_state))
gotofinish;
ret=sip_smc_amp_cfg(AMP_PE_STATE,cpu,pe_state,0);
if(ret){
AMP_E("smcpe-state,ret=%d
",ret);
returnret;
}
/*onlylinuxneedsbootargs*/
if(!is_linux)
gotofinish;
ret=sip_smc_amp_cfg(AMP_BOOT_ARG01,cpu,args->arg0,args->arg1);
if(ret){
AMP_E("smcbootarg01,ret=%d
",ret);
returnret;
}
ret=sip_smc_amp_cfg(AMP_BOOT_ARG23,cpu,args->arg2,args->arg3);
if(ret){
AMP_E("smcbootarg23,ret=%d
",ret);
returnret;
}
finish:
ret=psci_cpu_on(cpu,entry);
if(ret){
printf("cpuupfailed,ret=%d
",ret);
returnret;
}
printf("OK
");
return0;
}

Multi-Core Booting Results

Multi-core configuration: 3 kernel (SMP) + 1 RT-Thread

We need to prepare two serial ports: one for Linux terminal information printing (UART2), and one for RT-Thread terminal information printing (UART4).

YY3568 has all the serial ports exposed, so debugging is very convenient. The wiring diagram is as follows:

Results: