System architecture

The RISC-V ISA, on which Cartesi Machines are based, consists of a minimal 32-bit integer instruction set to which several extensions can be added. The standard defines a privileged architecture with features commonly used by modern operating systems, such as multiple privilege levels, paged-based virtual-memory, timers, interrupts, exceptions and traps, etc. Implementations are free to select the combination of extensions that better suit their needs.

The Cartesi Machine architecture can be separated into a processor and a board. The processor performs the computations, executing the traditional fetch-execute loop while maintaining a variety of registers. The board defines the surrounding environment with an assortment of memories (ROM, RAM, flash drives, rollup memory ranges) and a number of devices. To make verification possible, a Cartesi Machine maps its entire state to the physical address space in a well-defined way. This includes the internal states of the processor, the board, and of all attached devices. The contents of the address space therefore completely define the Cartesi Machine. Fortunately, this modification does not limit the operating system or the applications it hosts in any significant way.

Both the processor and board are implemented in the emulator. A full description of the RISC-V ISA is out of the scope of this documentation (See the volumes 1 and 2 of the ISA specification for details.) This section describes Cartesi's RISC-V architecture, the modifications made to support verification, the devices supported by the emulator, and the process the machine follows to boot the Linux kernel.

The processor

Following RISC-V terminology, Cartesi Machines implement the RV64IMAZicsr_Zifencei ISA. The letters after RV specify the extension set. This selection corresponds to a 64-bit machine, integer arithmetic with multiplication and division, atomic operations, as well as the optional supervisor and user privilege levels. In addition, Cartesi Machines support the Sv39 mode of address translation and memory protection. Sv39 provides a 39-bit protected virtual address space, divided into 4KiB pages, organized by a three-level page table. This set of features creates a balanced compromise between the simplicity demanded by a blockchain implementation and the flexibility expected from off-chain computations.

There are a total of 98 instructions, out of which 28 simply narrow or widen, respectively, their 64-bit or 32-bit counterparts. This being a RISC ISA, most instructions are very simple and can be emulated in a few lines of high-level code. In contrast, the x86 ISA defines at least 2000 (potentially complex) instructions. In fact, the only complex operation in RISC-V is the virtual-to-physical address translation. Instruction decoding is particularly simple due to the reduced number of formats (only 4, all taking 32-bits).

The entire processor state fits within 512 bytes, which are divided into 64 registers, each one holding 64-bits. It consists of 32 general-purpose integer registers and 26 control and status registers (CSRs). Most of these registers are defined by the RISC-V ISA; the remaining are Cartesi-specific. The processor makes its entire state available, externally-only and read-only, by mapping individual registers to the lowest 512 bytes in the physical address space (in the processor shadow). The adjacent 1.5KiB are reserved for future use. The entire mapping is given in the following table:

Offset	Register	Offset	Register	Offset	Register	Offset	Register
0x000	x0	0x120	mcycle	0x160	misa	0x1a0	sepc
0x008	x1	0x128	minstret	0x168	mie	0x1a8	scause
…	…	0x130	mstatus	0x170	mip	0x1b0	stval
0x0f8	x31	0x138	mtvec	0x178	medeleg	0x1b8	satp
0x100	pc	0x140	mscratch	0x180	mideleg	0x1c0	scounteren
0x108	mvendorid	0x148	mepc	0x188	mcounteren	0x1c8	ilrsc
0x110	marchid	0x150	mcause	0x190	stvec	0x1d0	iflags
0x118	mimpid	0x158	mtval	0x198	sscratch

The only generally relevant standard register is mcycle. Since its value is advanced at every CPU cycle, it can be used to identify a particular step in the computation being performed by a Cartesi Machine. This is a key component of the verification process, and can also be used to bound the amount of computation.

The registers whose names start with “i” are Cartesi additions, and have the following semantics:

• The layout for register iflags can be seen below:

  Bits	63–5	4–3	2	1	0
  Field	Reserved	PRV	X	Y	H

Bit PRV gives the current privilege level (0 for User, 1 for Supervisor, and 3 for Machine), bit X is set to 1 when the processor has yielded automatic, bit Y is set to 1 when the processor has yielded manual, bit H is set to 1 to signal the processor has been permanently halted.

• Register ilrsc holds the reservation address for the LR/SC atomic memory operations;

The board

The interaction between board and processor happens through interrupts and the memory bus. Devices are mapped to the processor's physical address space. The mapping can be seen in the following table:

Physical address	Mapping
0x00000000–0x000003ff	Processor shadow
0x00000800–0x00000bff	Board shadow
0x00001000–0x000ffff	ROM (Bootstrap & Devicetree)
0x02000000–0x020bffff	Core Local Interruptor
0x40008000–0x40008fff	Host-Target Interface
0x60000000–0x600fffff (configurable)	Rollup RX buffer
0x60200000–0x602FFFFF (configurable)	Rollup TX buffer
0x60400000–0x60400FFF (configurable)	Rollup Input Metadata
0x60600000–0x606FFFFF (configurable)	Rollup Voucher Hashes
0x60800000–0x608FFFFF (configurable)	Rollup Notice Hashes
0x80000000–configurable	RAM
configurable	Flash drive 0
…	…
configurable	Flash drive 7

There are 60KiB of ROM starting at address 0x1000, where execution starts by default. The amount of RAM is user-configurable, but always starts at address 0x80000000. Finally, a number of additional physical memory ranges can be set aside for flash-memory devices. These will typically be preloaded with file-system images, but can also hold raw data.

The board maps two non-memory devices to the physical address space: CLINT and HTIF.

CLINT

The Core Local Interruptor (or CLINT) controls the timer interrupt. The active addresses are 0x0200bff8 (mtime) and 0x02004000 (mtimecmp). The CLINT issues a hardware interrupt whenever mtime equals mtimecmp. Since Cartesi Machines must ensure reproducibility, the processor's clock and the timer are locked by a constant frequency divisor of 100. In other words, mtime is incremented once for every 100 increments of mcycle. There is no notion of wall-clock time.

HTIF

The Host-Target Interface (HTIF) mediates communication with the external world. It is mapped to a physical memory starting at 0x40008000, where registers can be accessed at the following offsets:

Offset	Register
0x000	tohost
0x008	fromhost
0x010	ihalt
0x018	iconsole
0x020	iyield
0x028	Reserved
…	…
0x218	Reserved

The format of CSRs tohost and fromhost are as follows:

Bits	63–56	55–48	47–0
Field	DEV	CMD	DATA

Interactions with Cartesi's HTIF device follow the following protocol:

1. start by writing 0 to fromhost;
2. write the request to tohost;
3. read the response from fromhost.

Cartesi's HTIF supports 3 subdevices: Halt, Console, and Yield. These are identified by the following values for the field DEV.

`DEV`
Name	Value
HTIF_DEVICE_HALT	0
HTIF_DEVICE_CONSOLE	1
HTIF_DEVICE_YIELD	2

Registers ihalt, iconsole, and iyield are bit masks specifying the commands that are available for the respective devices. Unavailable commands are silently ignored by the machine.

Halt

`CMD`
Name	Value
HTIF_HALT_HALT	0

The Halt device (DEV=HTIF_DEVICE_HALT) is used to halt the machine. This will permanently set bit H in iflags and return control back to the host.

Send request CMD=HTIF_HALT_HALT and DATA containing bit 0 set to 1. Bits 47–1 can be set to an arbitrary exit code.

Console

`CMD`
Name	Value
HTIF_CONSOLE_GETCHAR	0
HTIF_CONSOLE_PUTCHAR	1

The Console device (DEV=HTIF_DEVICE_CONSOLE) can be used to input/output characters.

To input a character from console (in interactive sessions), request CMD=HTIF_CONSOLE_GETCHAR, DATA=0, then read response CMD=HTIF_CONSOLE_GETCHAR, DATA=<ch>+1. (DATA=0 means no character was available);

To output a character <ch> to console, request CMD=HTIF_CONSOLE_PUTCHAR, with DATA=<ch>.

Yield

The Yield device can be used to return control to the host. It uses a slight refinement to the format of CSRs tohost and fromhost, by splitting out a REASON field from DATA:

Bits	63–56	55–48	47–32	31–0
Field	DEV	CMD	REASON	DATA

There are two types of yield: automatic and manual.

`CMD`
Name	Value
HTIF_YIELD_AUTOMATIC	0
HTIF_YIELD_MANUAL	1

To issue an automatic yield, request CMD=HTIF_YIELD_AUTOMATIC. An automatic yield sets the bit X in iflags and returns control back to the host. There are currently 4 supported reasons for automatic yields:

`REASON`
Name	Value
HTIF_YIELD_REASON_PROGRESS	0
HTIF_YIELD_REASON_TX_VOUCHER	3
HTIF_YIELD_REASON_TX_NOTICE	4
HTIF_YIELD_REASON_TX_REPORT	5

To report progress, set REASON=HTIF_YIELD_REASON_PROGRESS, and DATA=<permil>, where <permil> gives the progress in parts per thousand. The other reasons for automatic yield signal the production of Cartesi Rollups responses. REASON=HTIF_YIELD_REASON_TX_VOUCHER, REASON=HTIF_YIELD_REASON_TX_NOTICE, and REASON=HTIF_YIELD_REASON_TX_REPORT denote, respectively, transfers of a voucher, a notice, and a report from target to host. The DATA field in tohost is ignored in these cases.

To issue a manual yield, request CMD=HTIF_YIELD_MANUAL. A manual yield sets the bit Y in iflags and returns control back to the host. There are currently 3 supported reasons for manual yields, all used with Cartesi Rollups:

`REASON`
Name	Value
HTIF_YIELD_REASON_RX_ACCEPTED	1
HTIF_YIELD_REASON_RX_REJECTED	2
HTIF_YIELD_REASON_TX_EXCEPTION	6

To accept or reject the previous request, set REASON=HTIF_YIELD_REASON_RX_ACCEPTED or REASON=HTIF_YIELD_REASON_RX_REJECTED, respectively. The DATA field in tohost is ignored in these cases. Upon return, the DATA field in fromhost will contain the type of the next request:

`DATA` in response
Name	Value
HTIF_YIELD_ADVANCE_STATE	0
HTIF_YIELD_INSPECT_STATE	1

The signal the throwing of a rollup exception, set REASON=HTIF_YIELD_REASON_TX_EXCEPTION. The DATA field in tohost is ignored in this case.

Before resuming the emulator after a manual yield, the host must manually reset the Y bit in iflags. Otherwise, the emulator will immediately return with no changes to its state.

Rollup

In order to interact with Cartesi Rollups, the host application controlling the emulator and the target application running inside the emulator must follow an agreed-upon protocol, mediated by the HTIF Yield device.

The low-level view of what happens inside the machine is as follows:

Initialize
Repeat
    `voucher_index` = 0
    `notice_index` = 0
    `reason` = HTIF_YIELD_REASON_RX_ACCEPTED
    Yield manual with `reason` as `REASON` in `tohost`
    If `DATA` in `fromhost` is `HTIF_YIELD_ADVANCE_STATE`
        Read input metadata from Rollup Input Metadata
        Read input data from Rollup RX Buffer
        Process advance-state request
        For each voucher to emit
            Write voucher data to Rollup TX Buffer
            Write voucher hash to slot `voucher_index` in Rollup Voucher Hashes
            `voucher_index` = `voucher_index` + 1
            Yield automatic with HTIF_YIELD_REASON_TX_VOUCHER as `REASON` in `tohost`
        End
        For each notice to emit
            Write notice data to Rollup TX Buffer
            Write notice hash to slot `notice_index` in Rollup Notice Hashes
            `notice_index` = `notice_index` + 1
            Yield automatic with HTIF_YIELD_REASON_TX_NOTICE as `REASON` in `tohost`
        End
        For each report to emit
            Write report data to Rollup TX Buffer
            Yield automatic with HTIF_YIELD_REASON_TX_REPORT as `REASON` in `tohost`
        End
        If exception to emit
            Write exception data to Rollup TX Buffer
            Yield automatic with HTIF_YIELD_REASON_TX_EXCEPTION as `REASON` in `tohost`
        ElseIf input rejected
            `reason` = HTIF_YIELD_REASON_RX_REJECTED
        End
    End
    If `DATA` in `fromhost` is `HTIF_YIELD_INSPECT_STATE`
        Read query data from Rollup RX Buffer
        Process inspect-state request
        For each report
            Write report data to Rollup TX Buffer
            Yield automatic with HTIF_YIELD_REASON_TX_REPORT as `REASON` in `tohost`
        End
        If exception
            Write exception data to Rollup TX Buffer
            Yield automatic with HTIF_YIELD_REASON_TX_EXCEPTION as `REASON` in `tohost`
        ElseIf input rejected
            `reason` = HTIF_YIELD_REASON_RX_REJECTED
        End
    End
End

At a higher level, the target application running inside the emulator is supported by the /dev/rollup Linux device driver via its ioctl interface, or by even higher-level interfaces based on it, such as /opt/cartesi/bin/rollup command-line utility or the HTTP API exposed by the /opt/cartesi/bin/rollup-http-server command-line utility. It is the /dev/rollup device that copies data to and from all rollup memory ranges, and that uses the /dev/yield device to perform the required yields.

There are two types of request: advance-state requests and inspect-state requests. The loop processes one request per iteration. To transition between requests, the application accepts or rejects the previous request by issuing a command to the HTIF yield device, or throws an exception. The return from the yield defines the type of the next request.

When the application identifies an advance-state request, it obtains input medatada from the Rollup Input Metadata memory range, and input from the Rollup RX Buffer memory range. While processing advance-state requests, the application can emit vouchers, notices, reports, or exceptions. It writes the data for all these to the Rollup TX buffer memory range. Moreover, when emitting the ith voucher (respectively, notice) in response to a given input, it writes its hash to the ith 32-byte slot in the Rollup Voucher Hashes (respectively, Rollup Notice Hashes) memory range. It then issues the appropriate command to the HTIF yield device.

When an application identifies an inspect-state request, it obtains the query from the Rollup RX buffer memory range. While processing inspect-state requests, the application can emit vouchers, or exceptions. It writes data for all these to the Rollup TX buffer memory range and sends the appropriate command to the HTIF yield device.

The format for all these request and response data are as follows:

Format for input metadata
Offset (bytes)	Field
0–31	message sender (address hash)
32–63	block number (number)
64–95	time stamp (number)
96–127	epoch index (number)
128–159	input index (number)
Format for voucher
Offset (bytes)	Field
0–31	address (address hash)
32–63	offset (number, always 64)
64–95	length (number)
96–96+length-1	payload (raw data)
Format for input, notice, report, and exception
Offset (bytes)	Field
0–31	offset (number, always 32)
32–63	length (number)
64–64+length-1	payload (raw data)

All numbers are encoded as 256-bit big-endian integers. Address hashes are encoded in the least significant 160-bits of a zero-padded 256-bit big-endian integer.

In the host, the loop is as follows:

While bit `H` in `iflags` is not set (machine has not halted)
    Snapshot machine state
    Resume machine
    If bit `Y` in `iflags` is set (i.e. manual yield)
        If `REASON` in `tohost` is HTIF_YIELD_REASON_RX_REJECTED
            Discard vouchers and notices emitted from previous request, if any
            Rollback machine state
        End
        If `REASON` in `tohost` is HTIF_YIELD_REASON_TX_EXCEPTION
            Read exception data from Rollup TX Buffer
            Discard vouchers and notices emitted from previous request, if any
            Rollback machine state
        End
        If `REASON` in `tohost` is HTIF_YIELD_REASON_RX_ACCEPTED
            If previous request was advance-state
                Read Rollup Voucher Hashes for previous request
                Read Rollup Notice Hashes for previous request
            End
            Obtain the next request from an external source
            If advance-state request
                Clear Rollup Voucher Hashes
                Clear Rollup Notice Hashes
                Write input data to Rollup RX Buffer
                Write input metadata to Rollup Input Metadata
                Write HTIF_YIELD_ADVANCE_STATE to `DATA` in `fromhost`
            End
            If inspect-state request
                Write query data to Rollup RX Buffer
                Write HTIF_YIELD_INSPECT_STATE to `DATA` in `fromhost`
            End
            Clear `Y` bit in `iflags`
        End
    End
    If bit `X` in `iflags` is set (i.e. automatic yield)
        If `REASON` in `tohost` is HTIF_YIELD_REASON_TX_VOUCHER
            Read voucher data from rollup memory ranges
        End
        If `REASON` in `tohost` is HTIF_YIELD_REASON_TX_NOTICE
            Read notice data from rollup memory ranges
        End
        If `REASON` in `tohost` is HTIF_YIELD_REASON_TX_REPORT
            Read report data from Rollup TX Buffer
        End
    End
End

In production, the host application controlling the emulator is the Server Manager While prototyping, it can be the cartesi-machine command-line utility, or even a custom script using the Lua API.

PMAs

The physical memory mapping is described by Physical Memory Attribute records (PMAs) that start at address 0x00000800 (the board shadow) . Each PMA consists of 2 64-bit words. The first word gives the start of a range and the second word its length. These words are readable both internally and externally. Since the ranges must be aligned to 4KiB page boundaries, the lowest 12-bits of each word are available for attributes. The meaning of each attribute field is as follows:

Bits	63–12	11–8	7	6	5	4	3	2	1	0
Field	start	DID	IW	IR	X	W	R	E	IO	M
Bits	63–12	11–0
Field	length	Reserved (=0)

The M, IO, and E bits are mutually exclusive, and respectively mark the range as memory, I/O mapped, or excluded. Bits R, W, and X mark read, write, and execute permissions, respectively. The IR and IW bits mark the range as idempotent for reads and writes, respectively. Finally, the DID gives the device id, which can have the following values:

Name	Value
PMA_MEMORY_DID	0
PMA_SHADOW_DID	1
PMA_FLASH_DRIVE_DID	2
PMA_CLINT_DID	3
PMA_HTIF_DID	4
PMA_DHD_DID	5
PMA_ROLLUP_RX_BUFFER_DID	6
PMA_ROLLUP_TX_BUFFER_DID	7
PMA_ROLLUP_INPUT_METADATA_DID	8
PMA_ROLLUP_VOUCHER_HASHES_DID	9
PMA_ROLLUP_NOTICE_HASHES_DID	10

The list of PMA records ends with an invalid PMA entry for which length=0.

Linux setup

By default, pc starts at 0x1000, pointing to the start of the ROM region. Before control reaches the RAM image (and ultimately the Linux kernel), a small program residing in ROM builds a devicetree describing the hardware. Cartesi's ROM image rom.bin containing this program can be generated from the rom/ directory of the Cartesi Machine SDK. To do so, it goes over the PMA entries identifying the devices and their locations in the physical address space. It also looks for a null-terminated string, starting at the last 4k of the ROM region, that will be used as the command-line for the Linux kernel. Once the devicetree is ready, the ROM program sets register x10 to 0 (the value of mhartid), x11 to point to the devicetree (which it places at the end of the RAM region), and then jumps to RAM-base at  address 0x80000000. This is where the entry point of the RAM image is expected to reside.

The dtc command-line utility can be used to inspect the devicetree:

cartesi-machine \
    --append-rom-bootargs="single=yes" \
    --rollup \
    -- "dtc -I dtb -O dts /sys/firmware/fdt"

The result is

         .
        / \
      /    \
\---/---\  /----\
 \       X       \
  \----/  \---/---\
       \    / CARTESI
        \ /   MACHINE
         '

<stdout>: Warning (interrupt_provider): /cpus/cpu@0/interrupt-controller: Missing #address-cells in interrupt provider
/dts-v1/;

/ {
    #address-cells = <0x02>;
    #size-cells = <0x02>;
    compatible = "ucbbar,riscvemu-bar_dev";
    model = "ucbbar,riscvemu-bare";

    cpus {
        #address-cells = <0x01>;
        #size-cells = <0x00>;
        timebase-frequency = <0xf4240>;

        cpu@0 {
            device_type = "cpu";
            reg = <0x00>;
            status = "okay";
            compatible = "riscv";
            riscv,isa = "rv64aimsu";
            mmu-type = "riscv,sv39";
            clock-frequency = <0x5f5e100>;

            interrupt-controller {
                #interrupt-cells = <0x01>;
                interrupt-controller;
                compatible = "riscv,cpu-intc";
                phandle = <0x01>;
            };
        };
    };

    memory@80000000 {
        device_type = "memory";
        reg = <0x00 0x80000000 0x00 0x3ff0000>;
    };

    flash@8000000000000000 {
        #address-cells = <0x02>;
        #size-cells = <0x02>;
        compatible = "mtd-ram";
        bank-width = <0x04>;
        reg = <0x80000000 0x00 0x00 0x5000000>;
        linux,mtd-name = "flash.0";
    };

    rollup {
        #address-cells = <0x02>;
        #size-cells = <0x02>;
        compatible = "ctsi-rollup";

        rx_buffer@60000000 {
            reg = <0x00 0x60000000 0x00 0x200000>;
        };

        tx_buffer@60200000 {
            reg = <0x00 0x60200000 0x00 0x200000>;
        };

        input_metadata@60400000 {
            reg = <0x00 0x60400000 0x00 0x1000>;
        };

        voucher_hashes@60600000 {
            reg = <0x00 0x60600000 0x00 0x200000>;
        };

        notice_hashes@60800000 {
            reg = <0x00 0x60800000 0x00 0x200000>;
        };
    };

    yield {
        compatible = "ctsi-yield";
        automatic;
        manual;
    };

    soc {
        #address-cells = <0x02>;
        #size-cells = <0x02>;
        compatible = "ucbbar,riscvemu-bar-soc\0simple-bus";
        ranges;

        htif@40008000 {
            compatible = "ucb,htif0";
            reg = <0x00 0x40008000 0x00 0x1000>;
            interrupts-extended = <0x01 0x0d>;
        };
    };

    chosen {
        bootargs = "console=hvc0 rootfstype=ext2 root=/dev/mtdblock0 rw quiet swiotlb=noforce mtdparts=flash.0:-(root) single=yes -- dtc -I dtb -O dts /sys/firmware/fdt";
    };
};

Halted
Cycles: 71150848

The memory@80000000 section describes 64MiB of RAM starting at address 0x80000000. The flash@8000000000000000 describes flash drive 0: a memory region of 60MiB, starting at address 0x8000000000000000, under the control of the mtd-ram driver, with name flash.0. This will eventually become available as block device /dev/mtdblock0. The rollup section specifies the starts and lengths of all rollup memory ranges. The yield section specifies that the machine will process automatic and manual yields. Finally, section chosen includes the bootargs string that will be used as the kernel command-line parameters. Notice the specification of the root file-system pointing to /dev/mtdblock0, i.e., flash.0, and the mtdparts giving it the label root. Also notice the command dtc -I dtb -O dts /sys/firmware/fdt coming directly from the cartesi-machine command line.

Linux support for RISC-V is upstream in the Linux kernel archives. The kernel runs in supervisor mode, on top of a Supervisor Binary Interface (SBI) provided by a machine-mode shim: the Berkeley Boot Loader (BBL). The BBL is linked against the Linux kernel and this resulting RAM image is preloaded into RAM. Cartesi's RAM image linux.bin can be generated from the kernel/ directory of the Cartesi Machine SDK. The SBI provides a simple interface through which the kernel interacts with CLINT and HTIF. Besides implementing the SBI, the BBL also installs a trap that catches invalid instruction exceptions. This mechanism can be used, for example, to emulate floating-point instructions, although it is more efficient to setup the target toolchain to replace floating point instructions with calls to a soft-float implementation instead. After installing the trap, BBL switches to supervisor mode and cedes control to the kernel entry point.

After completing its own initialization, the kernel mounts the root file-system and eventually cedes control to /sbin/init. Cartesi's root file-system rootfs.ext2 can be generated from the fs/ directory in the Cartesi Machine SDK. The Cartesi-provided /sbin/init script scans all flash devices /dev/mtdblock1–/dev/mtdblock7 for valid file-systems. When a file-system is found, the script obtains the corresponding <label> (set in the mtdparts kernel command-line parameter) by inspecting /sys/block/mtdblock*/device/name and mounts the filesystem at /mnt/<label>. The kernel passes to /sbin/init as command-line parameters all arguments after the separator -- in the bootargs string it found in the devicetree. The Cartesi-provided /sbin/init script concatenates all arguments into a string and executes the command in this string in a shell. When the shell returns, /sbin/init unmount all file-systems and gracefully halts the machine.