> For the complete documentation index, see [llms.txt](https://docs.cartesi.io/llms.txt) --- title: System architecture --- The RISC-V ISA, on which Cartesi Machines are based, consists of a minimal 32-bit integer instruction set to which several extensions can be added. The standard defines a privileged architecture with features commonly used by modern operating systems, such as multiple privilege levels, paged-based virtual-memory, timers, interrupts, exceptions and traps, etc. Implementations are free to select the combination of extensions that better suit their needs. The Cartesi Machine architecture can be separated into a processor and a board. The processor performs the computations, executing the traditional fetch-execute loop while maintaining a variety of registers. The board defines the surrounding environment with an assortment of memories (ROM, RAM, flash drives, rollup memory ranges) and a number of devices. To make verification possible, a Cartesi Machine maps its entire state to the physical address space in a well-defined way. This includes the internal states of the processor, the board, and of all attached devices. The contents of the address space therefore completely define the Cartesi Machine. Fortunately, this modification does not limit the operating system or the applications it hosts in any significant way. Both the processor and board are implemented in the emulator. A full description of the RISC-V ISA is out of the scope of this documentation (See the volumes [1 and 2](https://riscv.org/technical/specifications/) of the ISA specification for details.) This section describes Cartesi's RISC-V architecture, the modifications made to support verification, the devices supported by the emulator, and the process the machine follows to boot the Linux kernel. ## The processor Following RISC-V terminology, Cartesi Machines implement the `RV64IMAZicsr_Zifencei` ISA. The letters after RV specify the extension set. This selection corresponds to a 64-bit machine, integer arithmetic with multiplication and division, atomic operations, as well as the optional supervisor and user privilege levels. In addition, Cartesi Machines support the Sv39 mode of address translation and memory protection. Sv39 provides a 39-bit protected virtual address space, divided into 4KiB pages, organized by a three-level page table. This set of features creates a balanced compromise between the simplicity demanded by a blockchain implementation and the flexibility expected from off-chain computations. There are a total of 98 instructions, out of which 28 simply narrow or widen, respectively, their 64-bit or 32-bit counterparts. This being a RISC ISA, most instructions are very simple and can be emulated in a few lines of high-level code. In contrast, the x86 ISA defines at least 2000 (potentially complex) instructions. In fact, the only complex operation in RISC-V is the virtual-to-physical address translation. Instruction decoding is particularly simple due to the reduced number of formats (only 4, all taking 32-bits). The entire processor state fits within 512 bytes, which are divided into 64 registers, each one holding 64-bits. It consists of 32 general-purpose integer registers and 26 control and status registers (CSRs). Most of these registers are defined by the RISC-V ISA; the remaining are Cartesi-specific. The processor makes its entire state available, externally-only and read-only, by mapping individual registers to the lowest 512 bytes in the physical address space (in the processor shadow). The adjacent 1.5KiB are reserved for future use. The entire mapping is given in the following table:
Offset Register Offset Register Offset Register Offset Register
0x000 x0 0x120 mcycle 0x160 misa 0x1a0 sepc
0x008 x1 0x128 minstret 0x168 mie 0x1a8 scause
… … 0x130 mstatus 0x170 mip 0x1b0 stval
0x0f8 x31 0x138 mtvec 0x178 medeleg 0x1b8 satp
0x100 pc 0x140 mscratch 0x180 mideleg 0x1c0 scounteren
0x108 mvendorid 0x148 mepc 0x188 mcounteren 0x1c8 ilrsc
0x110 marchid 0x150 mcause 0x190 stvec 0x1d0 iflags
0x118 mimpid 0x158 mtval 0x198 sscratch
The only generally relevant standard register is `mcycle`. Since its value is advanced at every CPU cycle, it can be used to identify a particular step in the computation being performed by a Cartesi Machine. This is a key component of the verification process, and can also be used to bound the amount of computation. The registers whose names start with “`i`” are Cartesi additions, and have the following semantics: * The layout for register iflags can be seen below:

Bits 63–5 4–3 2 1 0
Field Reserved PRV X Y H
Bit `PRV` gives the current privilege level (0 for User, 1 for Supervisor, and 3 for Machine), bit `X` is set to 1 when the processor has yielded automatic, bit `Y` is set to 1 when the processor has yielded manual, bit `H` is set to 1 to signal the processor has been permanently halted. * Register `ilrsc` holds the reservation address for the LR/SC atomic memory operations; ## The board The interaction between board and processor happens through interrupts and the memory bus. Devices are mapped to the processor's physical address space. The mapping can be seen in the following table:
Physical address Mapping
0x00000000–0x000003ff Processor shadow
0x00000800–0x00000bff Board shadow
0x00001000–0x000ffff ROM (Bootstrap & Devicetree)
0x02000000–0x020bffff Core Local Interruptor
0x40008000–0x40008fff Host-Target Interface
0x60000000–0x600fffff (configurable) Rollup RX buffer
0x60200000–0x602FFFFF (configurable) Rollup TX buffer
0x60400000–0x60400FFF (configurable) Rollup Input Metadata
0x60600000–0x606FFFFF (configurable) Rollup Voucher Hashes
0x60800000–0x608FFFFF (configurable) Rollup Notice Hashes
0x80000000–configurable RAM
configurable Flash drive 0
… …
configurable Flash drive 7
There are 60KiB of ROM starting at address `0x1000`, where execution starts by default. The amount of RAM is user-configurable, but always starts at address `0x80000000`. Finally, a number of additional physical memory ranges can be set aside for flash-memory devices. These will typically be preloaded with file-system images, but can also hold raw data. The board maps two non-memory devices to the physical address space: CLINT and HTIF. ### CLINT The Core Local Interruptor (or CLINT) controls the timer interrupt. The active addresses are `0x0200bff8` (`mtime`) and `0x02004000` (`mtimecmp`). The CLINT issues a hardware interrupt whenever `mtime` equals `mtimecmp`. Since Cartesi Machines must ensure reproducibility, the processor's clock and the timer are locked by a constant frequency divisor of `100`. In other words, `mtime` is incremented once for every 100 increments of `mcycle`. There is no notion of wall-clock time. ### HTIF The Host-Target Interface (HTIF) mediates communication with the external world. It is mapped to a physical memory starting at `0x40008000`, where registers can be accessed at the following offsets:
Offset Register
0x000 tohost
0x008 fromhost
0x010 ihalt
0x018 iconsole
0x020 iyield
0x028 Reserved
… …
0x218 Reserved
The format of CSRs `tohost` and `fromhost` are as follows:

Bits 63–56 55–48 47–0
Field DEV CMD DATA
Interactions with Cartesi's HTIF device follow the following protocol: 1. start by writing 0 to `fromhost`; 1. write the request to `tohost`; 1. read the response from `fromhost`. Cartesi's HTIF supports 3 subdevices: Halt, Console, and Yield. These are identified by the following values for the field `DEV`.
`DEV`
Name Value
HTIF_DEVICE_HALT 0
HTIF_DEVICE_CONSOLE 1
HTIF_DEVICE_YIELD 2
Registers `ihalt`, `iconsole`, and `iyield` are bit masks specifying the commands that are available for the respective devices. Unavailable commands are silently ignored by the machine. ##### Halt
`CMD`
Name Value
HTIF_HALT_HALT 0
The Halt device (`DEV=HTIF_DEVICE_HALT`) is used to halt the machine. This will permanently set bit `H` in `iflags` and return control back to the host. Send request `CMD=HTIF_HALT_HALT` and `DATA` containing bit 0 set to 1. Bits 47–1 can be set to an arbitrary exit code. ##### Console
`CMD`
Name Value
HTIF_CONSOLE_GETCHAR 0
HTIF_CONSOLE_PUTCHAR 1
The Console device (`DEV=HTIF_DEVICE_CONSOLE`) can be used to input/output characters. To input a character from console (in interactive sessions), request `CMD=HTIF_CONSOLE_GETCHAR`, `DATA=0`, then read response `CMD=HTIF_CONSOLE_GETCHAR`, `DATA=+1`. (`DATA=0` means no character was available); To output a character `` to console, request `CMD=HTIF_CONSOLE_PUTCHAR`, with `DATA=`. ##### Yield The Yield device can be used to return control to the host. It uses a slight refinement to the format of CSRs `tohost` and `fromhost`, by splitting out a `REASON` field from `DATA`:

Bits 63–56 55–48 47–32 31–0
Field DEV CMD REASON DATA
There are two types of yield: _automatic_ and _manual_.
`CMD`
Name Value
HTIF_YIELD_AUTOMATIC 0
HTIF_YIELD_MANUAL 1
To issue an automatic yield, request `CMD=HTIF_YIELD_AUTOMATIC`. An automatic yield sets the bit `X` in `iflags` and returns control back to the host. There are currently 4 supported reasons for automatic yields:
`REASON`
Name Value
HTIF_YIELD_REASON_PROGRESS 0
HTIF_YIELD_REASON_TX_VOUCHER 3
HTIF_YIELD_REASON_TX_NOTICE 4
HTIF_YIELD_REASON_TX_REPORT 5
To report `progress`, set `REASON=HTIF_YIELD_REASON_PROGRESS`, and `DATA=`, where `` gives the progress in parts per thousand. The other reasons for automatic yield signal the production of Cartesi Rollups responses. `REASON=HTIF_YIELD_REASON_TX_VOUCHER`, `REASON=HTIF_YIELD_REASON_TX_NOTICE`, and `REASON=HTIF_YIELD_REASON_TX_REPORT` denote, respectively, transfers of a voucher, a notice, and a report from target to host. The `DATA` field in `tohost` is ignored in these cases. To issue a manual yield, request `CMD=HTIF_YIELD_MANUAL`. A manual yield sets the bit `Y` in `iflags` and returns control back to the host. There are currently 3 supported reasons for manual yields, all used with Cartesi Rollups:
`REASON`
Name Value
HTIF_YIELD_REASON_RX_ACCEPTED 1
HTIF_YIELD_REASON_RX_REJECTED 2
HTIF_YIELD_REASON_TX_EXCEPTION 6
To accept or reject the previous request, set `REASON=HTIF_YIELD_REASON_RX_ACCEPTED` or `REASON=HTIF_YIELD_REASON_RX_REJECTED`, respectively. The `DATA` field in `tohost` is ignored in these cases. Upon return, the `DATA` field in `fromhost` will contain the type of the next request:
`DATA` in response
Name Value
HTIF_YIELD_ADVANCE_STATE 0
HTIF_YIELD_INSPECT_STATE 1
The signal the throwing of a rollup exception, set `REASON=HTIF_YIELD_REASON_TX_EXCEPTION`. The `DATA` field in `tohost` is ignored in this case. Before resuming the emulator after a manual yield, the host must manually reset the `Y` bit in `iflags`. Otherwise, the emulator will immediately return with no changes to its state. ### Rollup In order to interact with Cartesi Rollups, the host application controlling the emulator and the target application running inside the emulator must follow an agreed-upon protocol, mediated by the HTIF Yield device. The low-level view of what happens inside the machine is as follows: ``` Initialize Repeat `voucher_index` = 0 `notice_index` = 0 `reason` = HTIF_YIELD_REASON_RX_ACCEPTED Yield manual with `reason` as `REASON` in `tohost` If `DATA` in `fromhost` is `HTIF_YIELD_ADVANCE_STATE` Read input metadata from Rollup Input Metadata Read input data from Rollup RX Buffer Process advance-state request For each voucher to emit Write voucher data to Rollup TX Buffer Write voucher hash to slot `voucher_index` in Rollup Voucher Hashes `voucher_index` = `voucher_index` + 1 Yield automatic with HTIF_YIELD_REASON_TX_VOUCHER as `REASON` in `tohost` End For each notice to emit Write notice data to Rollup TX Buffer Write notice hash to slot `notice_index` in Rollup Notice Hashes `notice_index` = `notice_index` + 1 Yield automatic with HTIF_YIELD_REASON_TX_NOTICE as `REASON` in `tohost` End For each report to emit Write report data to Rollup TX Buffer Yield automatic with HTIF_YIELD_REASON_TX_REPORT as `REASON` in `tohost` End If exception to emit Write exception data to Rollup TX Buffer Yield automatic with HTIF_YIELD_REASON_TX_EXCEPTION as `REASON` in `tohost` ElseIf input rejected `reason` = HTIF_YIELD_REASON_RX_REJECTED End End If `DATA` in `fromhost` is `HTIF_YIELD_INSPECT_STATE` Read query data from Rollup RX Buffer Process inspect-state request For each report Write report data to Rollup TX Buffer Yield automatic with HTIF_YIELD_REASON_TX_REPORT as `REASON` in `tohost` End If exception Write exception data to Rollup TX Buffer Yield automatic with HTIF_YIELD_REASON_TX_EXCEPTION as `REASON` in `tohost` ElseIf input rejected `reason` = HTIF_YIELD_REASON_RX_REJECTED End End End ``` At a higher level, the target application running inside the emulator is supported by the `/dev/rollup` Linux device driver via its `ioctl` interface, or by even higher-level interfaces based on it, such as `/opt/cartesi/bin/rollup` command-line utility or the HTTP API exposed by the `/opt/cartesi/bin/rollup-http-server` command-line utility. It is the `/dev/rollup` device that copies data to and from all rollup memory ranges, and that uses the `/dev/yield` device to perform the required yields. There are two types of request: advance-state requests and inspect-state requests. The loop processes one request per iteration. To transition between requests, the application accepts or rejects the previous request by issuing a command to the HTIF yield device, or throws an exception. The return from the yield defines the type of the next request. When the application identifies an advance-state request, it obtains input medatada from the Rollup Input Metadata memory range, and input from the Rollup RX Buffer memory range. While processing advance-state requests, the application can emit vouchers, notices, reports, or exceptions. It writes the data for all these to the Rollup TX buffer memory range. Moreover, when emitting the ith voucher (respectively, notice) in response to a given input, it writes its hash to the ith 32-byte slot in the Rollup Voucher Hashes (respectively, Rollup Notice Hashes) memory range. It then issues the appropriate command to the HTIF yield device. When an application identifies an inspect-state request, it obtains the query from the Rollup RX buffer memory range. While processing inspect-state requests, the application can emit vouchers, or exceptions. It writes data for all these to the Rollup TX buffer memory range and sends the appropriate command to the HTIF yield device. The format for all these request and response data are as follows:
Format for input metadata
Offset (bytes) Field
0–31 message sender (address hash)
32–63 block number (number)
64–95 time stamp (number)
96–127 epoch index (number)
128–159 input index (number)
Format for voucher
Offset (bytes) Field
0–31 address (address hash)
32–63 offset (number, always 64)
64–95 length (number)
96–96+length-1 payload (raw data)
Format for input, notice, report, and exception
Offset (bytes) Field
0–31 offset (number, always 32)
32–63 length (number)
64–64+length-1 payload (raw data)
All numbers are encoded as 256-bit big-endian integers. Address hashes are encoded in the least significant 160-bits of a zero-padded 256-bit big-endian integer. In the host, the loop is as follows: ``` While bit `H` in `iflags` is not set (machine has not halted) Snapshot machine state Resume machine If bit `Y` in `iflags` is set (i.e. manual yield) If `REASON` in `tohost` is HTIF_YIELD_REASON_RX_REJECTED Discard vouchers and notices emitted from previous request, if any Rollback machine state End If `REASON` in `tohost` is HTIF_YIELD_REASON_TX_EXCEPTION Read exception data from Rollup TX Buffer Discard vouchers and notices emitted from previous request, if any Rollback machine state End If `REASON` in `tohost` is HTIF_YIELD_REASON_RX_ACCEPTED If previous request was advance-state Read Rollup Voucher Hashes for previous request Read Rollup Notice Hashes for previous request End Obtain the next request from an external source If advance-state request Clear Rollup Voucher Hashes Clear Rollup Notice Hashes Write input data to Rollup RX Buffer Write input metadata to Rollup Input Metadata Write HTIF_YIELD_ADVANCE_STATE to `DATA` in `fromhost` End If inspect-state request Write query data to Rollup RX Buffer Write HTIF_YIELD_INSPECT_STATE to `DATA` in `fromhost` End Clear `Y` bit in `iflags` End End If bit `X` in `iflags` is set (i.e. automatic yield) If `REASON` in `tohost` is HTIF_YIELD_REASON_TX_VOUCHER Read voucher data from rollup memory ranges End If `REASON` in `tohost` is HTIF_YIELD_REASON_TX_NOTICE Read notice data from rollup memory ranges End If `REASON` in `tohost` is HTIF_YIELD_REASON_TX_REPORT Read report data from Rollup TX Buffer End End End ``` In production, the host application controlling the emulator is the Server Manager While prototyping, it can be the `cartesi-machine` command-line utility, or even a custom script using the Lua API. ### PMAs The physical memory mapping is described by Physical Memory Attribute records (PMAs) that start at address `0x00000800` (the board shadow) . Each PMA consists of 2 64-bit words. The first word gives the start of a range and the second word its length. These words are readable both internally and externally. Since the ranges must be aligned to 4KiB page boundaries, the lowest 12-bits of each word are available for attributes. The meaning of each attribute field is as follows:
Bits 63–12 11–8 7 6 5 4 3 2 1 0
Field start DID IW IR X W R E IO M
Bits 63–12 11–0
Field length Reserved (=0)
The `M`, `IO`, and `E` bits are mutually exclusive, and respectively mark the range as memory, I/O mapped, or excluded. Bits `R`, `W`, and `X` mark read, write, and execute permissions, respectively. The `IR` and `IW` bits mark the range as idempotent for reads and writes, respectively. Finally, the `DID` gives the device id, which can have the following values:
Name Value
PMA_MEMORY_DID 0
PMA_SHADOW_DID 1
PMA_FLASH_DRIVE_DID 2
PMA_CLINT_DID 3
PMA_HTIF_DID 4
PMA_DHD_DID 5
PMA_ROLLUP_RX_BUFFER_DID 6
PMA_ROLLUP_TX_BUFFER_DID 7
PMA_ROLLUP_INPUT_METADATA_DID 8
PMA_ROLLUP_VOUCHER_HASHES_DID 9
PMA_ROLLUP_NOTICE_HASHES_DID 10
The list of PMA records ends with an invalid PMA entry for which `length=0`. ## Linux setup By default, `pc` starts at `0x1000`, pointing to the start of the ROM region. Before control reaches the RAM image (and ultimately the Linux kernel), a small program residing in ROM builds a [devicetree](http://devicetree.org/) describing the hardware. Cartesi's ROM image `rom.bin` containing this program can be generated from the `rom/` directory of the [Cartesi Machine SDK](https://github.com/cartesi/machine-emulator-sdk). To do so, it goes over the PMA entries identifying the devices and their locations in the physical address space. It also looks for a null-terminated string, starting at the last 4k of the ROM region, that will be used as the command-line for the Linux kernel. Once the devicetree is ready, the ROM program sets register `x10` to 0 (the value of `mhartid`), `x11` to point to the devicetree (which it places at the end of the RAM region), and then jumps to RAM-base at  address `0x80000000`. This is where the entry point of the RAM image is expected to reside. The `dtc` command-line utility can be used to inspect the devicetree: ```bash cartesi-machine \ --append-rom-bootargs="single=yes" \ --rollup \ -- "dtc -I dtb -O dts /sys/firmware/fdt" ``` The result is ``` %machine.target.architecture.dtc ``` The `memory@80000000` section describes 64MiB of RAM starting at address `0x80000000`. The `flash@8000000000000000` describes flash drive 0: a memory region of 60MiB, starting at address `0x8000000000000000`, under the control of the `mtd-ram` driver, with name `flash.0`. This will eventually become available as block device `/dev/mtdblock0`. The `rollup` section specifies the starts and lengths of all rollup memory ranges. The `yield` section specifies that the machine will process automatic and manual yields. Finally, section `chosen` includes the `bootargs` string that will be used as the kernel command-line parameters. Notice the specification of the root file-system pointing to `/dev/mtdblock0`, i.e., `flash.0`, and the `mtdparts` giving it the label `root`. Also notice the command `dtc -I dtb -O dts /sys/firmware/fdt` coming directly from the `cartesi-machine` command line. Linux support for RISC-V is upstream in the [Linux kernel archives](https://www.kernel.org/). The kernel runs in supervisor mode, on top of a Supervisor Binary Interface (SBI) provided by a machine-mode shim: the Berkeley Boot Loader (BBL). The BBL is linked against the Linux kernel and this resulting RAM image is preloaded into RAM. Cartesi's RAM image `linux.bin` can be generated from the `kernel/` directory of the [Cartesi Machine SDK](https://github.com/cartesi/machine-emulator-sdk). The SBI provides a simple interface through which the kernel interacts with CLINT and HTIF. Besides implementing the SBI, the BBL also installs a trap that catches invalid instruction exceptions. This mechanism can be used, for example, to emulate floating-point instructions, although it is more efficient to setup the target toolchain to replace floating point instructions with calls to a soft-float implementation instead. After installing the trap, BBL switches to supervisor mode and cedes control to the kernel entry point. After completing its own initialization, the kernel mounts the root file-system and eventually cedes control to `/sbin/init`. Cartesi's root file-system `rootfs.ext2` can be generated from the `fs/` directory in the [Cartesi Machine SDK](https://github.com/cartesi/machine-emulator-sdk). The Cartesi-provided `/sbin/init` script scans all flash devices `/dev/mtdblock1`–`/dev/mtdblock7` for valid file-systems. When a file-system is found, the script obtains the corresponding `