Rocket Chip Generator

This repository contains the Rocket chip generator necessary to instantiate the RISC-V Rocket Core. For more information on Rocket Chip, please consult our technical report.

Table of Contents

• Quick instructions for those who want to dive directly into the details without knowing exactly what's in the repository.
• What's in the Rocket chip generator repository?
• How should I use the Rocket chip generator?
  • Using the high-performance cycle-accurate C++ emulator
  • Mapping a Rocket core down to an FPGA
  • Pushing a Rocket core through the VLSI tools
• How can I parameterize my Rocket chip?
• Contributors

Quick Instructions

Checkout The Code

$ git clone https://github.com/ucb-bar/rocket-chip.git
$ cd rocket-chip
$ git submodule update --init

Setting up the RISCV environment variable

To build the rocket-chip repository, you must point the RISCV environment variable to your riscv-tools installation directory.

$ export RISCV=/path/to/riscv/toolchain/installation

The riscv-tools repository is already included in rocket-chip as a git submodule. You must build this version of riscv-tools:

$ cd rocket-chip/riscv-tools
$ git submodule update --init --recursive
$ export RISCV=/path/to/install/riscv/toolchain
$ ./build.sh

For more information (or if you run into any issues), please consult the riscv-tools/README.

Building The Project

First, to build the C simulator:

$ cd emulator
$ make

Or to build the VCS simulator:

$ cd vsim
$ make

In either case, you can run a set of assembly tests or simple benchmarks (Assuming you have N cores on your host system):

$ make -jN run-asm-tests
$ make -jN run-bmark-tests

To build a C simulator that is capable of VCD waveform generation:

$ cd emulator
$ make debug

And to run the assembly tests on the C simulator and generate waveforms:

$ make -jN run-asm-tests-debug
$ make -jN run-bmark-tests-debug

To generate FPGA-synthesizable verilog (output will be in fsim/generated-src):

$ cd fsim
$ make verilog

Similarly, to generate VLSI-synthesizable verilog (output will be in vsim/generated-src):

$ cd vsim
$ make verilog

Keeping Your Repo Up-to-Date

If you are trying to keep your repo up to date with this github repo, you also need to keep the submodules and tools up to date.

$ # Get the newest versions of the files in this repo
$ git pull origin master
$ # Make sure the submodules have the correct versions
$ git submodule update --init --recursive

If riscv-tools version changes, you should recompile and install riscv-tools according to the directions in the riscv-tools/README.

$ cd riscv-tools
$ ./build.sh

If firrtl version changes and you are using Chisel3, you may need to clean and recompile:

$ cd firrtl
$ sbt clean
$ sbt assembly

What's in the Rocket chip generator repository?

The rocket-chip repository is the head git repository that points to many sub-repositories (e.g. the riscv-tools repository) using git submodules. While we're aware of the ongoing debate as to how meta-projects should be managed (i.e. a big monolithic repository vs. smaller repositories tracked as submodules), we've found that for our chip-building projects at Berkeley, the ability to compose a subset of private and public sub-repositories on a per-chip basis is a killer feature of git submodule.

The Submodules

Here's a look at all the git submodules that are currently tracked in the rocket-chip repository:

• chisel2 (https://github.com/ucb-bar/chisel): At Berkeley, we write RTL in Chisel. For those who are not familiar with Chisel, please go take a look at http://chisel.eecs.berkeley.edu. We have submoduled a specific git commit tag of the Chisel compiler rather than pointing to a versioned Chisel release as an external dependency; so far we were developing Chisel and the rocket core at the same time, and hence it was easiest to use submodule to track bleeding edge commits to Chisel, which contained a bunch of new features and bug fixes. As Chisel gets more stable, we will likely replace this submodule with an external dependency.
• chisel3 (https://github.com/ucb-bar/chisel3): Chisel3 is a newer version of Chisel, which is based on FIRRTL. The Chisel code in this repository is generally compatible with both Chisel2 and Chisel3. The chisel3/README.gives instructions on how to build your design with Chisel3 instead of Chisel2.
• firrtl (https://github.com/ucb-bar/firrtl): FIRRTL (Flexible Internal Representation for RTL) is the intermediate format which Chisel3 is based upon. The Chisel3 compiler generates a FIRRTL representation, from which the final product (Verilog code, C code, etc) is generated.
• rocket (https://github.com/ucb-bar/rocket): The rocket repository holds the actual source code of the Rocket core. Note that the L1 blocking I$ and the L1 non-blocking D$ are considered part of the core, and hence we keep the L1 cache source code in this repository. This repository is not meant to stand alone; it needs to be included in a chip repository (e.g. rocket-chip) that instantiates the core within a memory system and connects it to the outside world.
• uncore (https://github.com/ucb-bar/uncore): This repository implements the uncore logic, such as the coherence hub (the agent that keeps multiple L1 D$ coherent). The definition of the coherent interfaces between tiles ("tilelink") and the debug interface also live in this repository.
• junctions (https://github.com/ucb-bar/junctions): This repository contains code and converters for various bus protocols and interfaces.
• hardfloat (https://github.com/ucb-bar/berkeley-hardfloat): This repository holds the parameterized IEEE 754-2008 compliant floating-point units for fused multiply-add operations, conversions between integer and floating-point numbers, and conversions between floating-point conversions with different precision. The floating-point units in this repository work on an internal recoded format (exponent has an additional bit) to handle subnormal numbers more efficiently in the processor. Please take a look at the README in the repository for more information.
• context-dependent-environments (https://github.com/ucb-bar/context-dependent-environments): The rocket-chip Chisel code is highly parameterizable, and utilizes the classes in this subrepo to set and pass parameters to different levels of the design. Note that in Chisel2, this was handled by Chisel itself, but has been moved into a seperate library for use with Chisel3.
• dramsim2 (https://github.com/dramninjasUMD/DRAMSim2): Currently, the DRAM memory system is implemented in the testbench. We use dramsim2 to emulate DRAM timing.
• fpga-zynq (https://github.com/ucb-bar/fpga-zynq): We also tag a version of the FPGA infrastructure that works with the RTL committed in the rocket-chip repository.
• riscv-tools (https://github.com/riscv/riscv-tools): We tag a version of riscv-tools that works with the RTL committed in the rocket-chip repository. Once the software toolchain stabilizes, we might turn this submodule into an external dependency.
• groundtest (https://github.com/ucb-bar/groundtest): This repository contains code which can test the uncore by generating randomized instruction streams. It replaces the rocket processor with an instruction stream generator to stress-test the uncore portions of the design.
• torture (https://github.com/ucb-bar/torture): The torture test code is used to generate randomized instruction streams which are then run as code on the rocket core(s). These are constrained random tests to stress-test both the core and uncore portions of the design.

The Top Level Module

Next, take a look at rocket-chip's src/main/scala directory. This directory has the Chisel source files including the top level RocketChip.scala, which instantiates both a Rocket core and the uncore logic, and then glues them together. Here's a brief overview of source files found in the rocket-chip repository:

• RocketChip.scala: Top-level source file (Top is the top-level module name), which instantiates a Rocket core, uncore logic, and glues them together.
• Network.scala: This source file holds the crossbar network used in the uncore for multi-core implementations.
• Configs.scala: This holds all the rocket-chip parameters. Probably this file is the most important file for external users. We will revisit this topic in the next section "How should I use the Rocket chip generator?", and will also post a more detailed explanation of the parameter infrastructure in the near future.
• Backends.scala: An example of how the Chisel compiler's VLSI backend can be extended to route a pin named "init" to all SRAM blocks used in the design. This separation cleans up the source RTL of the design, since we don't need to add all the vendor-specific stuff in the Chisel source code, yet still can correctly hook up our particular SRAMs. The transformation is just a "compiler pass" in the Chisel backend that happens as the compiler translates the Chisel source code down to Verilog. Pretty neat huh?
• Vlsi.scala: This file is pretty specific to our tapeouts. It implements logic to interface with an arbitrary number of slow single-ended digital I/Os when implementing a test chip.

Take a look at the top-level I/O pins. Open up src/main/scala/RocketChip.scala, and search for TopIO. You will read the following:

/** Top-level io for the chip */
class BasicTopIO(implicit val p: Parameters) extends ParameterizedBundle()(p)
    with HasTopLevelParameters

class TopIO(implicit p: Parameters) extends BasicTopIO()(p) {

  val mem_axi = Vec(nMemAXIChannels, new NastiIO)
  val mem_ahb = Vec(nMemAHBChannels, new HastiMasterIO)
  val interrupts = Vec(p(NExtInterrupts), Bool()).asInput
  val mmio_axi = Vec(p(NExtMMIOAXIChannels), new NastiIO)
  val mmio_ahb = Vec(p(NExtMMIOAHBChannels), new HastiMasterIO)
  val debug = new DebugBusIO()(p).flip
}

There are 4 major I/O ports coming out of the top-level module:

• Debug interface (debug): The debug interface can be used to both debug the processor as it is executing, and to read and write memory. It is slated to repalce the host interface in the near future.
• **High-performance memory interface (mem_*) : Memory requests from the processor comes out the mem_ ports. Depending on the configuration of the design, these may be visible as AXI or AHB protocol. The mem_ port(s) uses the same uncore clock, and is intended to be connected to something on the same chip.
• ** Memory mapped I/O interface (mmio_*) *: The optional mmio_ interfaces can be used to communicate with devices on the chip but outside of the rocket-chip boundary. Depending on the configuration of the design, these may be visible as AXI or AHB.
• ** Interrupts interface (interrupts) **: This interface is used to deliver external interrupts to the processor core.

Of course, there's a lot more in the actual submodules, but hopefully this would be enough to get you started with using the Rocket chip generator. We will keep documenting more about our designs in the respective README of each submodules, release notes, and even blog posts. In the mean time, please post questions to the hw-dev mailing list.

How should I use the Rocket chip generator?

Chisel can generate code for three targets: a high-performance cycle-accurate C++ emulator, Verilog optimized for FPGAs, and Verilog for VLSI. The rocket-chip generator can target all three backends. You will need a Java runtime installed on your machine, since Chisel is overlaid on top of Scala. Chisel RTL (i.e. rocket-chip source code) is a Scala program executing on top of your Java runtime. To begin, ensure that the ROCKETCHIP environment variable points to the rocket-chip repository.

$ git clone https://github.com/ucb-bar/rocket-chip.git
$ cd rocket-chip
$ export ROCKETCHIP=`pwd`
$ git submodule update --init
$ cd riscv-tools
$ git submodule update --init --recursive riscv-tests

Before going any further, you must point the RISCV environment variable to your riscv-tools installation directory. If you do not yet have riscv-tools installed, follow the directions in the riscv-tools/README.

export RISCV=/path/to/install/riscv/toolchain

Otherwise, you will see the following error message while executing any command in the rocket-chip generator:

*** Please set environment variable RISCV. Please take a look at README.

1) Using the high-performance cycle-accurate C++ emulator

Your next step is to get the C++ emulator working. Assuming you have N cores on your host system, do the following:

$ cd $ROCKETCHIP/emulator
$ make -jN run

By doing so, the build system will generate C++ code for the cycle-accurate emulator, compile the emulator, compile all RISC-V assembly tests and benchmarks, and run both tests and benchmarks on the emulator. If make finished without any errors, it means that the generated Rocket chip has passed all assembly tests and benchmarks!

You can also run assembly tests and benchmarks separately:

$ make -jN run-asm-tests
$ make -jN run-bmark-tests

To generate vcd waveforms, you can run one of the following commands:

$ make -jN run-debug
$ make -jN run-asm-tests-debug
$ make -jN run-bmark-tests-debug

Or call out individual assembly tests or benchmarks:

$ make output/rv64ui-p-add.out
$ make output/rv64ui-p-add.vcd

Now take a look in the emulator/generated-src directory. You will find Chisel generated C++ code.

$ ls $ROCKETCHIP/emulator/generated-src
Top.DefaultConfig-0.cpp
Top.DefaultConfig-0.o
Top.DefaultConfig-1.cpp
Top.DefaultConfig-1.o
Top.DefaultConfig-2.cpp
Top.DefaultConfig-2.o
Top.DefaultConfig-3.cpp
Top.DefaultConfig-3.o
Top.DefaultConfig-4.cpp
Top.DefaultConfig-4.o
Top.DefaultConfig-5.cpp
Top.DefaultConfig-5.o
Top.DefaultConfig.cpp
Top.DefaultConfig.h
emulator.h
emulator_api.h
emulator_mod.h

Also, output of the executed assembly tests and benchmarks can be found at emulator/output/*.out. Each file has a cycle-by-cycle dump of write-back stage of the pipeline. Here's an excerpt of emulator/output/rv64ui-p-add.out:

C0: 483 [1] pc=[00000002138] W[r 3=000000007fff7fff][1] R[r 1=000000007fffffff] R[r 2=ffffffffffff8000] inst=[002081b3] add s1, ra, s0
C0: 484 [1] pc=[0000000213c] W[r29=000000007fff8000][1] R[r31=ffffffff80007ffe] R[r31=0000000000000005] inst=[7fff8eb7] lui t3, 0x7fff8
C0: 485 [0] pc=[00000002140] W[r 0=0000000000000000][0] R[r 0=0000000000000000] R[r 0=0000000000000000] inst=[00000000] unknown

The first [1] at cycle 483, core 0, shows that there's a valid instruction at PC 0x2138 in the writeback stage, which is 0x002081b3 (add s1, ra, s0). The second [1] tells us that the register file is writing r3 with the corresponding value 0x7fff7fff. When the add instruction was in the decode stage, the pipeline had read r1 and r2 with the corresponding values next to it. Similarly at cycle 484, there's a valid instruction (lui instruction) at PC 0x213c in the writeback stage. At cycle 485, there isn't a valid instruction in the writeback stage, perhaps, because of a instruction cache miss at PC 0x2140.

2) Mapping a Rocket core down to an FPGA

We use Synopsys VCS for Verilog simulation. We acknowledge that using a proprietary Verilog simulation tool for an open-source project is not ideal; we ask the community to help us move DirectC routines (VCS's way of gluing Verilog testbenches to arbitrary C/C++ code) into DPI/VPI routines so that we can make Verilog simulation work with an open-source Verilog simulator. In the meantime, you can use the C++ emulator to generate vcd waveforms, which you can view with an open-source waveform viewer such as GTKWave.

So assuming you have a working Rocket chip, you can generate Verilog for the FPGA tools with the following commands:

$ cd $ROCKETCHIP/fsim
$ make verilog

The Verilog used for the FPGA tools will be generated in fsim/generated-src. Please proceed further with the directions shown in the README of the fpga-zynq repository.

However, if you have access to VCS, you will be able to run assembly tests and benchmarks with the following commands (again assuming you have N cores on your host machine):

$ cd $ROCKETCHIP/fsim
$ make -jN run

The generated output looks similar to those generated from the emulator. Look into fsim/output/*.out for the output of the executed assembly tests and benchmarks.

3) Pushing a Rocket core through the VLSI tools

You can generate Verilog for your VLSI flow with the following commands:

$ cd $ROCKETCHIP/vsim
$ make verilog

Now take a look at vsim/generated-src, and the contents of the Top.DefaultConfig.conf file:

$ cd $ROCKETCHIP/vsim/generated-src
Top.DefaultConfig.conf
Top.DefaultConfig.prm
Top.DefaultConfig.v
consts.DefaultConfig.vh
$ cat $ROCKETCHIP/vsim/generated-src/*.conf
name MetadataArray_tag_arr depth 128 width 84 ports mwrite,read mask_gran 21
name ICache_tag_array depth 128 width 38 ports mrw mask_gran 19
name DataArray_T6 depth 512 width 128 ports mwrite,read mask_gran 64
name HellaFlowQueue_ram depth 32 width 133 ports write,read
name ICache_T157 depth 512 width 128 ports rw

The conf file contains information for all SRAMs instantiated in the flow. If you take a close look at the $ROCKETCHIP/Makefrag, you will see that during Verilog generation, the build system calls a $(mem_gen) script with the generated configuration file as an argument, which will fill in the Verilog for the SRAMs. Currently, the $(mem_gen) script points to vsim/vlsi_mem_gen, which simply instantiates behavioral SRAMs. You will see those SRAMs being appended at the end of vsim/generated-src/Top.DefaultConfig.v. To target vendor-specific SRAMs, you will need to make necessary changes to vsim/vlsi_mem_gen.

Similarly, if you have access to VCS, you can run assembly tests and benchmarks with the following commands (again assuming you have N cores on your host machine):

$ cd $ROCKETCHIP/vsim
$ make -jN run

The generated output looks similar to those generated from the emulator. Look into vsim/output/*.out for the output of the executed assembly tests and benchmarks.

How can I parameterize my Rocket chip?

By now, you probably figured out that all generated files have a configuration name attached, e.g. DefaultConfig. Take a look at src/main/scala/Configs.scala. Search for NSets and NWays defined in BaseConfig. You can change those numbers to get a Rocket core with different cache parameters. For example, by changing L1I, NWays to 4, you will get a 32KB 4-way set-associative L1 instruction cache rather than a 16KB 2-way set-associative L1 instruction cache.

Further down, you will be able to see two FPGA configurations: DefaultFPGAConfig and DefaultFPGASmallConfig. DefaultFPGAConfig inherits from BaseConfig, but overrides the low-performance memory port (i.e., backup memory port) to be turned off. This is because the high-performance memory port is directly connected to the high-performance AXI interface on the ZYNQ FPGA. DefaultFPGASmallConfig inherits from DefaultFPGAConfig, but changes the cache sizes, disables the FPU, turns off the fast early-out multiplier and divider, and reduces the number of TLB entries (all defined in SmallConfig). This small configuration is used for the Zybo FPGA board, which has the smallest ZYNQ part.

Towards the end, you can also find that ExampleSmallConfig inherits all parameters from BaseConfig but overrides the same parameters of SmallConfig.

Now take a look at fsim/Makefile and vsim/Makefile. Search for the CONFIG variable. DefaultFPGAConfig is used for the FPGA build, while DefaultConfig is used for the VLSI build. You can also change the CONFIG variable on the make command line:

$ cd $ROCKETCHIP/vsim
$ make -jN CONFIG=ExampleSmallConfig run-asm-tests

Or, even by defining CONFIG as an environment variable:

$ export CONFIG=ExampleSmallConfig
$ make -jN run-asm-tests

This parameterization is one of the many strengths of processor generators written in Chisel, and will be more detailed in a future blog post, so please stay tuned.

To override specific configuration items, such as the number of external interrupts, you can create your own Configuration(s) and compose them with Config's ++ operator

class WithNExtInterrupts extends Config (nExt: Int) {
  (pname, site, here) => pname match {
  case (NExtInterrupts => nExt)
  }
} 
class MyConfig extends Config (new WithNExtInterrupts(16) ++ new DefaultSmallConfig)

Then you can build as usual with CONFIG=MyConfig.

Contributors

• Scott Beamer
• Henry Cook
• Yunsup Lee
• Stephen Twigg
• Huy Vo
• Andrew Waterman

Attribution

If used for research, please cite Rocket Chip by the technical report:

Krste Asanović, Rimas Avižienis, Jonathan Bachrach, Scott Beamer, David Biancolin, Christopher Celio, Henry Cook, Palmer Dabbelt, John Hauser, Adam Izraelevitz, Sagar Karandikar, Benjamin Keller, Donggyu Kim, John Koenig, Yunsup Lee, Eric Love, Martin Maas, Albert Magyar, Howard Mao, Miquel Moreto, Albert Ou, David Patterson, Brian Richards, Colin Schmidt, Stephen Twigg, Huy Vo, and Andrew Waterman, The Rocket Chip Generator, Technical Report UCB/EECS-2016-17, EECS Department, University of California, Berkeley, April 2016