ethereum/execution-spec-tests is both a collection of test cases and a framework implemented in Python to generate tests for Ethereum execution clients.

The framework collects and executes the test cases in order to generate test fixtures (JSON) which can be consumed by any execution client to verify their implementation of ethereum/execution-specs. Currently, the fixtures, which define state transition and block tests, are generated by the framework using the t8n command from the ethereum/go-ethereum evm command-line tool. Integration of other t8n tools is planned.

---
title: Test Fixture Generation with execution-spec-tests
---
flowchart LR
  style C stroke:#333,stroke-width:2px
  style D stroke:#333,stroke-width:2px
  style G stroke:#F9A825,stroke-width:2px
  style H stroke:#F9A825,stroke-width:2px
  
  subgraph ethereum/go-ethereum
    C[<code>evm t8n</code>\nexternal executable]
  end

  subgraph ethereum/solidity
    D[<code>solc</code>\nexternal executable]
  end

  subgraph ethereum/EIPs
    E(<code>EIPS/EIP-*.md</code>\nSHA digest via Github API)
  end

  subgraph "ethereum/execution-spec-tests"
    A(<code>./tests/**/*.py</code>\nPython Test Cases)
    B([<code>$ fill ./tests/</code>\nPython Framework])
  end

  subgraph Test Fixture Consumers
    subgraph ethereum/hive
      G([<code>$ hive ...</code>\nGo Test Framework])
    end
    H([Client executables])
  end

  C <-.-> B  
  D <-.-> B
  A --> B
  E <-.-> |retrieve latest spec version\ncheck tested spec version| B
  B -->|output| F(<code>./fixtures/**/*.json</code>\nJSON Test Fixtures)
  F -->|input| G
  F -->|input| H

The generated test fixtures can be used:

1. Directly by client teams' test frameworks, and,
2. In the integration tests executed in the ethereum/hive framework.

The following requires a Python 3.10 or Python 3.11 installation.

This guide installs stable versions of the required external evm and solc executables and will only enable generation of test fixtures for features deployed to mainnet. In order to generate fixtures for features under active development, you can follow the steps below and then follow the additional steps in the online doc.

1. Ensure go-ethereum's evm tool and solc are in your path. Either build the required versions, or alternatively:

   sudo add-apt-repository -y ppa:ethereum/ethereum
   sudo apt-get update
   sudo apt-get install ethereum solc

   More help:

   • geth installation doc.
   • solc installation doc.

   Help for other platforms is available in the online doc.

2. Clone the execution-spec-tests repo and install its and dependencies (it's recommended to use a virtual environment for the installation):

   git clone https://github.com/ethereum/execution-spec-tests
   cd execution-spec-tests
   python3 -m venv ./venv/
   source ./venv/bin/activate
   pip install -e '.[docs,lint,test]'

3. Verify the installation:

   1. Explore test cases:

      fill --collect-only

      Expected console output:

   2. Execute the test cases (verbosely) in the ./tests/berlin/eip2930_access_list/test_acl.py module:

      fill -v tests/berlin/eip2930_access_list/test_acl.py

      Expected console output: Check:

      1. The versions of the evm and solc tools are as expected (your versions may differ from those in the highlighted box).

      2. The corresponding fixture file has been generated:

         head fixtures/berlin/eip2930_access_list/acl/access_list.json

The available test cases can be browsed in the Test Case Reference doc.

See the online documentation for further help with working with this codebase:

1. Learn useful command-line flags.
2. Execute tests for features under development via the --from=FORK1 and --until=FORK2 flags.
3. Optional: Configure VS Code to auto-format Python code and execute tests within VS Code.
4. Implement a new test case, see Writing Tests.

Contributions and feedback are welcome. Please see the online documentation for this repository's coding standards and help on implementing new tests.

This project is licensed under the MIT License - see the LICENSE file for details.