A headless workflow execution engine for bioinformatics that supports local, cloud, and HPC.

Request Feature · Report Bug · ⭐ Consider starring the repo! ⭐

crankshaft is a headless workflow execution engine written in Rust: it's being developed in coordination with the sprocket project with the goal of enabling large-scale bioinformatics analyses. There is no associated crankshaft command line tool—the end user is really engine developers who want to include it as a core task execution library in their own command line tools.

• crankshaft aims to be a high-performance workflow execution engine capable of concurrently managing and executing upwards of 20,000 concurrent tasks. The core focus is enabling middle- to large-scale bioinformatics analyses, though it can also be used to design smaller scale execution engines.
• crankshaft is headless, which means that it doesn't do anything on its own; in fact, it must be driven by some external orchestration code. This allows the crankshaft library itself to focus on performance improvements that can be enjoyed across the entire community.
• crankshaft is developed independently of any particular workflow language. Though it's part of the Sprocket project, it's not based on WDL, and, in theory, multiple frontends based on different workflow languages can exist (and we hope this is the case)!

To use crankshaft, you'll need to install Rust. We recommend using rustup to accomplish this. Once Rust is installed, you can create a new project and add the latest version of crankshaft using the following command.

cargo add crankshaft

Once you've added crankshaft to your dependencies, you should head over to the /examples to see how you can use the library in your projects.

To bootstrap a development environment, please use the following commands.

# Clone the repository
git clone [email protected]:stjude-rust-labs/crankshaft.git
cd crankshaft

# Build the crate in release mode
cargo build --release

Before submitting any pull requests, please make sure the code passes the following checks (from the root directory).

# Run the project's tests.
cargo test --all-features

# Run the tests for the examples.
cargo test --examples --all-features

# Ensure the project doesn't have any linting warnings.
cargo clippy --all-features

# Ensure the project passes `cargo fmt`.
cargo fmt --check

# Ensure the docs build.
cargo doc

Contributions, issues and feature requests are welcome! Feel free to check issues page.

This project is licensed as either Apache 2.0 or MIT at your discretion.

Copyright © 2024-Present St. Jude Children's Research Hospital.