A PBWT-based light index for UK Biobank scale genotype data.
μ-PBWT is available for Gnu/Linux on conda (bioconda channel):
conda install -c bioconda mupbwt
Prepare the cmake for building the current project in ‘.’ into the ‘build’ folder
cmake -S . -B build
Build μ-PBWT:
cmake --build build
Install μ-PBWT (default in /usr/local/bin/
, sudo
required):
cmake --install build
Use --prefix <path>
for custom path.
File format supported:
- BCF/VCF
- MaCS
cd build
Usage: ./mupbwt [options]
Options:
-i, --input_file <path> vcf/bcf file for panel
-s, --save <path> path to save index
-l, --load <path> path to load index
-o, --output <path> path to query output
-q, --query <path> path to query file (vcf/bcf)
-m, --macs use macs as file format for both input and query file
-v, --verbose extra prints
-d, --details print memory usage details
-h, --help show this help message and exit
Build the index:
./mupbwt -i <input file> -s <index file>
Query the index:
./mupbwt -l <index file> -q <query file> -o <output file>
Query without save the index:
./mupbwt -i <input file> -q <query file> -o <output file>
Query and save the index:
./mupbwt -i <input file> -s <index file> -q <query file> -o <output file>
Using examples in sample_data
:
./mupbwt -i sample_data/panel.bcf -s sample_data/index.ser
./mupbwt -l sample_data/index.ser -q sample_data/query.bcf -o sample_data/sample_data_results
./mupbwt -i sample_data/panel.bcf -q sample_data/query.bcf -o sample_data/sample_data_results
./mupbwt -i sample_data/panel.bcf -s sample_data/index.ser -q sample_data/query.bcf -o sample_data/sample_data_results
Load the index and print details to stdout:
./mupbwt -l <index file> -d
An output example is:
> ./mupbwt -l sample_data/index.ser -d
built/loaded in: 0.015628 s
----
Total haplotypes: 900
Total sites: 499
----
Total runs: 27512
Average runs: 55
----
run: 0.0386925 megabytes
thr: 0.0387306 megabytes
uv: 0.0380135 megabytes
samples: 0.0833178 megabytes
rlpbwt (mapping): 0.201148 megabytes
phi panels: 0.414757 megabytes
phi support: 0.126385 megabytes
phi data structure (panels support): 0.541142 megabytes
rlpbwt: 0.74229 megabytes
----
estimated dense size: 36.4132 megabytes
----
Only bialleic case is supported. In case of vcf/bcf bcftools can be used to filter the input:
bcftools view -m2 -M2 -v snps <input vcf/vcf> > <filtered vcf/bcf file>
Output file follow the standard proposed in Durbin's PBWT. Each row contain a SMEM:
MATCH <query index> <row index> <staring column> <ending column> <SMEM length>
For example:
MATCH 99 150 414 430 17
Row index and query index are incrementally so the name of the sample and the precise haplotype can be calculated using the output of bcftools.
The command:
bcftools query -l <input vcf/bcf> > samples.txt
store in samples.txt
all the samples name, in order. So, for example, row indices 0 and 1 corresponds to the two haplotypes of the first sample, row indices 2 and 3 to the second one etc...
Eventually you can use script/mem_sample.py
:
> python mem_sample.py -h
usage: mem_sample.py [-h] [-i INPUT] [-p PANEL] [-q QUERIES] [-o OUTPUT]
options:
-h, --help show this help message and exit
-i INPUT, --input INPUT
SMEM file in Durbin's format
-p PANEL, --panel PANEL
panel as VCF/BCF (optional)
-q QUERIES, --queries QUERIES
queries as VCF/BCF (optional)
-o OUTPUT, --output OUTPUT
output file
Esample:
python mem_sample.py -i sample_data/sample_data_results -p sample_data/panel.bcf -q sample_data/query.bcf -o sample_data/sample_data_results_new
Only one between PANEL
and QUERIES
can be specified.
New SMEM file will contain in each row:
MATCH <querySample_haplotype> <panelSample_haplotype> <staring column> <ending column> <SMEM length>
For example, assuming both PANEL
and QUERIES
:
MATCH 1318026_1 4919834_0 414 430 17
Results on high-coverage whole genome sequencing data from UK Biobank (chromosome 20):
Region | #Samples | #Sites | Size BCF (GB) | μ-PBWT (GB) | Construction time (hh:mm) | Construction memory peak (GB) |
---|---|---|---|---|---|---|
chr20:60061-4060065 | 150119 | 865267 | 1.9 | 0.88 | 06:25 | 2.27 |
chr20:4060066-8060066 | 150119 | 880899 | 2 | 0.85 | 06:28 | 2.22 |
chr20:8060067-12515479 | 150119 | 961591 | 2.1 | 0.77 | 07:04 | 2.05 |
chr20:12515480-16768988 | 150119 | 917468 | 2 | 0.73 | 06:47 | 1.97 |
chr20:16768989-21050967 | 150119 | 931010 | 2 | 0.71 | 06:53 | 1.92 |
chr20:21050968-31549151 | 150119 | 1919134 | 4.2 | 1.20 | 13:54 | 3.06 |
chr20:31549152-38282825 | 150119 | 1436549 | 2.8 | 0.99 | 10:25 | 2.63 |
chr20:38282826-43181963 | 150119 | 1056144 | 2.2 | 0.76 | 07:42 | 2.06 |
chr20:43181964-47619489 | 150119 | 955970 | 2 | 0.79 | 06:56 | 2.09 |
chr20:47619490-51789198 | 150119 | 923178 | 2 | 0.80 | 06:44 | 2.12 |
chr20:51789199-55789212 | 150119 | 911452 | 2 | 0.81 | 06:45 | 2.13 |
chr20:55789213-59874964 | 150119 | 925442 | 2 | 0.84 | 06:49 | 2.20 |
chr20:59874965-64334101 | 150119 | 1096089 | 2.4 | 0.93 | 08:00 | 2.42 |
Total | 150119 | 13780193 | 29.6 | 11.08 | - | 29.15 |
Results on 1000 Genome Project phase 3 data including the average number of runs per site:
Chr | #Samples | #Sites | #Runs/site | Size BCF (GB) | μ-PBWT (GB) | Construction time (hh:mm) | Construction memory peak (GB) |
---|---|---|---|---|---|---|---|
1 | 2454 | 6196151 | 11 | 0.78 | 1.44 | 00:19 | 4.59 |
2 | 2454 | 6786300 | 10 | 0.84 | 1.47 | 00:21 | 4.76 |
3 | 2454 | 5584397 | 10 | 0.71 | 1.20 | 00:18 | 4.24 |
4 | 2454 | 5480936 | 10 | 0.71 | 1.19 | 00:17 | 4.28 |
5 | 2454 | 5037955 | 9 | 0.63 | 1.08 | 00:16 | 4.22 |
6 | 2454 | 4800101 | 10 | 0.64 | 1.06 | 00:15 | 4.28 |
7 | 2454 | 4517734 | 10 | 0.58 | 1.03 | 00:14 | 4.34 |
8 | 2454 | 4417368 | 10 | 0.56 | 0.97 | 00:14 | 4.30 |
9 | 2454 | 3414848 | 11 | 0.43 | 0.81 | 00:11 | 2.54 |
10 | 2454 | 3823786 | 10 | 0.50 | 0.87 | 00:12 | 2.77 |
11 | 2454 | 3877543 | 10 | 0.49 | 0.84 | 00:12 | 2.71 |
12 | 2454 | 3698099 | 10 | 0.47 | 0.82 | 00:12 | 2.63 |
13 | 2454 | 2727881 | 10 | 0.35 | 0.60 | 00:9 | 2.14 |
14 | 2454 | 2539149 | 11 | 0.32 | 0.58 | 00:8 | 2.18 |
15 | 2454 | 2320474 | 12 | 0.29 | 0.57 | 00:7 | 2.30 |
16 | 2454 | 2596072 | 12 | 0.32 | 0.63 | 00:8 | 2.28 |
17 | 2454 | 2227080 | 12 | 0.28 | 0.55 | 00:7 | 2.32 |
18 | 2454 | 2171378 | 11 | 0.28 | 0.51 | 00:7 | 2.23 |
19 | 2454 | 1751878 | 13 | 0.23 | 0.45 | 00:6 | 1.43 |
20 | 2454 | 1739315 | 11 | 0.22 | 0.41 | 00:5 | 1.30 |
21 | 2454 | 1054447 | 14 | 0.14 | 0.30 | 00:3 | 1.26 |
22 | 2454 | 1055454 | 14 | 0.14 | 0.29 | 00:3 | 1.24 |
Total | 2454 | 77818346 | 11 | 9.91 | 17.67 | - | 64.34 |
Note that total building times are not printed due to the fact that all the computations have been done in parallel.
The pipeline for 1000 Genome Project phase 3 data is available at dlcgold/muPBWT-1KGP-workflow.
μ-PBWT results are currently available on Bioinformatics.
Bibtex:
@article{10.1093/bioinformatics/btad552,
author = {Cozzi, Davide and Rossi, Massimiliano and Rubinacci, Simone and Gagie, Travis and Köppl, Dominik and Boucher, Christina and Bonizzoni, Paola},
title = "{μ- PBWT: a lightweight r-indexing of the PBWT for storing and querying UK Biobank data}",
journal = {Bioinformatics},
volume = {39},
number = {9},
pages = {btad552},
year = {2023},
month = {09},
abstract = "{The Positional Burrows–Wheeler Transform (PBWT) is a data structure that indexes haplotype sequences in a manner that enables finding maximal haplotype matches in h sequences containing w variation sites in O(hw) time. This represents a significant improvement over classical quadratic-time approaches. However, the original PBWT data structure does not allow for queries over Biobank panels that consist of several millions of haplotypes, if an index of the haplotypes must be kept entirely in memory.In this article, we leverage the notion of r-index proposed for the BWT to present a memory-efficient method for constructing and storing the run-length encoded PBWT, and computing set maximal matches (SMEMs) queries in haplotype sequences. We implement our method, which we refer to as μ-PBWT, and evaluate it on datasets of 1000 Genome Project and UK Biobank data. Our experiments demonstrate that the μ-PBWT reduces the memory usage up to a factor of 20\\% compared to the best current PBWT-based indexing. In particular, μ-PBWT produces an index that stores high-coverage whole genome sequencing data of chromosome 20 in about a third of the space of its BCF file. μ-PBWT is an adaptation of techniques for the run-length compressed BWT for the PBWT (RLPBWT) and it is based on keeping in memory only a succinct representation of the RLPBWT that still allows the efficient computation of set maximal matches (SMEMs) over the original panel.Our implementation is open source and available at https://github.com/dlcgold/muPBWT. The binary is available at https://bioconda.github.io/recipes/mupbwt/README.html.}",
issn = {1367-4811},
doi = {10.1093/bioinformatics/btad552},
url = {https://doi.org/10.1093/bioinformatics/btad552},
eprint = {https://academic.oup.com/bioinformatics/article-pdf/39/9/btad552/51556136/btad552.pdf},
}