Skip to content
/ muPBWT Public

A PBWT-based light index for UK Biobank scale genotype data.

License

Notifications You must be signed in to change notification settings

dlcgold/muPBWT

Repository files navigation

install with bioconda Conda Conda GitHub stars

μ-PBWT

A PBWT-based light index for UK Biobank scale genotype data.

Conda install

μ-PBWT is available for Gnu/Linux on conda (bioconda channel):

conda install -c bioconda mupbwt

Build from source

Prepare the cmake for building the current project in ‘.’ into the ‘build’ folder

cmake -S . -B build 

Build μ-PBWT:

cmake --build build

Install from source (optional)

Install μ-PBWT (default in /usr/local/bin/, sudo required):

cmake --install build

Use --prefix <path> for custom path.

Usage

File format supported:

cd build
Usage: ./mupbwt [options]

Options:
  -i, --input_file <path>	 vcf/bcf file for panel
  -s, --save <path>	  path to save index
  -l, --load <path>	 path to load index
  -o, --output <path>	 path to query output
  -q, --query <path>	 path to query file (vcf/bcf)
  -m, --macs	use macs as file format for both input and query file
  -v, --verbose	 extra prints
  -d, --details	 print memory usage details
  -h, --help	 show this help message and exit

Build the index:

./mupbwt -i <input file> -s <index file>

Query the index:

./mupbwt -l <index file> -q <query file> -o <output file> 

Query without save the index:

./mupbwt -i <input file> -q <query file> -o <output file>

Query and save the index:

./mupbwt -i <input file> -s <index file> -q <query file> -o <output file>

Using examples in sample_data:

./mupbwt -i sample_data/panel.bcf -s sample_data/index.ser
./mupbwt -l sample_data/index.ser -q sample_data/query.bcf -o sample_data/sample_data_results 
./mupbwt -i sample_data/panel.bcf -q sample_data/query.bcf -o sample_data/sample_data_results
./mupbwt -i sample_data/panel.bcf -s sample_data/index.ser -q sample_data/query.bcf -o sample_data/sample_data_results

Load the index and print details to stdout:

./mupbwt -l <index file> -d

An output example is:

> ./mupbwt -l sample_data/index.ser -d
built/loaded in: 0.015628 s

----
Total haplotypes: 900
Total sites: 499
----
Total runs: 27512
Average runs: 55
----
run: 0.0386925 megabytes
thr: 0.0387306 megabytes
uv: 0.0380135 megabytes
samples: 0.0833178 megabytes
rlpbwt (mapping): 0.201148 megabytes
phi panels: 0.414757 megabytes
phi support: 0.126385 megabytes
phi data structure (panels   support): 0.541142 megabytes
rlpbwt: 0.74229 megabytes
----
estimated dense size: 36.4132 megabytes
----

Input

Only bialleic case is supported. In case of vcf/bcf bcftools can be used to filter the input:

bcftools view -m2 -M2  -v snps <input vcf/vcf> > <filtered vcf/bcf file>

Output

Output file follow the standard proposed in Durbin's PBWT. Each row contain a SMEM:

MATCH   <query index>   <row index> <staring column>    <ending column> <SMEM length>

For example:

MATCH	99	150	414	430	17

Row index and query index are incrementally so the name of the sample and the precise haplotype can be calculated using the output of bcftools.

The command:

bcftools query -l <input vcf/bcf> > samples.txt

store in samples.txt all the samples name, in order. So, for example, row indices 0 and 1 corresponds to the two haplotypes of the first sample, row indices 2 and 3 to the second one etc...

Eventually you can use script/mem_sample.py:

> python mem_sample.py -h
usage: mem_sample.py [-h] [-i INPUT] [-p PANEL] [-q QUERIES] [-o OUTPUT]

options:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
                        SMEM file in Durbin's format
  -p PANEL, --panel PANEL
                        panel as VCF/BCF (optional)
  -q QUERIES, --queries QUERIES
                        queries as VCF/BCF (optional)
  -o OUTPUT, --output OUTPUT
                        output file

Esample:

python mem_sample.py -i sample_data/sample_data_results -p sample_data/panel.bcf -q sample_data/query.bcf -o sample_data/sample_data_results_new

Only one between PANEL and QUERIES can be specified. New SMEM file will contain in each row:

MATCH   <querySample_haplotype>   <panelSample_haplotype> <staring column>    <ending column> <SMEM length>

For example, assuming both PANEL and QUERIES:

MATCH	1318026_1	4919834_0	414	430	17

Results

Results on high-coverage whole genome sequencing data from UK Biobank (chromosome 20):

Region #Samples #Sites Size BCF (GB) μ-PBWT (GB) Construction time (hh:mm) Construction memory peak (GB)
chr20:60061-4060065 150119 865267 1.9 0.88 06:25 2.27
chr20:4060066-8060066 150119 880899 2 0.85 06:28 2.22
chr20:8060067-12515479 150119 961591 2.1 0.77 07:04 2.05
chr20:12515480-16768988 150119 917468 2 0.73 06:47 1.97
chr20:16768989-21050967 150119 931010 2 0.71 06:53 1.92
chr20:21050968-31549151 150119 1919134 4.2 1.20 13:54 3.06
chr20:31549152-38282825 150119 1436549 2.8 0.99 10:25 2.63
chr20:38282826-43181963 150119 1056144 2.2 0.76 07:42 2.06
chr20:43181964-47619489 150119 955970 2 0.79 06:56 2.09
chr20:47619490-51789198 150119 923178 2 0.80 06:44 2.12
chr20:51789199-55789212 150119 911452 2 0.81 06:45 2.13
chr20:55789213-59874964 150119 925442 2 0.84 06:49 2.20
chr20:59874965-64334101 150119 1096089 2.4 0.93 08:00 2.42
Total 150119 13780193 29.6 11.08 - 29.15

Results on 1000 Genome Project phase 3 data including the average number of runs per site:

Chr #Samples #Sites #Runs/site Size BCF (GB) μ-PBWT (GB) Construction time (hh:mm) Construction memory peak (GB)
1 2454 6196151 11 0.78 1.44 00:19 4.59
2 2454 6786300 10 0.84 1.47 00:21 4.76
3 2454 5584397 10 0.71 1.20 00:18 4.24
4 2454 5480936 10 0.71 1.19 00:17 4.28
5 2454 5037955 9 0.63 1.08 00:16 4.22
6 2454 4800101 10 0.64 1.06 00:15 4.28
7 2454 4517734 10 0.58 1.03 00:14 4.34
8 2454 4417368 10 0.56 0.97 00:14 4.30
9 2454 3414848 11 0.43 0.81 00:11 2.54
10 2454 3823786 10 0.50 0.87 00:12 2.77
11 2454 3877543 10 0.49 0.84 00:12 2.71
12 2454 3698099 10 0.47 0.82 00:12 2.63
13 2454 2727881 10 0.35 0.60 00:9 2.14
14 2454 2539149 11 0.32 0.58 00:8 2.18
15 2454 2320474 12 0.29 0.57 00:7 2.30
16 2454 2596072 12 0.32 0.63 00:8 2.28
17 2454 2227080 12 0.28 0.55 00:7 2.32
18 2454 2171378 11 0.28 0.51 00:7 2.23
19 2454 1751878 13 0.23 0.45 00:6 1.43
20 2454 1739315 11 0.22 0.41 00:5 1.30
21 2454 1054447 14 0.14 0.30 00:3 1.26
22 2454 1055454 14 0.14 0.29 00:3 1.24
Total 2454 77818346 11 9.91 17.67 - 64.34

Note that total building times are not printed due to the fact that all the computations have been done in parallel.

The pipeline for 1000 Genome Project phase 3 data is available at dlcgold/muPBWT-1KGP-workflow.

Reference

μ-PBWT results are currently available on Bioinformatics.

Bibtex:

@article{10.1093/bioinformatics/btad552,
    author = {Cozzi, Davide and Rossi, Massimiliano and Rubinacci, Simone and Gagie, Travis and Köppl, Dominik and Boucher, Christina and Bonizzoni, Paola},
    title = "{μ- PBWT: a lightweight r-indexing of the PBWT for storing and querying UK Biobank data}",
    journal = {Bioinformatics},
    volume = {39},
    number = {9},
    pages = {btad552},
    year = {2023},
    month = {09},
    abstract = "{The Positional Burrows–Wheeler Transform (PBWT) is a data structure that indexes haplotype sequences in a manner that enables finding maximal haplotype matches in h sequences containing w variation sites in O(hw) time. This represents a significant improvement over classical quadratic-time approaches. However, the original PBWT data structure does not allow for queries over Biobank panels that consist of several millions of haplotypes, if an index of the haplotypes must be kept entirely in memory.In this article, we leverage the notion of r-index proposed for the BWT to present a memory-efficient method for constructing and storing the run-length encoded PBWT, and computing set maximal matches (SMEMs) queries in haplotype sequences. We implement our method, which we refer to as μ-PBWT, and evaluate it on datasets of 1000 Genome Project and UK Biobank data. Our experiments demonstrate that the μ-PBWT reduces the memory usage up to a factor of 20\\% compared to the best current PBWT-based indexing. In particular, μ-PBWT produces an index that stores high-coverage whole genome sequencing data of chromosome 20 in about a third of the space of its BCF file. μ-PBWT is an adaptation of techniques for the run-length compressed BWT for the PBWT (RLPBWT) and it is based on keeping in memory only a succinct representation of the RLPBWT that still allows the efficient computation of set maximal matches (SMEMs) over the original panel.Our implementation is open source and available at https://github.com/dlcgold/muPBWT. The binary is available at https://bioconda.github.io/recipes/mupbwt/README.html.}",
    issn = {1367-4811},
    doi = {10.1093/bioinformatics/btad552},
    url = {https://doi.org/10.1093/bioinformatics/btad552},
    eprint = {https://academic.oup.com/bioinformatics/article-pdf/39/9/btad552/51556136/btad552.pdf},
}