A Binary GEN file format - BGEN

This page documents version 1.1 of the BGEN format.

Change history

Version	Date	Description
v1.1	March 2012	Update designed to cope with the long alleles present at indels and structural variants in recent releases of the 1000 genomes project. Also, we use an altered scaling of probabilities to maximise stored precision. This version is backwards-compatible with v1.0, via the LongIds flag documented below.
v1.0	2009	Initial version.

Background

A GEN file typically contains millions of floating-point numbers -- often in a fixed format -- stored in the file in a textual representation. For example, for a cohort of 1500 individuals, typed at 30000 SNPs, to store the AA, AB and BB genotype probabilities takes 1500 x 30000 x 3 = 135 million floating-point numbers. Consequently, programs which manipulate this data must spend a long time parsing the numbers to produce in-memory float or double quantities. For simple programs this time can dominate the program execution time.

This page describes a binary GEN file format (the "BGEN" format) which overcomes this problem. Tests show that using this binary format can achieve a file input speed increase of 5-10x. Genotype data is stored compressed so that BGEN files typically take up no more space (and usually less space) than a corresponding gzipped GEN file.

The current specification has been updated to handle long alleles such as those present in the latest 1000 genomes release.

A C++ implementation of this file format is available as the "genfile" sublibrary of QCTOOL, available here.

Overview

A BGEN file consists of a header block, followed by a series of blocks called snp blocks. The first four bytes of the file indicate the start position of the first snp block (relative to the fifth byte of the file).

Note: All numbers in the file are stored as integers in little endian (least significant byte first) order. This choice coincides with the memory layout used on most common architectures. see the wikipedia page for more details.

Detailed specification

The first four bytes

The first four bytes of the file encode an unsigned integer indicating the offset, relative to the 5th byte of the file, of the start of the first snp block (or the end of the file if there are 0 snp blocks). For example, if this offset is 0, the snp blocks start at byte 5.

No. of bytes	Description
4	An unsigned integer offset indicating the offset, relative to the fifth byte of the file, of the first byte of the first snp block (or the end of the file if there are no snp blocks).
4	TOTAL

The header block

The header block contains global information about the file.

No. of bytes	Description
4	An unsigned integer H indicating the length, in bytes, of the header block. This must not be larger than offset.
4	An unsigned integer indicating the number of snp blocks stored in the file.
4	An unsigned integer indicating the number of samples represented in the snp blocks in the file.
4	Reserved. (Writers should write 0 here, readers should ignore these bytes.)
H-20	Free data area. This could be used to store, for example, identifying information about the file
4	A set of flags, with bits numbered as for an unsigned integer. See below for flag definitions.
20 + L	TOTAL

Header block -- flag definitions

The following flags can be contained in the flags field in the header block. Note: all bits not listed here must be set to 0.

Bit	Name	Value	Description
0	CompressedSNPBlocks	0	Indicates SNP block probability data is not compressed.
		1	Indicates SNP block probability data is compressed using zlib's compress() function.
3	LongIds	0	Indicates alleles are stored as single characters. SNP blocks are layed out according to the v1.0 spec.
		1	Indicates version 1.1 of the SNP block layout is used. This allows for multiple characters in alleles and is supported in SNPTEST from version 2.3.0, and in QCTOOL version 1.1.

SNP blocks

Following the header comes a sequence of 0 or more SNP blocks. Each SNP block consists of the following data in order. (Note: the following description is valid when LongIds=1. When LongIds=0, SNP blocks are layout out as per the v1.0 spec, described here.)

No. of bytes	Description
4	The number of individuals the row represents, hereafter denoted N.
2	The length LS of the SNP id.
LS	The SNP id.
2	The length LR of the rsid.
LR	The rsid.
2	The length LC of the chromosome
LC	The chromosome
4	The SNP position, encoded as an unsigned 32-bit integer.
4	The length LA of the A allele.
LA	The A allele.
4	The length LB of the B allele.
LB	The B allele.
P	Genotype probability data for the SNP for each of the N individuals in the cohort. If the CompressedSNPBlocks flag is not set, this field consists of P=6*N bytes representing the probabilities. If CompressedSNPBlocks is set, this field contains a 32-bit unsigned integer specifying the length of the compressed data, followed by the compressed data itself. See below for details of the storage scheme used.
21 + LS + LR + LA + LB + P	TOTAL

SNP block probability data

The probability data is listed as a sequence of 2-byte unsigned integers. These should be interpreted in triples, the first member being the probability of AA, the second the probability of AB, the third the probability of BB. Altogether these occupy 6*N bytes where N is the number of samples. When CompressedSNPBlocks is not set, these 6 * N bytes are stored directly. When CompressedSNPBlocks is set, these 6 * N bytes are first compressed using zlib, and the length of the compressed data is stored as a 4-byte integer, followed by the compressed data itself.

To convert the stored 2-byte integers into probabilities, the following calculation should be performed:

1. Convert the number into a floating-point format (e.g. float or double).
2. Divide by 32,768.

Note that the range of a two-byte unsigned integer is 0 - 65,535 inclusive. Thus the resulting probabilities can take on values between 0 and 65,535/32768 ~ 1.9999 inclusive and they are accurate to four decimal places.

Note: to convert a floating point probability to its integer representation, do the following:

1. Multiply by 32,768.
2. Check that the number is in the half-open interval [0,65535.5) and round to the nearest integer.

All numbers are stored in little-endian (least significant byte first) order.