inflate

Pure Dawn readers for raw DEFLATE, gzip and ZIP, plus CRC-32.

use inflate/gzip
use inflate/zip

let data = gzip.gunzip(src)?                       # at most 64 MiB out
let big = gzip.gunzip(src, cap: Some(268435456))?  # a larger ceiling
let files = zip.entries(archive)?                  # name, is_dir, data

Output ceiling

Every entry point takes cap: Option[Int], a ceiling on the bytes it may write. It defaults to deflate.DEFAULT_CAP, 64 MiB. Pass cap: Some(n) for another ceiling and cap: None for none. A refusal by the default says it was the default and ends with pass cap: Some(n) for a larger limit, or cap: None for no limit. gzip.gunzip applies its cap to all members together and zip.entries to all entries together: a later member or entry gets only what the earlier ones left.

Gzip

gzip.gunzip reads the complete RFC 1952 stream. Concatenated members are accepted and their payloads concatenated in order; each member's CRC-32 and ISIZE are checked, FHCRC is verified when present, reserved flag bits are rejected, and bytes after a trailer must begin another complete member, so trailing garbage is an error rather than ignored.

Raw DEFLATE

deflate.inflate(src) decodes a stream that starts at byte zero. deflate.inflate_from(src, from: k) starts at offset k and also answers the absolute offset just past the stream, which is what a container needs to find its trailer. Its cap counts only the output of that call.

Changes between versions: CHANGELOG.md.

API

inflate/crc32

crc32.sum

fn sum(b: Bytes) -> Int

The CRC-32 of b, as the unsigned 32-bit value the formats store.

Bit at a time, eight steps per byte, with no table. The usual 256-entry table would have to be a List here, and a List read is a trie lookup plus reference counting: on the native backend that one read per byte cost more than the eight shifts and masks it replaces. 0 - (c & 1) is all ones when the low bit is set and zero when it is not, so the conditional XOR needs no branch.

inflate/deflate

deflate.DEFAULT_CAP

const DEFAULT_CAP: Int

The ceiling an entry point applies when its caller passes no cap: 64 MiB of output. cap: None is no ceiling at all, and cap: Some(n) any other one.

deflate.inflate

fn inflate(src: Bytes, cap: Option[Int] = Some(DEFAULT_CAP)) -> Result[Bytes, String]

Decompress a raw DEFLATE stream that begins at byte zero. cap is a ceiling on the uncompressed size, DEFAULT_CAP unless given; None is no ceiling.

deflate.inflate_from

fn inflate_from(src: Bytes, from: Int = 0, cap: Option[Int] = Some(DEFAULT_CAP)) -> Result[(Bytes, Int), String]

Decompress a raw DEFLATE stream beginning at from, and return the output together with the absolute byte index immediately after the stream.

The end offset is what a container checks the member against: zip knows how long the compressed member is supposed to be, and without this it has to take the decompressor's word that whatever followed the final block was not part of the member. Gzip needs the cursor form because slicing off the remaining suffix for every concatenated member would copy that suffix every time and make many small members quadratic in the archive size.

cap limits the output of this call alone, DEFAULT_CAP unless given.

inflate/gzip

gzip.gunzip

fn gunzip(src: Bytes, cap: Option[Int] = Some(deflate.DEFAULT_CAP)) -> Result[Bytes, String]

Decompress every gzip member and concatenate their uncompressed payloads in member order. Bytes after the final trailer are another member or damage; they are never ignored.

cap is one ceiling over the aggregate uncompressed output, deflate.DEFAULT_CAP unless given; None is no ceiling. Each member receives only the remaining budget, so concatenation cannot reset the limit.

inflate/zip

Entry

type Entry = { name: String, is_dir: Bool, data: Bytes }

One entry, fully decompressed. Directories carry no data.

Meta

type Meta = { name: String, is_dir: Bool, flags: Int, method: Int, crc: Int, csize: Int, size: Int, local: Int }

One central directory record, before any data is read. size is what the directory claims the entry expands to -- a claim, not a measurement, and the only one available to a caller deciding whether to unpack at all.

zip.headers

fn headers(src: Bytes) -> Result[List[Meta], String]

Every entry's metadata, in central directory order, without unpacking anything. This is what a caller with a budget needs first: how many entries there are and how big each says it will be, decided before the first byte of the first one is materialised.

zip.read

fn read(src: Bytes, m: Meta, cap: Option[Int] = Some(deflate.DEFAULT_CAP)) -> Result[Bytes, String]

One entry's bytes, with a ceiling on how big they may get: deflate.DEFAULT_CAP unless given, and None is no ceiling. A directory entry is empty.

The ceiling matters even though m.size is right there: that number is the archive's claim about itself, checked afterwards, and a bomb is a small compressed member with a large true expansion and whatever size its writer felt like putting in the directory.

zip.entries

fn entries(src: Bytes, cap: Option[Int] = Some(deflate.DEFAULT_CAP)) -> Result[List[Entry], String]

Every entry in the archive, fully unpacked, in central directory order. An archive with no entries reads as the empty list: that is a legal zip, and whether it is an acceptable one is the caller's question.

cap is one ceiling over all entries' output together, deflate.DEFAULT_CAP unless given; None is no ceiling. Each entry is read against what the earlier ones left, so entries that share compressed bytes are each paid for.