linux-imx

============================

XZ data compression in Linux

============================

​

Introduction

============

​

XZ is a general purpose data compression format with high compression

ratio and relatively fast decompression. The primary compression

algorithm (filter) is LZMA2. Additional filters can be used to improve

compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters

improve compression ratio of executable data.

​

The XZ decompressor in Linux is called XZ Embedded. It supports

the LZMA2 filter and optionally also BCJ filters. CRC32 is supported

for integrity checking. The home page of XZ Embedded is at

<http://tukaani.org/xz/embedded.html>, where you can find the

latest version and also information about using the code outside

the Linux kernel.

​

For userspace, XZ Utils provide a zlib-like compression library

and a gzip-like command line tool. XZ Utils can be downloaded from

<http://tukaani.org/xz/>.

​

XZ related components in the kernel

===================================

​

The xz_dec module provides XZ decompressor with single-call (buffer

to buffer) and multi-call (stateful) APIs. The usage of the xz_dec

module is documented in include/linux/xz.h.

​

The xz_dec_test module is for testing xz_dec. xz_dec_test is not

useful unless you are hacking the XZ decompressor. xz_dec_test

allocates a char device major dynamically to which one can write

.xz files from userspace. The decompressed output is thrown away.

Keep an eye on dmesg to see diagnostics printed by xz_dec_test.

See the xz_dec_test source code for the details.

​

For decompressing the kernel image, initramfs, and initrd, there

is a wrapper function in lib/decompress_unxz.c. Its API is the

same as in other decompress_*.c files, which is defined in

include/linux/decompress/generic.h.

​

scripts/xz_wrap.sh is a wrapper for the xz command line tool found

from XZ Utils. The wrapper sets compression options to values suitable

for compressing the kernel image.

​

For kernel makefiles, two commands are provided for use with

$(call if_needed). The kernel image should be compressed with

$(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2

dictionary. It will also append a four-byte trailer containing the

uncompressed size of the file, which is needed by the boot code.

Other things should be compressed with $(call if_needed,xzmisc)

which will use no BCJ filter and 1 MiB LZMA2 dictionary.

​

Notes on compression options

============================

​

Since the XZ Embedded supports only streams with no integrity check or

CRC32, make sure that you don't use some other integrity check type

when encoding files that are supposed to be decoded by the kernel. With

liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32

when encoding. With the xz command line tool, use --check=none or