213 lines
7.4 KiB
Text
213 lines
7.4 KiB
Text
============================================================================
|
|
LZO Frequently Asked Questions
|
|
============================================================================
|
|
|
|
|
|
I hate reading docs - just tell me how to add compression to my program
|
|
=======================================================================
|
|
|
|
This is for the impatient: take a look at examples/simple.c and
|
|
examples/lzopack.c and see how easy this is.
|
|
|
|
But you will come back to read the documentation later, won't you ?
|
|
|
|
|
|
Can you explain the naming conventions of the algorithms ?
|
|
==========================================================
|
|
|
|
Let's take a look at LZO1X:
|
|
|
|
The algorithm name is LZO1X.
|
|
The algorithm category is LZO1.
|
|
Various compression levels are implemented.
|
|
|
|
LZO1X-999
|
|
!---------- algorithm category
|
|
!--------- algorithm type
|
|
!!!----- compression level (1-9, 99, 999)
|
|
|
|
LZO1X-1(11)
|
|
!---------- algorithm category
|
|
!--------- algorithm type
|
|
!------- compression level (1-9, 99, 999)
|
|
!!---- memory level (memory requirements for compression)
|
|
|
|
All compression/memory levels generate the same compressed data format,
|
|
so e.g. the LZO1X decompressor handles all LZO1X-* compression levels
|
|
(for more information about the decompressors see below).
|
|
|
|
Category LZO1 algorithms: compressed data format is strictly byte aligned
|
|
Category LZO2 algorithms: uses bit-shifting, slower decompression
|
|
|
|
|
|
Why are there so many algorithms ?
|
|
==================================
|
|
|
|
Because of historical reasons - I want to support unlimited
|
|
backward compatibility.
|
|
|
|
Don't get misled by the size of the library - using one algorithm
|
|
increases the size of your application by only a few kB.
|
|
|
|
If you just want to add a little bit of data compression to your
|
|
application you may be looking for miniLZO.
|
|
See minilzo/README.LZO for more information.
|
|
|
|
|
|
Which algorithm should I use ?
|
|
==============================
|
|
|
|
LZO1X seems to be best choice in many cases, so:
|
|
- when going for speed use LZO1X-1
|
|
- when generating pre-compressed data use LZO1X-999
|
|
- if you have little memory available for compression use LZO1X-1(11)
|
|
or LZO1X-1(12)
|
|
|
|
Of course, your mileage may vary, and you are encouraged to run your
|
|
own experiments. Try LZO1Y and LZO1F next.
|
|
|
|
|
|
What's the difference between the decompressors per algorithm ?
|
|
===============================================================
|
|
|
|
Once again let's use LZO1X for explanation:
|
|
|
|
- lzo1x_decompress
|
|
The `standard' decompressor. Pretty fast - use this whenever possible.
|
|
|
|
This decompressor expects valid compressed data.
|
|
If the compressed data gets corrupted somehow (e.g. transmission
|
|
via an erroneous channel, disk errors, ...) it will probably crash
|
|
your application because absolutely no additional checks are done.
|
|
|
|
- lzo1x_decompress_safe
|
|
The `safe' decompressor. Somewhat slower.
|
|
|
|
This decompressor will catch all compressed data violations and
|
|
return an error code in this case - it will never crash.
|
|
|
|
- lzo1x_decompress_asm
|
|
Same as lzo1x_decompress - written in assembler.
|
|
|
|
- lzo1x_decompress_asm_safe
|
|
Same as lzo1x_decompress_safe - written in assembler.
|
|
|
|
- lzo1x_decompress_asm_fast
|
|
Similar to lzo1x_decompress_asm - but even faster.
|
|
|
|
For reasons of speed this decompressor can write up to 3 bytes
|
|
past the end of the decompressed (output) block.
|
|
[ technical note: because data is transferred in 32-bit units ]
|
|
|
|
Use this when you are decompressing from one memory block to
|
|
another memory block - just provide output space for 3 extra bytes.
|
|
You shouldn't use it if e.g. you are directly decompressing to video
|
|
memory (because the extra bytes will be show up on the screen).
|
|
|
|
- lzo1x_decompress_asm_fast_safe
|
|
This is the safe version of lzo1x_decompress_asm_fast.
|
|
|
|
|
|
Notes:
|
|
------
|
|
- When using a safe decompressor you must pass the number of
|
|
bytes available in `dst' via the parameter `dst_len'.
|
|
|
|
- If you want to be sure that your data is not corrupted you must
|
|
use a checksum - just using the safe decompressor is not enough,
|
|
because many data errors will not result in a compressed data violation.
|
|
|
|
- Assembler versions are only available for the i386 family yet.
|
|
Please see also asm/i386/00README.TXT
|
|
|
|
- You should test if the assembler versions are actually faster
|
|
than the C version on your machine - some compilers can do a very
|
|
good optimization job and they also can optimize the code
|
|
for a specific processor.
|
|
|
|
|
|
What is this optimization thing ?
|
|
=================================
|
|
|
|
The compressors use a heuristic approach - they sometimes code
|
|
information that doesn't improve compression ratio.
|
|
|
|
Optimization removes this superfluos information in order to
|
|
increase decompression speed.
|
|
|
|
Optimization works similar to decompression except that the
|
|
compressed data is modified as well. The length of the compressed
|
|
data block will not change - only the compressed data-bytes will
|
|
get rearranged a little bit.
|
|
|
|
Don't expect too much, though - my tests have shown that the
|
|
optimization step improves decompression speed by about 1-3%.
|
|
|
|
|
|
I need even more decompression speed...
|
|
=======================================
|
|
|
|
Many RISC processors (like MIPS) can transfer 32-bit words much
|
|
faster than bytes - this can significantly speed up decompression.
|
|
So after verifying that everything works fine you can try if activating
|
|
the LZO_ALIGNED_OK_4 macro improves LZO1X and LZO1Y decompression
|
|
performance. Change the file config.h accordingly and recompile everything.
|
|
|
|
On a i386 architecture you should evaluate the assembler versions.
|
|
|
|
|
|
How can I reduce memory requirements when (de)compressing ?
|
|
===========================================================
|
|
|
|
If you cleverly arrange your data, you can do an overlapping (in-place)
|
|
decompression which means that you can decompress to the *same*
|
|
block where the compressed data resides. This effectively removes
|
|
the space requirements for holding the compressed data block.
|
|
|
|
This technique is essential e.g. for usage in an executable packer.
|
|
|
|
You can also partly overlay the buffers when doing compression.
|
|
|
|
See examples/overlap.c for a working example.
|
|
|
|
|
|
Can you give a cookbook for using pre-compressed data ?
|
|
=======================================================
|
|
|
|
Let's assume you use LZO1X-999.
|
|
|
|
1) pre-compression step
|
|
- call lzo_init()
|
|
- call lzo1x_999_compress()
|
|
- call lzo1x_optimize()
|
|
- compute an adler32 checksum of the *compressed* data
|
|
- store the compressed data and the checksum in a file
|
|
- if you are paranoid you should verify decompression now
|
|
|
|
2) decompression step within your application
|
|
- call lzo_init()
|
|
- load your compressed data and the checksum
|
|
- optionally verify the checksum of the compressed data
|
|
(so that you can use the standard decompressor)
|
|
- decompress
|
|
|
|
See examples/precomp.c and examples/precomp2.c for a working example.
|
|
|
|
|
|
How much can my data expand during compression ?
|
|
================================================
|
|
|
|
LZO will expand incompressible data by a little amount.
|
|
I still haven't computed the exact values, but I suggest using
|
|
these formulas for a worst-case expansion calculation:
|
|
|
|
Algorithm LZO1, LZO1A, LZO1B, LZO1C, LZO1F, LZO1X, LZO1Y, LZO1Z:
|
|
----------------------------------------------------------------
|
|
output_block_size = input_block_size + (input_block_size / 16) + 64 + 3
|
|
|
|
[This is about 106% for a large block size.]
|
|
|
|
Algorithm LZO2A:
|
|
----------------
|
|
output_block_size = input_block_size + (input_block_size / 8) + 128 + 3
|
|
|