The Data Compression News Blog

All about the most recent compression techniques, algorithms, patents, products, tools and events.

Subscribe

Posts: RSS Feed
Comments: RSS Feed

Recent Posts

  • Google Snaps Up On2 (8 Comments)

  • Bijective BWT (34 Comments)

    David Scott has written a bijective BWT transform, which brings all the advantages of bijectiveness to BWT based compressors. Among other things, making BWT more suitable for compression-before-encryption and also give (slightly) better compression.

  • Asymmetric Binary System (170 Comments)

    Jarek Duda’s “Asymmetric Binary System” promises to be an alternate to arithmetic coding, having all the advantages, but being much simpler. Matt has coded a PAQ based compressor using ABS for back-end encoding. Update: Andrew Polar has written an alternate implementation of ABS.

  • Precomp: More Compression for your Compressed Files (6 Comments)

    So many of today’s files are already compressed (using old, outdated algorithms) that newer algorithms don’t even get a chance to touch them. Christian Schneider’s Precomp comes to rescue by undoing the harm.

  • On2 Technologies is Hiring (4 Comments)

    There aren’t too many companies working on cutting edge codecs, and of those few this one is hiring. Best of luck.

PIGZ: Parallel GZIP

Posted by Sachin Garg on 13th April 2007 | Permanent Link

Mark Adler, maintainer of popular zlib library has released a multicore capable Parallel Implementation of GZip, nicknamed PIGZ. Version 1.5 implements nearly all of gzip’s functionality, including decompression of .gz and .Z (Unix compress) files.

You can get the latest code, it needs latest version of zlib (1.2.3) to build.

16 Responses to “PIGZ: Parallel GZIP”

  1. Mark Adler Says:

    That link won’t work. The latest distributed version is 1.6, and the link is http://zlib.net/pigz16.c.gz.

    It will also optionally compress to the zlib format.


  2. Sachin Garg Says:

    Mark has released another update, version 1.7 decompresses the first entry of a zip file (if deflated), and compresses to a single-entry zip file if requested with –zip.

    http://zlib.net/pigz17.c.gz


  3. Kristiyan Georgiev Says:

    http://zlib.net/pigz16.c.gz is broken, can you point me to a different location?

    Cheers,
    kris


  4. Sachin Garg Says:

    Just use the link to version 1.7, its in the comment above yours.


  5. Andrew Polar Says:

    I wish to use this opportunity and ask Mark Adler a question. In 1990 Martucci published article regarding usage of simple predictor in combination with entropy encoder for compression of images. Following this article it is possible to write image compression routine of 300 lines that makes photo images to be compressed as 2.5 to 1. Three years later PNG format was created that required 20 times longer code and compressed photo images typically as 1.7 to 1. PNG format happened to be incorporated into major browsers. 16 years after mentioned publication Microsoft introduced HD Photo that compress images to the same ratio as in Marucci method published in 1990. And few years ago JPEG2000 anounced format that is about 7% better than Martucci suggested in 1990. The question is
    Wouldn’t that be nice to incorporate new format into all browsers that require 300 lines of coding, so programmers will be able to generate images on the fly in web applications without using so called new approach of Microsoft that is called Silverlight.
    Thanks,
    Andrew Polar


  6. girlfriend Says:

    According to my personal benchmarks, pigz doesn’t speed up compression, unlike pbzip. In fact, in some cases it’s performance may be slower than single-threaded version!


  7. Earl Colby Pottinger Says:

    Girlfriend: What is your hardware/software setup?


  8. Mark Adler Says:

    Version 2.0 of the parallel implementation of gzip called “pigz” is
    available here:

    http://zlib.net/pigz/

    pigz exploits multiple processors and multiple cores for compression.
    The use of threads has been completely rewritten to be more efficient
    and clean. It compiles on Unixish operating systems that provide the
    zlib and pthread libraries.

    If you have success or failure in compiling and using pigz, I would
    appreciate knowing about either way. Thanks.


  9. Eugene Says:

    Nice article. Thanks. :) Eugene


  10. Laura Says:

    Does this run on Windows systems? Is the compression faster than something like gzip?


  11. Sachin Garg Says:

    You will need to compile it yourself (its just code but it should work on windows).

    It will work 2 times faster by using both cores if your CPU is dual core. 4x faster if you have a quad core, but no speed up if you have a single-core CPU.


  12. Mike W Says:

    You cant extract the resulting .tar file from http://zlib.net/pigz/ on an AIX server.

    # tar -xvf pigz-2.1.6.tar.tar
    tar: 0511-169 A directory checksum error on media; 0 not equal to 72695.
    [root@nzaxs22ch-cl] : /opt/pigz

    Are there limitations as to what O/S can use this?


  13. Mike W Says:

    Ignore my last post.

    The download randomly changed the filename to “tar -xvf pigz-2.1.6.tar.tar” when it should have been “tar -xvf pigz-2.1.6.tar.gz”

    …. go figure


  14. rmartinez Says:

    Just wondering how I can compile pigz “statically” in fedora and then copy it to a Redhat machine - because RH still uses zlib version 1.2 but pigz requires 1.3.


  15. Manoj Says:

    Hi all,

    I’ve compiled pigz 2.1.6 on AIX 5.3 but it won’t decompress gz files larger than 2GB. any idea how to enable large file support when compiling pigz on AIX?? I’ve not install gnu c compiler in it.. using the stock c compiler from aix.

    Thanks.


  16. chloe sims jwoww tanning lotion one and done Says:

    Lucky for you, I’ve discovered two places online where you
    can get some sweet deals on lotion and other awesome supplies.
    To test for allergies, dab a wee bit of it on your arm.
    Many works done for all the workers of the world who are all on the same
    team.

    Here is my web blog: chloe sims jwoww tanning lotion one and done