ZPack File Format, spec and tools
Posted by Sachin Garg on 10th May 2006 | Permanent Link
Today at comp.compression, Brendan G Bohannon announced a vaguely PAQ like file format supporting compression and read/write operation (as zipfiles are not that well suited to read/write). The format will assume no fragmentation of smaller files, rather a file is to be moved if it expands beyond the space available to it.
Current default file extension will be “zpk” and will use deflate with a 64kB window. The CRC algo will be the same algo used in ZIP and PNG for example.
You can checkout the code and format specs.
Another related comp.compression thread.
May 26th, 2007 at 1:03 am
i have one file .zpk and i kent open hellp me plese !
May 26th, 2007 at 1:53 am
Where did ou get this file from? How did you create it?
Can you tell what are the first few bytes of the file? it might or might not be a zpack file.
March 7th, 2009 at 11:14 am
How to use ZPack?
March 7th, 2009 at 11:18 am
It says “cannot find cygreadline6.dll “
March 7th, 2009 at 9:20 pm
It looks like it needs cygwin to run. Cygwin is a free linux emulator for windows, after installation you can find this DLL in cygwin/bin directory.
March 8th, 2009 at 3:35 am
What is the compression ratio and time for any of known test samples?
March 8th, 2009 at 3:52 am
What is compression ratio and compression time for any known data sample?
March 8th, 2009 at 11:05 am
Not to be confused with ZPAQ.
http://cs.fit.edu/~mmahoney/compression/#zpaq
The spec is not final yet, but I anticipate little or no change from v0.07. But more testing is needed first…
March 9th, 2009 at 5:08 am
I tested zpaq v0.07 on the set of 24 kodak images and compared it to my RunCoder1 (the link is http://www.ezcodesample.com/pre_ralpha.html). The compression ratio is very close and the speed is higher for RunCoder1. The compressed file size for zpaq
764858, 663633, 566903, 733193,
892010, 702116, 649576, 913864,
621880, 684079, 688136, 596514,
907422, 806378, 695578, 582305,
678350, except, 735866, 541908,
702254, 846802, 768315, 767838
The compressed size of RunCoder1
754589, 613762, 653145, 761417,
901241, 742443, 689280, 891806,
676201, 734790, 752603, 670401,
895200, 821011, 739335, 657648,
697579, 900958, 756522, 565596,
745434, 879776, 862447, 840038
The image kodim18.bmp failed on zpaq. Exception was thrown. The image itself is fine. Please take a look. If you need the image, let me know I’ll send it to you over e-mail.
Obviously this test is not exhaustive, just a particular set. The tool looks good, a little-bit slow but still within normal time and exception throwing must be addressed.
March 18th, 2009 at 6:24 pm
Thanks, but I couldn’t reproduce the error with zpaq 0.07 with cmin.cfg or cmax.cfg. Also, I tested kodim18.bmp with zpaq 1.00 and it worked OK.
Initial size 1179702
cmin.cfg -> 917500
cmid.cfg -> 678339
cmax.cfg -> 633571
I could probably improve speed and compression with a config file tuned for BMP files. These are for general purpose compression.
March 25th, 2009 at 9:59 pm
I downloaded zpaq007 again and tested. This time I could not reproduce bug either. May be it out of boundary array bug that can be seen randomly. This time I made some comparison. To be exact this is the link for download:
http://cs.fit.edu/~mmahoney/compression/zpaq007.zip
this is command for min and max:
zpaq cmin.cfg arch05.zpk kodim05.bmp
I tested files from kodim01 to kodim08. Original size is 1,179,702 for all. Here are results
zpaqmin time zpaqmax time
1. 837561 1.36 503016 30.78
2. 580758 1.34 378106 29.49
3. 654142 1.31 326597 29.07
4. 865588 1.38 441359 30.25
5. 1029356 1.40 639591 30.92
6. 827640 1.35 458283 30.06
7. 696798 1.34 384540 29.46
8. 1046823 1.41 664717 31.06
RunCoder1 time Bzip2
1. 754589 0.31 586257
2. 613762 0.25 453524
3. 653145 0.32 431099
4. 761417 0.33 551419
5. 901241 0.42 769452
6. 742443 0.30 546024
7. 689280 0.34 483482
8. 891806 0.44 759197
I will add comments in the next message.
March 25th, 2009 at 10:09 pm
First thing that max compression takes absurd time, such as 30 sec. per small file and will not be used. Min compression takes reasonable time such as 1.3 sec. but the size is not impressive. RunCoder1 is only 600 lines adaptive arithmetic coder that bits zpaq min compression in both time and compression ratio (and time is more than 4 times less). I also added BZIP2 for comparison. It also bits zpaq min in both time and ratio. I don’t show time but it is about one sec. per image. RunCoder1 is not file archiver. It encode data in streaming way without reading file to the end. On that reason BZIP2 is better, because it reads file to the end before compression, applies BWT and place labels within data, so it is not real time streaming tool. We have to distinguish what we are doing. Is that file compression or real time data compression. Such as speech compression during real time transmission over internet. For streaming tool RunCoder1 is better than zpaq007, for files BZIP2 is better.
March 26th, 2009 at 6:23 pm
zpaq 1.00 also has a mid.cfg for average compression and run time. min.cfg and max.cfg also compress a little better than 0.07. But again, none of these are tuned for .bmp compression. max.cfg is benchmarked at http://www.maximumcompression.com/data/summary_sf.php
Anyway I looked at runcoder1. If you make it into a compressor/decompressor I will benchmark it. Also, there are a few compiler errors with g++.
March 29th, 2009 at 10:46 pm
Matt,
I made cosmetic changes necessary for runcoder1 to be compiled by g++ and tested it in Linux. I also made changes that allows compress/decompress test run in the way you and some other people making and expecting to see. You can benchmark it.
I did not make it as pure streaming tool like reading from disk and writing to disk, but I will make it in the nearest future.
My position regarding new archivers is simple. Beat bzip2 in both time and compression and your encoder is good to become archiver. My runcoder1 got close but did not beat bzip2 in both, your coders did not beat bzip2 in both compression and time either, so we both can keep working on it, am I right?
March 30th, 2009 at 2:12 pm
Matt,
I replaced operations over memory buffers into files input/output. Now it is ready for benchmarking.
Thanks.
April 1st, 2009 at 6:50 pm
I added runcoder1 to my benchmarks.
http://cs.fit.edu/~mmahoney/compression/text.html#4581
http://cs.fit.edu/~mmahoney/compression/uiq/
BTW I did try ZPACK (the original topic from 3 years ago) but it is apparently broken and no longer supported. I let it run overnight compressing enwik8 but after 9 hours the archive was only 4096 bytes and still running. LZ77 ought to be faster than that.
April 28th, 2009 at 7:34 pm
I just ran across this, and as the original author I will say a few things:
ZPack is not ZPAQ, in that, it did not offer any new/novel compression technology (it is based on deflate, and so, if working, would give comparable results to good old ZIP).
it is also not to be confused with another ZPack, which was a different app with a different fileformat, but an unfortunate set of name clashes…
secondly, it is not an individual file compressor/decompressor (such as gzip or bzip2), rather it was a tool for packaging and unpackaging archives in a special format.
thirdly, this format and tool was replaced (in my own use), with ‘ZPack2′, which is essentially a library for giving random read/write access to plain-old ZIP files.
as for end-user use, none of this is likely to be so useful, but rather it would be “potentially useful”, to programmers, as a library…