Comments for The Data Compression News Blog

Comment on Asymmetric Binary System by Andrew Polar

Andrew Polar — Thu, 08 May 2008 04:13:04 +0000

I achieved computational stability for small alphabets in my version of ANS coder that I call Logical Entropy Encoder or Enlogen. It is faster than best arithmetic encoders. I compared it to my best version of arithmetic encoder RanCode that has same speed as FastAC. RanCode made round trip (encode/decode) with 8,000,000 small integers for 1.1 sec. and Enlogen needed only 0.8 sec. The alphabet in both cases was 25 symbols. The advantage is smaller than anticipated but it it noticeable at no doubt.
In the next upgrade I want to make Enlogen to be able to process large alphabets effectively.

Comment on Free ITU Standards, Yepiee by surya

surya — Thu, 24 Apr 2008 15:11:01 +0000

want to know more about itu standards.

Comment on Asymmetric Binary System by Andrew Polar

Andrew Polar — Tue, 22 Apr 2008 05:26:32 +0000

Regarding usage of different table values in ABS coder for encryption. Now it is clear that it is totally wrong way. It will take 15 minutes to break this code. You can look at my explanation of multi-alphabet ABS (ezcodesample.com) The tables may be fully expressed as sequence of bits. The tail of that sequence is key to decoding message starting from end. Cryptanalyst can implement brute force attack on tail only and continue it as soon as something making sense appear and then continue with other part, so message can be deciphered by parts.
There is also other part that I wanted to mention. Now Jarek’s concept is generalized to the case of multiple alphabets. The encoder that I described is pure logical it uses only logical operations for both encoding and decoding and it is so simple that can be explained over 15 minutes to someone who intuitively understands probability concept.

Comment on Another Failed Compression Promise: NearZero by Herbs

Herbs — Mon, 21 Apr 2008 02:23:23 +0000

More on compression and HDTV:
First, good candidates for compression are somewhat obvious like a file containing English words or an image using 32 bit color. In the word example, each repeating word can be stored or prestored in a table and then just referenced by it’s index into that table. A 32 bit color image will never use all 4 billion or so colors so color compression is very possible as well as other techniques to compress such as looking at neighboring pixels. Note that in both examples, we are compressing based on previously seen or known patterns.
Random data however is a very different beast. There are definately repeating patterns in random data but the encoding of them takes just as much if not more space to encode so it is a losing battle. Your only hope is the the original input file is not really random and has some bias in it (say an excessive amount of bitstrings 00110101 for example). Then you might get some minor compression.
Ok now HDTV. Mark Nelson stated over the air HDTV is better than Sat and cable HDTV because of different amounts of compression. I would like to know more about this. For example, what is a typical bitrate being broadcast by a good 1280×720p HD broadcast station? I have a card in my computer that can capture HDTV signals but it compresses it to about 13 Mbps (MPEG2 Transport Stream format). Playing that back doesn’t look as good as the live version. I can see a softer picture and some artifacting (color gradient blocking). Any accurate information on this topic would be much appreciated. Thanks.

Comment on Another Failed Compression Promise: NearZero by Mark Nelson

Mark Nelson — Sun, 20 Apr 2008 17:11:33 +0000

@Herb:

I bet you’re watching HDTV over cable or satellite. It looks a lot better over the air. But cable and satellite companies view bandwidth as an expense that hurts their bottom line, so they compress as much as they can get away with.

Comment on Another Failed Compression Promise: NearZero by Herbs

Herbs — Sun, 20 Apr 2008 16:10:09 +0000

One other thing: It may be frustrating to realize that if you take a large random file to compress (say 1MB for example), about half of the bits should be 0s and the other bits (also about half) being 1s. If you just compress the file saying they are all 0s or all 1s you are actually about 1/2 correct and get excellent compression. For example, if the bits represented pixels with 0 being a black pixel and a 1 being a white pixel, if you displayed these random black and white pixels on say a computer screen with adjacent bits being displayed as adjacent pixels, you would see a grey screen when you stepped back sufficiently far. Unfortunately, describing where the bits change is where the overhead becomes very significant. Sure there will be several long runs of the same bit which can be compressed but on average, you wont see any compression overall. One somewhat amazing thing is that a good compression program can take a human readable file such as this text I am posting here, and turn it into an almost random sequence of bits. In other words, it takes something very organized and easily readable by a human into something almost random and very unreadable to a human. From that “random” compressed file the original file can be “magically” regenerated. If you think about this it is quite impressive. Another point is that the almost random bits created by the compression program are not random because they hold specific meaning in the order they are in. If you were to change even 1 critical bit in the compressed file you may totally mess up the decoder. I have written coders and decoders and I have seen this. It has to be exact or the big mushroom cloud goes up fast.

Comment on Another Failed Compression Promise: NearZero by Herbs

Herbs — Sun, 20 Apr 2008 15:26:39 +0000

My understanding of compression is that it works on redundant data only. That is, if there is some type of predictability (pattern) to it, it can be compressed further. If you were to flip a coin and translate the results into 0s and 1s and then feed that into a compression program it would NOT be able to compress it (on average). This is because the compression program has no way to predict what the next bit will be based on any previous bits. I am speaking generally now about reasonably large files not some small file of under 10KB for example. I would say that small random files CAN be compressed because of their small size and limited randomness (in some cases) but generally, the savings is not really worth the effort. Compression is kinda like charging a battery. It is very easy (and fast) to get the bulk of the compression/charging to happen but to get the last 25% or so takes a lot more effort and time. One could argue that instead of working on an algorithm to compress random data, they should investigate methods to “massage” existing data so that it is more compressable. There are different ways to store the same data and they are compressable in different amounts. One simple way to compress is just to reorganize your data so the repeating parts are factored out (normalized). It is not really that compression is compressing the data, it is just that the data in it’s original form is redundant and wasteful. Compressing as we know it is just getting rid of some (but not all) of that waste. Since today’s computers are much faster and we have much faster internet links, networks, hard drives… getting the last ounce of compression is not really worth the effort. Going in the opposite direction is very efficient. That is, if you very quickly just do a mild compression on the data, you can get significant space savings with minimal CPU load. That allows compression and decompression on the fly. There are several backup programs that offer both mild and agressive compression options and the mild compression is much faster so this helps prove my point. I have written software compressors and decompressors and noticed that even a simple lookback encoder can do quite well of today’s fast computers. You dont even have to mess with any bits you can compress at the byte level and still get very good lossless compression but people have already done this. I think researchers should concentrate more effort on beefing up the technology (HDTV for example) so that it is possible to transmit many more frames per seconds and send higher quality images in almost real time. I have seen HDTVs up close and I still see artifacting similar to that of JPG (which I am also not a big fan of). Standard JPG to me is an embarassment because of the artifacting which could easily be corrected especially for still images. Moving images are a different beast. People say HDTV is superior to SDTV but try a freeze frame sometime on HDTV and might think otherwise. HDTV is only HD if the images are not moving quickly but if they are, I wouldn’t call it HD. I would never invest in a compression algorithm for random data because common sense says if such a beast existed, it could be run over and over compressing an original file to virtually nothing. Simple rule: truely random data of non trivial length CANNOT be compressed on average. There are no significant patterns to compress and the # of bit arrangements are so astronomical that encoding them is totally impractical and wont save any bits.

Comment on Another Failed Compression Promise: NearZero by Dodgy

Dodgy — Fri, 18 Apr 2008 05:15:32 +0000

So the first of the trials have been and gone…..silence still reigns out….no answers from anyone. I can confirm that the SFO are still investigating him and they expect their report to be complete within another month, but it would be nice to hear from either the team supporting Phil, or the Liquidators….seems silence is the answer….perhaps a wee trip to Nelson for a visit may be in order….

Comment on My Very Bad Euclid Discoveries Experience by Apo

Apo — Tue, 15 Apr 2008 09:30:23 +0000

I’m not investor, but as any normal person I would like to see some working results for sure.
At least this “talking head” of news speaker could be nice demo (-: (ED mention this as an example)

Such king of projects could waste a lot of money (even if it’s possible to realize somehow). I think such projects could succeed like Open-Source. With several hundreds enthusiasts without interests on money but focused on the goal.

By the way - is there any company with revolutionary promises ever show some results to the public?

Apo

Comment on Asymmetric Binary System by Andrew Polar

Andrew Polar — Tue, 15 Apr 2008 02:18:19 +0000

I added one more section in my description of ABS algorithm that shows related concept of entropy encoder implemented without formulas introduced by Jarek:
http://www.ezcodesample.com/abs/abs_article.html
It can be generalized into larger than binary alphabet, however, it is still limited to small alphabets due to computer resources. I believe that my explanation is generic when Jarek concept with formulas (no matter which ones) is particular case.

Comment on My Very Bad Euclid Discoveries Experience by Mark Nelson

Mark Nelson — Tue, 15 Apr 2008 00:46:13 +0000

@Apo:

I think the primary reason for skepticism centers around the business practices of Euclid Discoveries.

Obviously there are possibilities in object-based compression - in fact, the recent clarinet story shows just what one can do with a very accurate model - but much of ED’s behavior is reminiscent of behavior seen in other companies that have turned out to be duds, or worse yet, frauds.

This doesn’t mean Euclid Discoveries is a fraud, or is doome to failure, but given their claims, I feel the bar is set rather high for them to achieve any sort of legitimacy, and they really haven’t done *anything* to accomplish that.

Let me put it this way. If you had invested $100K of your retirement money as an ED Angel five or six years ago, and had been hearing hints that untold riches were just around the corner - but still had nothing to show for it, would the tone of your post be so reasonable and accomodating?

- Mark

Comment on My Very Bad Euclid Discoveries Experience by Apo

Apo — Mon, 14 Apr 2008 22:40:48 +0000

I don’t know anything in business field, but very interested in data compression.

First of all - why most of you are so pessimistic about new revolutionary technology?
I understand, if we are talking about random data compression or something like that, it makes me smile.
But (as I understand) we are talking about lossy compression technique, what makes possible great achievements.

Just imagine…

Let’s use some technique to store 3D objects, for example you can follow the link for video, to understand what I mean: http://movie.diginfo.tv/2007/07/11/07-0076-d.php

OK, now we have a bunch of objects. We can make it move and do any other morph operations.

Now it looks like cartoons and not realistic - that’s right. To fix this we could use relatively small amount of data to add some realistic effects (texture, some shadows e.t.c,), but even this effects could be calculated I guess.

Anyway, the bottleneck I see is that it’s work for some kind of AI, to encode/decode all this stuff…

Sounds philosophical, but still believable (-:
And I’m pretty sure there is no any significant reasons not to believe.

Regards,

Apo

Comment on Asymmetric Binary System by Andrew Polar

Andrew Polar — Tue, 08 Apr 2008 04:39:03 +0000

I see that interest to entropy encoding is exhausting. I found one application for ABS. It can treat assymetry in bits distribution after Huffman encoding. I provided example at http://www.ezcodesample.com/abs/abs_article.html
In spite of all attempts to write fast range or arithmetic encoder, Huffman encoder at least twice faster the fastest range encoder. It, however, may produce assymetrically distributed bits that can be corrected by ABS coder and brought the result after Huffman to the length of entropy encoder.

Comment on Asymmetric Binary System by Andrew Polar

Andrew Polar — Sat, 05 Apr 2008 04:36:58 +0000

I implemented second more advanced version of ABS coder that can be found at http://www.ezcodesample.com/abs/abs_article.html
It manages growing size of the state variable differently . It is easy to understand and customize. Now the things with binary ABS coder more or less clear. I’m still wondering if there is a way to handle multiple alphabet case without binary partitioning of array?

Comment on Asymmetric Binary System by Andrew Polar

Andrew Polar — Mon, 31 Mar 2008 22:07:24 +0000

In my previous text

NEXT = 10011001
SYMBOL = 3
PREV = 011001(11)
and BITLEN = 2

The table did not pass through clearly.