Image compression Snake Oil
Posted by SkyHook on 20th June 2005 | Permanent Link
Hello,
Over the past several months. Our communications group has been searching for a better compression scheme then JPEG2000 provides.
We are looking for the best possible image file compression with the least amount of image data loss.
This codec will be used with HF Radio mail and data transfer applications such as SCAMP, WINLINK 2000 and DRM™ - Digital Radio Mondiale to transmit missing person and other image data via HF radio.
To our disappointment we have not been able to find anything better then JPEG2000 other then three companies that appear no more then snake oil salesmen.
The reasons I make such a comment are:
This sounds like snake oil to me unless.
a) You are starting with a small picture in the first place. 64*64 pixels for example would give a picture to match the main features of the person in question, but I sure would not want to use it for access control.
b) What is store is a ‘fingerprint’ or hash-value of the features of the photo. In that case you could not really recreate the photo from the compressed data. Rather you compare the hash-value of the original master
photo to any later photos supplied to see how close a match you can get
> CodecID implements a truly unique technology =
> by improving by more than a factor three the compression obtained with =
> JPEG 2000, which is the state of the art technology for photo =
> compression. It is the result of a research program in mathematical =
> image processing led by internationally renowned scientists, major =
> contributors to the JPEG-2000 standard.=20
Second snake oil moment. ‘internationally renowned scientists’ but no names that you can call and check up on. Still sounds like snake oil to me.
First, before you pay a cent of your money to any of these people, have them show you it working on YOUR equipment first. And have YOUR techs check the equipment before, during, and after the demo for tampering.
It will also help if the pictures are supplied by you, and they never get a copy of the pictures until the very last minute before them start the demo.
If there are any delays and you lose track of your data disk/stick insist on restarting the demo with a new set of pictures.
If there are more delay, start smelling a rat. At the very least ask why are you paying good money for software that is so hard to use.
We are still interested in finding something better then JPEG2000. But at this time JPEG2000 appears to be he best.
Thankyou
June 20th, 2005 at 9:04 pm
Another technology which is NOT snake-oil
Allume (http://www.allume.com) talked about a JPEG recompression algorithm (claiming to reduce the size of JPEG photos and images up to 30% with absolutely no loss in image quality) some time ago which was also touted as snake oil.
But they have released it bundled with their StuffIt line of products thus drowning all doubts about it being another fake. More information at http://www.stuffit.com
You might want to try it out, a free trial is available for download.
June 21st, 2005 at 1:21 am
I find that the JPEG compression in Stuffit v9 typically achieves 20-23% on JPEGs.
I wish it could be released in some form as a browser plugin or personal proxy so that it could be used on websites. A 20-23% saving in JPEG size would probably take 10% off my company’s website traffic, but generally improve performance given the bottleneck nature of our Internet connection.
June 21st, 2005 at 3:16 am
Something like that might be on its way.
For more details on their plans, you might want to check http://www.stuffit.com/imagecompression/wp_stuffit_imgcomp.pdf
June 21st, 2005 at 8:27 am
We have investigated Allume JPEG recompression algorithm a few months ago.
But still find the JPEG 2000 codec much better for the following reasons.
JPEG2000 was designed to overcome the limitations of the original JPEG standard and provide high-quality images at low bit-rates. In addition, JPEG2000 includes new features and functionalities for client/server imaging applications and “resource constrained wireless devices.”
Notice Jpeg 2000 also gives you “25-35% smaller file size”
*************Same as Allume.*****************
JPEG2000: Advantages Over JPEG
Better image quality at the same file size
“25-35%” smaller file sizes at comparable image quality
Good image quality even at very high compression ratios, over 80:1
Low complexity option for devices with limited resources
Scalable image files — no decompression needed for reformatting. With JPEG 2000, the image that best matches the target device can be extracted from a single compressed file on a server. Options include:
Image sizes from thumbnail to full size
Grayscale to full 3 channel color
Low quality image to lossless (identical to original image)
Progressive rendering and transmission through a layered image file structure. Example: from a single 100 Kbyte image file of a 512×512 original image, a low resolution 32×32 pixel thumbnail image can be transmitted by sending only 10Kbytes. Sending an additional 15 KBytes increases the resolution to 64×64 pixels, and so on. Other layerings provide for progressive transmission and rendering based on quality, color component and spatial location in the image.
These features seem perfect for transmitting images via HF radio and Digital Radio Mondiale.
———————
We currently are transmitting images of 10-20 KBytes at 320 x 240 pixels. We find these images do the job very nicely but we would like to reduce the size down to 5-10 KBytes, if this is even possible in order to speed thing up.
Many SDK’S are also available.
As to krisjohn’s comment. There are already free Browser Plugins for the Jpeg 2000 format.
Morgan JPEG2000 Plug-in for IE (free)
http://www.morgan-multimedia.com/JPEG2000/
Freeware of “JP2view_Engine” now!
http://www.optimidata.com/english/jpeg2000sw/jp2_shell_extension.html
Demo of jpeg2000
http://www.luratech.com/products/lurawave/jp2/index.jsp?OnlineShopId=1764551118474501932
More about Jpeg2000 can be found here.
http://datacompression.info/JPEG2000.shtml
Now if wee can only find something to give us 30 to 50 percent smaller files sizes then JPEG2000.
But I guess there are physical limits to the amount of compression.
Thankyou.
June 21st, 2005 at 5:54 pm
The problem with JPEG2000 is to get that smaller file I have to go back to the original image and re-encode it. The advantage of Allume’s JPEG packing technique is that it shrinks an existing file without reducing quality or requiring the original.
Also, and this isn’t JPEG2000’s fault, having to use the embed tag is ugly. What I need is a system where the browser automatically negotiates receipt of a compressed version of a JPEG, like browsers already negotiated a compressed version of HTML files. Something where I can still use the img tag.
June 21st, 2005 at 8:04 pm
We start out with the original image.
Our procedure, is to scan the picture of a missing person add some text and reduce the file to 320 x 240 pixels then save it as a 15K JPEG2000 file ready to transmit.
What we noticed with the stuffit X compression is that we could not get the compression to reduce the file size much lower then 30K. Which is going in the wrong direction.
We currently use the Free program http://irfanview.com/ and the provided free Jpeg2000 plugin in the plugin downaload package.
Yes, I think this would be very nice to have a system where the browser automatically negotiates receipt of a compressed version of a JPEG, like browsers already negotiated a compressed version of HTML files. Something where I can still use the img tag.
June 21st, 2005 at 8:24 pm
I think it would be a very good idea if both of us dropped a message off to Allume Systems requesting display plugins for IE, Netscape and Firefox. Also a plugin that would work with IRFanView would be nice and a free windows shell interface that would display thumbnails.
This image did not come out to bad.
http://home.att.net/~ronchap/nataleehollowayclr.sitx
Then again people need to install the StuffIt Expander™ to view since creating a self extracting file becomes very large again.
June 22nd, 2005 at 5:19 pm
They had a “SIF”, Stuffit Image Format, file format detailed in their whitepaper, with mention of browser plugins. I’ve already asked about it in this thread.
http://support.allume.com/techsupport/forum/viewtopic.php?t=347
June 22nd, 2005 at 5:33 pm
I ran that holloway example through a few filters. Certainly can’t make a smaller JPEG with jpegoptim or jpegtran, but I did notice that Allume’s fancy JPEG repacking didn’t save much more than WinRK was able to squeeze out of the original .JPG (13,669 vs 13,973 from 14,247). It’d be great if Allume included a compression plug-in system for their repacking process, but in the mean time are there any programs that can “dehuff” a JPEG so it can be packed using any packer, then “rehuff” back into a usable JPEG? I’d very much like to test things like 7-Zip, Rar and WinRK on “decompressed JPEGs”.
September 28th, 2005 at 12:27 pm
SkyHook… were you implying that Even Technologies, Let it Wave and IBO ARE Snake Oil Salesmen or are the exception and ARE NOT Snake Oil Salesmen??
September 29th, 2005 at 12:01 am
I believe SkyHook meant that they ARE acting like snake oil salesmen. You can search though previous comp.compression posts to read what experts (or atleast some of those who REALLY understand the domain and these technologies) have said about them.
As an example, if you try to spend some time reading the WhitePapers at MatrixView website, which are SUPPOSED to describe their technology, you will notice that they have not even bothered to mention the set of images which were used for benchmarking.
As if that was not enough, the graphs which SUPPOSEDELY describe the benchmarking results don’t have any units mentioned on either x or y axis. It is clear that in all likelyhood no tests were ever run, graphs were probably made by marketing team with ABO bar stretched (or shortened if needed) at whim. The term “visually lossless” seems to be just another way of saying “lossy”.
AFAIK, Allume’s is the only technology which has been publically succesfully demostrated so far (in StuffIt).
Others on the other hand, even if they have anything substantial, have failed to demonstrate their technologies till now and all the material they have provided till now only makes them look more suspicious (but then maybe *I* have missed something).
If you have more time, read this:
http://www.matrixview.com/en/archive/articles/downloads/mv%20technology/MatrixView%20White%20Paper%20-%20Honey%20I%20Shrunk%20the%20Bits!.pdf
September 30th, 2005 at 3:40 pm
Skyhook, if you are interested in better image compression than JPEG2000 technology, please do take a look at our proprietary wavelet image technology at http://www.msoftware.co.nz. If you get in touch with us we can provide you with a demonstration tool that will allow you to test it on your own data.
Our technology provides higher quality than JPEG2000 for any given bitrate, and has an extremely flexible and fast decoder.
October 9th, 2005 at 7:22 pm
I received an Email today from our friends at Even Technologies Inc.
Not much to the mail, other then to announce a new Demo Webpage. Still no response to our questions about cost of SDK and browser plugins for the PSI_X™ Still Image Compression engine.
But after 9 months of waiting for a response it does appear we are on their mailing list.
Check out the Demo’s
http://www.eventechnologiesinc.com/demos.html
IMOP: The video looks great audio sounds great. But there seems to be a slight audo video delay problem.
I’m not ready to remove Even Technologies quite yet from the Image compression Snake Oil salesman list. But, when Q2 2006 comes around and they release the software and hardware. I sure will.
This is the link to their home page.
http://www.eventechnologiesinc.com/index02.html
October 17th, 2005 at 10:33 am
I found your page by chance and would like to give some comments on the information in the original post and in the comment #11 by the moderator concerning our company, Let It Wave. Please forward this message to SkyHook.
Let me begin by some facts.
“Internationally renowned scientists” mean among others our CEO, Mr Stephane Mallat. This name should be familiar to compression experts such as those who visit this page. In case it is not, type “wavelet history” in Google. You may also wish to try out the following links:
Google champion for “wavelet history”
Wavelet history in Matlab documentation
Wavelet article in Wikipedia
From technical point of view, we do NOT have generic image compression method. As it is clearly mentioned on our website, this algorithm is specially designed for identity photos.
We work with different resolutions, and those given at the website (150×200 pixels) are more than sufficient for person identification.
It is no feature-like but a true compression algorithm and it restores perfect photos if it has enough bytes.
And it is “three times better” than JPEG-2000, meaning that for a recognizable ID photo JPEG-2000 would need 1500 bytes againt only 500 bytes with our solution.
Finally, we do demonstrate our technology to those who really need it. These are professional system integrators providing badging or secure ID card solutions. There are few chances it goes to large public simply because public is not interested by gaining 1000 bytes.
Anyway, we should be pretty good at telling stories to convince European Commission to award the Grand Prize 2005 for innovation to this product:
Eurocommission press-release
It is probably a good thing to be a bit suspicious but I would suggest checking your posts more thoroughly before calling other people liars.
Thank you.
October 17th, 2005 at 11:22 am
Yours is not one of those mathematically impossible claims we guys very frequently see. And there is no denying the fact that you might have what you say. But without a public demo, public at large will find claims hard to believe.
I understand that you dont feel very good about what has been posted here about LetItWave. Anything exciting usually generates noise, probably it is the lack of available details. Thanks for providing all this information, it will help clear some doubts.
I have dropped a mail to SkyHook (I don’t know for sure if the mail ID I have from him is still valid).
October 17th, 2005 at 5:32 pm
LetItWave,
Thankyou for your comments.
Over many months before posting the original message. Our communications groups have sent out many letters and made phone calls to all 3 companies; Even Technologies, Let it Wave and
ABO Image compression
These letters explained how and why we were interested in this technology. We are not the general public.
Sorry to say we did not receive any response to our emails until recently (Even Technologies) and the phone lines seemed to be always off hook.
From June to August we made every attempt to contact all 3.
Our posts, where our very last attempt to obtain some sort of response.
Sorry to say the responses have come a little too late for our current software release but will be looking forward to what Even Technologies will be offering Q1 2006.
LetItWave, We wish you good luck with your ID photo product.
Thankyou
October 18th, 2005 at 5:39 am
I just wanted to post some information which would clearly explain what we do because it is presented in a not fully objective way here.
Skyhook: Although it looks like we do not have the product you are looking for (generic compression), I think that I return all email quieries we get. Neither I have any phone line problems… That is why I was surprised when I saw your post.
December 3rd, 2005 at 4:32 pm
Just a thought.
I would love to see something worked out with WB to demo all 3 Codecs using the Trailer from the new “Superman Returns movie”
Even Technologies
Let it Wave
ABO Image compression
This was the response I got back from Even Technologies.
Not a bad idea, but I doubt it is public domain AND we would need a very high quality original to capture the best our codec can give.
Regards,
Patrick
Just thinking that by adding nice clean copy of the Movie trailer of ‘The return of Superman” to your DEMO page would be a great way to spead the word about your new codec.
http://www.imdb.com/title/tt0348150/trailers
December 11th, 2005 at 5:19 am
I thought these people were about still images, not videos. (Yes Even tech. is in motion too but I doubt about others).
This test probably just doesnt applies to all.
December 11th, 2005 at 1:41 pm
Let it Wave also offers video solutions.
But Even Technologies seems to be the one to watch.
Again, I think it would be neet to see something like.
This movie Trailer brought to the net By Even Technologies.
April 5th, 2006 at 5:27 am
Good
September 6th, 2006 at 10:24 pm
[...] We have talked about this many times earlier, and others have too: MatrixView may need to work hard to establish its bona fides. It has already been tarred with the same brush as several ‘breakthrough’ video compression ‘technologies’ which did not live up to their claims. The most notorious of these, to Australians at least, was Adams Platform, but there have been plenty of others. According to one, albeit anonymous and undated web page, the MatrixView technology is little more than existing compression algorithms cobbled together. [...]
June 16th, 2007 at 11:00 pm
In response to LetItWave, I have personally sent 6 emails on different dates requesting information and have never received one response from your company.
I even went so far as to translate one of my requests into French in hopes this would solicit some kind of response…I’m still waiting.
If you would like to contact me my address, phone number, name, etc. are located in the plethora of mailings I’ve submitted.
However, if you cannot locate them here it is again:
brad [dot] gose [at] dlaser [dot] com
I do sincerely hope you will respond as I believe your product (LH-101M) is a perfect solution for us, if in fact it can perform as stated in your white papers.
Thank you,
Brad
June 18th, 2007 at 8:52 am
Well, I stand corrected. Let it wave did mail me back yesterday.
I think the only reason someone responded to me was because I managed to find a direct address to a person at Let It Wave. Their general mailbox apparently doesn’t get viewed very often, if at all.
August 2nd, 2007 at 11:48 am
Try ezcodesample.com
The source code is published, the advantage over JPEG2000 on shown sample of images is about 11%. However, it is not guaranteed for all images and to establish a record was not an objective. It just happened. The published method is improvement by separate compression of low order bits. When added to any method it can either improve or make no difference, so I can guarantee beating JPEG2000 statistically by adding this processing technique. However, I don’t think this single method makes critical difference in the industry and keep working on it, watch my site periodically.
August 12th, 2007 at 10:08 am
Try Matt Mahoney’s open-source PAQ8M WIN32 command-line compressor.
PAQ8M compresses JPEG files to close to that achieved by the commercial StuffIt 11 compressor using the proprietary SITX format.
I use PAQ8M to compress a folder of JPEGs into a PAQ8M to send by email.
I also include the tiny 38KB PAQ8M.exe file as well so the receipient can dragg & drop the PAQ8M file onto executable to easily decompress the JPEGs.
Much smaller sizes than 7-Zip’s .ZIP Deflate32 or .7z LZMA formats.
Try it at http://cs.fit.edu/~mmahoney/compression/paq8m.zip
Learn more from http://cs.fit.edu/~mmahoney/compression/
August 19th, 2007 at 10:10 pm
Question? Why am I being quoted at the start of this thread without any citation?
see:
http://groups.google.ca/group/comp.compression/msg/9ea51509a1832f5f?dmode=source
August 23rd, 2007 at 10:57 am
Some common sense remarks for those who participated in discussion.
1. What to expect from data compression firm? When we watch light athletic championship how far is the champion, say in jumping from the next competitor? Isn’t that a few percent? Then how we expect one company to be much better than the other?
2. When investor is smart enough to ask a demo before to make an investment that is not going to happen, because when company has demo the investors are no longer needed. So the investor either invest without a demo or refrain from investment.
3. Even when dealing with big corporation we have to understand that data compression algorithm is designed by a single individual who is on trusted position and when someone beat corporation in data compression it is one individual against another (not the group of scientists).
4. When individual is challenging the corporation’s format he is in much better condition. He has data samples, benchmarks generic overview of algorithms and approaches used by corporation and can continue trying until succeeding in better compression. In case of success the result is published in case of failure the attempt even has not been manifested.