Comments on: Bijective BWT

By: David Scott

David Scott — Mon, 14 Jun 2010 14:48:42 +0000

There are many things BWTS could be used for. I also have a bijective distance coder. If I can get off my lazy ass and actually code the two together it would make a nice bijective compressor especially for short files where it would be a good compressor to use before an encryption pass. I have not written much here lately I thought this thread was dead.

By: Jarek

Jarek — Mon, 14 Jun 2010 07:54:58 +0000

Thanks - it was only a comment to the book - there is a lot about BWT there, but I haven’t seen this cheap generalization of this transformation: looking useful preparation for the main compression …
About symbol grouping - my thought was that language evolution works with phonetics rather - so maybe it might be more effective to block groups of letters corresponding to phonemes …
Generally for a supercompressors I would think about some dynamically evolving DMC-like structure for local correlations, ’supervised’ by synchronously evolving NN searching for larger scale relations … but I agree that it’s huge work - I’m focusing on physics now and I’m generally too theoretical for such .. marathons :)
Have fun!

By: Maloeran

Maloeran — Mon, 14 Jun 2010 03:34:26 +0000

Jarek, reodering characters to group them by typical context has been done, for text it does usually provide a small gain over plain BWT.

I don’t think making groups of symbols would be a wise thing to do given the nature of BWT. With its string ordering, it exploits very well the prediction of a byte given all previous ones.

If you want to look at something fancier and a *lot* more computationally expensive than BWT, have a look at PPM or PAQ algorithms.

Have fun!

By: Jarek Duda

Jarek Duda — Sun, 13 Jun 2010 17:11:39 +0000

Hi all,
I’m just reading Matt Mahoney’s book and started appreciating BWT - my comment is that it’s much more general transformation than just using standard (..ABCDE..) order among chars as I’ve seen in the book.
If we are fighting for a few bytes like in BBWT, while focusing on text in a given language, it could help a bit to use some better BWT, optimized for this language:
- DIFFERENT SYMBOL ORDER, like accordingly to phonetics: for example separating vowels and consonants (like ..AEBCD..) - thanks of it lexicographic order could group contexts a bit better and so the BWT could be less ‘ragged’. It’s cheap - just bijectively remap characters before and after BWT,
- DIFFERENT ALPHABET: maybe grouping a few symbols into a new one (like ‘AE’), or splitting them (like vowel nr. 1 instead of ‘A’) could also make it a bit more efficient…

It would require a lot of experiments, but using BWT optimized for a given purpose could cheaply improve ratio a few percents(?)
Hope it help,
Jarek

By: Vivien Greene

Vivien Greene — Wed, 10 Mar 2010 22:40:26 +0000

Blog looks great and reads even better! You share some great opinions and insight here. Always looking for motivating blogs to keep mine going!

By: David Scott

David Scott — Sun, 28 Feb 2010 16:23:42 +0000

Yes it looks great and for simple strings when you
have an index equal to zero your UNBWT is the same
as UNBWTS so since not using an index for inverse on the bottom part you could always just do the UNBWTS since most of the time UNBWT does not exist. But When it does
its the same as UNBWTS
Where you see the difference is if you BWT a string and
get an index of nonzero from your code.
When you UNBWTS the resultant string you get something else. That however when BWTed goes to a string with an index that is not zero.
go ahead do the BWTS and UNBWTS
David

By: Kragen Javier Sitaker

Kragen Javier Sitaker — Sun, 28 Feb 2010 13:37:21 +0000

Hey, for people who aren’t that familiar with the BWT, if you want to play with an interactive BWT (the standard version, not Scott’s bijective version), I wrote one a while back in JS at http://canonical.org/~kragen/sw/bwt.html. You can fiddle with changing the BWTed string or saved index and watch the poor algorithm struggle to invert it.

I wonder if I’ll get around to implementing the Scottified version. It looks like fun!

By: David Scott

David Scott — Tue, 22 Dec 2009 16:14:12 +0000

Of course its impossible to tell why it was rejected. I am definitely not part of the old boy network. But when I see that really good papers and people being punished for not bowing to the gods of global warming don’t get published its almost an honor not to get published. I don’t know how good the paper is but I feel confident that the really good innovative papers never make the light of day first time up in todays political strangle hold on science. Fact its getting colder the last few years yet the so called scientists that are peer reviewed are doing there best to hid those facts.
Buy the way the german guy mentions my name several times so in way I think I do get credit for the bijective BWT even though most idiots think the regular BWT is bijective.

Sorry alan I don’t get it what are you ranting about.

By: Maloeran

Maloeran — Tue, 22 Dec 2009 11:51:05 +0000

David,

That’s ridiculous, the paper was rejected on the basis that you can just add an EOF symbol? This is so utterly stupid…

If you were really to first to implement a index-free bijective BWT, I sincerely hope you manage to get credits for it.

By: alan

alan — Tue, 22 Dec 2009 02:37:55 +0000

Clue everyone:

what is “sound”?

By: alan

alan — Tue, 22 Dec 2009 02:35:00 +0000

Re: Franc Jarnovic,

do you get it ? (”honour thy error” - Brian Eno; musician)

By: David Scott

David Scott — Mon, 21 Dec 2009 17:42:08 +0000

It figures just when I said I have not heard from Yossi I get an email. So maybe we will do more next year.

By: David Scott

David Scott — Mon, 21 Dec 2009 15:55:31 +0000

The paper was rejected. As some one commented you can do a BWT using a special EOF symbol so that an index is not necessary.
I don’t think some of the reviews had a clue what it what it really meant.
Since most blocks of data regardless if you allow a free EOF or an index still don’t have a reverse BWT.
The concept was not well understood by the reviewers.
During the time we started writing this another guy
who asked to join us started writing us own that also
had bijective ST transfomrs of various orders. His paper
was accepted but maybe those running the conferences
in the Czech Republic have a better understanging of
just what the concept was anyway here is his paper

http://arxiv.org/abs/0908.0239

Have a nice day
dave

P.S. as for Gil I have not heard from him since
Oct of this year. But I emailed him earlier this
month and no reply.

By: Maloeran

Maloeran — Sun, 20 Dec 2009 23:32:29 +0000

Well done David!

Last time I played with BWT, I just thought that there had be a way to reconstruct the original sequence without that index… Of course, it wasn’t that simple, but you solved it very nicely!

I believe I’ll be reading the paper and trying to squeeze a few extra bytes out of my compressor, thanks David.

By: David A. Scott

David A. Scott — Wed, 08 Jul 2009 04:56:34 +0000

Here is a draft of the paper on the bijective form of BWT

http://bijective.dogma.net/00yyy.pdf