SBE fix gives different result with shntool & flac front-end?

[fatoldpig]

here's the shntool info on original files:
length expanded size cdr WAVE problems filename
8:51.48 93781908 -b- -- ---xx Track01.flac
14:12.74 150467836 -b- -- ---xx Track02.flac
11:51.70 125585712 -b- -- ---xx Track03.flac
17:21.18 183674548 -b- -- ---xx Track04.flac
52:17.61 553510004 B (totals for 4 files, 0.5096 overall compression ratio)

if i use shntool fix -o flac *.flac, i get:
Track01-fixed.flac:f1775ff933b949f297f0ff836d82bee2
Track02-fixed.flac:2a5f56b3c375b8d61f881f2d88ce808d
Track03-fixed.flac:2442d9a69bde6b6911eb1174547d83b7
Track04-fixed.flac:fce2288e1d15fd84d706d5b53defb01c

if i use flac > wav > flac (flac front-end with align on SBE), I get:
Track01.flac:f1775ff933b949f297f0ff836d82bee2
Track02.flac:2a5f56b3c375b8d61f881f2d88ce808d
Track03.flac:2442d9a69bde6b6911eb1174547d83b7
Track04.flac:a7e3d48aeefa4d2c5e5a4721063d2b5b

flac front-end log
options: --sector-align -P 4096 -b 4608 -m -l 8 -e -q 0 -r 0,6 -V
Track01.wav: Verify OK, wrote 50181676 bytes, ratio=0.535
Track01.wav: INFO: sector alignment causing 142 samples to be carried over
Track02.wav: Verify OK, wrote 78855773 bytes, ratio=0.524
Track02.wav: INFO: sector alignment causing 378 samples to be carried over
Track03.wav: Verify OK, wrote 61637536 bytes, ratio=0.491
Track03.wav: INFO: sector alignment causing 535 samples to be carried over
Track04.wav: Verify OK, wrote 91908702 bytes, ratio=0.500
Track04.wav: INFO: sector alignment causing 111 zero samples to be appended


notice, last track's ffp doesn't match. any idea why?

[TheMamba]

I'll let the true Techno experts give you a definitive answer but...the fact that it is on the last track only seems to mean that they pad/don't pad the last file differently. Being that it is the last file, it should be the last track you listen to and won't notice a SBE...

[4candles]

I'll let the true Techno experts give you a definitive answer but...the fact that it is on the last track only seems to mean that they pad/don't pad the last file differently. Being that it is the last file, it should be the last track you listen to and won't notice a SBE...

That was my first guess, but at least in the version of shntool I have, the default action is to pad the last track. From the output posted, it seems that FLAC does the same thing.

Can you post the output of "shntool info" on those two versions of track 4?

[fatoldpig]

Can you post the output of "shntool info" on those two versions of track 4?
shntool version:
length expanded size cdr WAVE problems filename
8:51.48 93781340 --- -- ---xx Track01-fixed.flac
14:12.74 150466892 --- -- ---xx Track02-fixed.flac
11:51.70 125585084 --- -- ---xx Track03-fixed.flac
17:21.19 183677132 --- -- ---xx Track04-fixed.flac
52:17.61 553510448 B (totals for 4 files, 0.5111 overall compression ratio)
Track01-fixed.flac:f1775ff933b949f297f0ff836d82bee2
Track02-fixed.flac:2a5f56b3c375b8d61f881f2d88ce808d
Track03-fixed.flac:2442d9a69bde6b6911eb1174547d83b7
Track04-fixed.flac:fce2288e1d15fd84d706d5b53defb01c


flac front-end version:
length expanded size cdr WAVE problems filename
8:51.48 93781340 --- -- ---xx Track01.flac
14:12.74 150466892 --- -- ---xx Track02.flac
11:51.70 125585084 --- -- ---xx Track03.flac
17:21.19 183677132 --- -- ---xx Track04.flac
52:17.61 553510448 B (totals for 4 files, 0.5105 overall compression ratio)
Track01.flac:f1775ff933b949f297f0ff836d82bee2
Track02.flac:2a5f56b3c375b8d61f881f2d88ce808d
Track03.flac:2442d9a69bde6b6911eb1174547d83b7
Track04.flac:a7e3d48aeefa4d2c5e5a4721063d2b5b

shntool info seems identical

[Ted]

I don't know much about fixing tracks and the padding and such yet, but I do know that a single "bit" of data will make a completely different hash. How does the track sound? Is the FA and SA much different between the two? - or don't those matter in this case?

[AAR.oner]

SA/FA wouldn't matter in this...this isn't my area of expertise, but i'd try running it thru TLH's sbe fix and compare with the other two you've already run...

[TheMamba]

SA/FA wouldn't matter in this...this isn't my area of expertise, but i'd try running it thru TLH's sbe fix and compare with the other two you've already run...


That's a pretty good idea too. Where are Five and Raindawg? ;)

[fatoldpig]

is there any way to know how many sample it's shifting/carrying over or padding during the shntool fix? that might give a clue what shntool is doing compare to flac front-end.

[4candles]

I undertook an experiment and created a set of 4 CD-quality WAV files that weren't cut on sector boundaries.

I then created a set of fixed WAV files using "shntool fix", and a set of fixed FLAC files used "flac --sector-align". I then compared the output of "shntool md5" and get different checksums for the final file. "shntool len" gives the same output for both sets of four files.

Using a hex editor I compared the two resulting WAV versions of track 04. The shntool-fixed version contains 448 zero bytes (112 samples). The FLAC-fixed version contains about 224 zero bytes, but then about 224 bytes containing data at the very end. So this seems to be a bug in FLAC.

The above tests were done using FLAC 1.1.1. I don't have the latest version installed (1.1.2) to test with, but the "changelog" for v1.1.2 doesn't mention any bugs regarding the --sector-align option.

Is anyone able to do a similar test using FLAC 1.1.2?

[fatoldpig]

I undertook an experiment and created a set of 4 CD-quality WAV files that weren't cut on sector boundaries.

I then created a set of fixed WAV files using "shntool fix", and a set of fixed FLAC files used "flac --sector-align". I then compared the output of "shntool md5" and get different checksums for the final file. "shntool len" gives the same output for both sets of four files.

Using a hex editor I compared the two resulting WAV versions of track 04. The shntool-fixed version contains 448 zero bytes (112 samples). The FLAC-fixed version contains about 224 zero bytes, but then about 224 bytes containing data at the very end. So this seems to be a bug in FLAC.

The above tests were done using FLAC 1.1.1. I don't have the latest version installed (1.1.2) to test with, but the "changelog" for v1.1.2 doesn't mention any bugs regarding the --sector-align option.

Is anyone able to do a similar test using FLAC 1.1.2?good find 4candles. i'll do some test with flac 1.1.2 when i get home.

[uhclem]

is there any way to know how many sample it's shifting/carrying over or padding during the shntool fix? that might give a clue what shntool is doing compare to flac front-end.
Yes. Shntool prints it to the screen while processing the files.

[4candles]

good find 4candles. i'll do some test with flac 1.1.2 when i get home.

There's no need - I can confirm the bug is there in 1.1.2 as well.

I've tracked down the bug in the source code and I'll let the developer know. The buffer used to pad the end of the data with zeros is not being entirely cleared - only the first half of the buffer is being cleared.

So until flac 1.1.3 (assuming the bug is fixed) is released I would strongly advise against using the --sector-align option in FLAC. In all cases, the second half the "zero padding" at the end of the last track is not guaranteed to contain zeros.

[Five]

I was fixing a similar set recently where it just needed padding on the last track. I was thinking that I could add the padding manually in CEP, so I decided to try it with FLAC frontend first and really zoom in and look at what it does. The last few samples aren't zero samples, they have values like 12, which is very very very close to zero. I remember something about how cd players don't like it when the final sample is a zero, it has to be a little different. So I'm thinking that maybe both FLAC frontend and SHNtool pad the last track with "almost zero" samples at the tail end, perhaps these samples are slightly different. I've also taken a track which was padded by FLAC frontend and mixpased the inverted original over it and came up with a perfect flat line, so the difference should be in the padding by my theory. I can't recall if I did this, but another test to do would be to pad the track, then open it with a wav editor and only remove the padding (remember to turn smoothing off!), resave then generate a SHNtool md5 and see if there is any variance.

Hopefully we can get to the bottom of this when I find a couple hours to test all these variations. I can't get to it tonight, if somebody wants to beat me to it I won't be offended.

[4candles]

I was fixing a similar set recently where it just needed padding on the last track. I was thinking that I could add the padding manually in CEP, so I decided to try it with FLAC frontend first and really zoom in and look at what it does. The last few samples aren't zero samples, they have values like 12, which is very very very close to zero. I remember something about how cd players don't like it when the final sample is a zero, it has to be a little different. So I'm thinking that maybe both FLAC frontend and SHNtool pad the last track with "almost zero" samples at the tail end, perhaps these samples are slightly different. I've also taken a track which was padded by FLAC frontend and mixpased the inverted original over it and came up with a perfect flat line, so the difference should be in the padding by my theory. I can't recall if I did this, but another test to do would be to pad the track, then open it with a wav editor and only remove the padding (remember to turn smoothing off!), resave then generate a SHNtool md5 and see if there is any variance.

Hopefully we can get to the bottom of this when I find a couple hours to test all these variations. I can't get to it tonight, if somebody wants to beat me to it I won't be offended.

I don't think it's a case of FLAC doing something clever - it's clear from looking at the source code that it's a bug.

If you use a hex editor to look at the WAV file "fixed" by shntool, you will see that it contains the correct number of zero-valued bytes at the end.

If you look at the WAV version of the last track "fixed" by FLAC, then you will see that the first half of the padding is zero, but the second half isn't. This is easier to see if the last track you are fixing doesn't already have silence at the end.

[uhclem]

I remember something about how cd players don't like it when the final sample is a zero, it has to be a little different. So I'm thinking that maybe both FLAC frontend and SHNtool pad the last track with "almost zero" samples at the tail end, perhaps these samples are slightly different.
I've never heard this before. I'm skeptical. Even if it were true I seriously doubt that FLAC is randomly putting in a few non-zero samples to deal with it. There is no mention of this in the FLAC documentation. I think 4candles indentification of a bug in the source code sounds like the best explanation. Good catch, 4candle & pig. We need to put something about this in the FAQ and in the etree wiki.

[4candles]

I've never heard this before. I'm skeptical. Even if it were true I seriously doubt that FLAC is randomly putting in a few non-zero samples to deal with it. There is no mention of this in the FLAC documentation. I think 4candles indentification of a bug in the source code sounds like the best explanation. Good catch, 4candle & pig. We need to put something about this in the FAQ and in the etree wiki.

I think this bug (assuming I'm right) has been in flac for a long time, so there's probably no urgent rush to document it without confirming the bug with the flac developers.

I've emailed the FLAC developers mailing list about it and I'll report back here when I get a reply.

[fatoldpig]

thanks for looking into this. hopefully it's a bug with flac front-end and they'll fix it soon.

[Five]

I've never heard this before. I'm skeptical. Even if it were true I seriously doubt that FLAC is randomly putting in a few non-zero samples to deal with it. There is no mention of this in the FLAC documentation. I think 4candles indentification of a bug in the source code sounds like the best explanation. Good catch, 4candle & pig. We need to put something about this in the FAQ and in the etree wiki.
after searching for a bit I can't find any reference to this anywhere online & I can't remember where I heard it. So it looks to me like this is a minor bug with FLAC frontend like everybody's been saying... sorry guys for putting forward this questionable information without doing my research first!

[4candles]

thanks for looking into this. hopefully it's a bug with flac front-end and they'll fix it soon.

It's nothing to do with flac front-end, it's a bug in the "flac" program itself (i.e. flac.exe for Windows users).

I've had no reply yet, but my bug report can be found here:

http://lists.xiph.org/pipermail/flac-dev/2005-July/date.html

[skyofcrack]

In the meantime, just to be safe, should we untick the box in FLAC Frontend for 'align on sector boundaries,' if I'm understanding this conversation correctly? :confused:

soc

[feralicious]

How come noone has mentioned that simply going from flac > wav > flac (align on sector boundaries) isn't the proper way to fix SBEs? I distincly remember being schooled on that in a thread a while back. Doesn't that cause it to not shift the data from track to track but rather pad each track individually thereby adding silence between tracks? It should go flac > wav > fix SBEs > flac, no?

Or am I completely mad?

[4candles]

How come noone has mentioned that simply going from flac > wav > flac (align on sector boundaries) isn't the proper way to fix SBEs? I distincly remember being schooled on that in a thread a while back. Doesn't that cause it to not shift the data from track to track but rather pad each track individually thereby adding silence between tracks? It should go flac > wav > fix SBEs > flac, no?

Or am I completely mad?

Taking the simple example of three files, there are two ways to convert them to flac. Firstly, by running flac.exe three times:

flac -8 file1.wav
flac -8 file2.wav
flac -8 file3.wav

or the alternative is to just run flac.exe once:

flac -8 file1.wav file2.wav file3.wav

The "--sector-align" option only makes sense when you are using the second method. Using the first method (running flac separately for each file) will do as you describe, and simply pad each track with silence. The second method will move partial sectors from the end of one track to the start of the following track - exactly the same as shntool does (by default).

I believe that early versions of FLAC Front-end used the earlier method, and hence the sector-align facility in FLAC Front-end was useless, but I believe that recent versions of FLAC Front-end do the right thing.

And to answer the previous question, yes, I wouldn't use the sector-align option in FLAC any more - at least until a bug-fixed release is made.

I still haven't heard anything back from my bug report, but it's only been a little more than 24 hours.

[skyofcrack]

I've always used shntool to fix my sbe's and I'll definitely turn off that function until I hear back about the bug.

Thanks for the heads up. :)

soc

[Beleaguered]

It's nothing to do with flac front-end, it's a bug in the "flac" program itself (i.e. flac.exe for Windows users).

I've had no reply yet, but my bug report can be found here:

http://lists.xiph.org/pipermail/flac-dev/2005-July/date.html
You might also want to document your bug on the bug tracker:
http://sourceforge.net/tracker/?group_id=13478&atid=113478

[4candles]

You might also want to document your bug on the bug tracker:
http://sourceforge.net/tracker/?group_id=13478&atid=113478

You're right - that's the correct way to report a FLAC bug. I've added it here:

http://sourceforge.net/tracker/index.php?func=detail&aid=1237707&group_id=13478&atid=113478

Thanks.

[fatoldpig]

for now i'll use, shntool fix -o flac *.flac to fix sbe.

[ssamadhi97]

cute, flac accidentally pads with samples from "earlier"

http://img341.imageshack.us/img341/6483/gosh8ff.th.png (http://img341.imageshack.us/my.php?image=gosh8ff.png)

padded data (196 samples) is identical with the highlighted selection

[Ted]

I pretty sure that's what 4candles found out when he earlier posted about it not clearing the buffer before padding. I think he discovered that it cleared only the first half of the buffer.

[Five]

and it looks like the bug was missed because the 2nd half of the buffer will often be zero samples anyways.

[4candles]

cute, flac accidentally pads with samples from "earlier"

http://img341.imageshack.us/img341/6483/gosh8ff.th.png (http://img341.imageshack.us/my.php?image=gosh8ff.png)

padded data (196 samples) is identical with the highlighted selection

Nice picture - it summarises the bug nicely, which as you say is that flac only half-clears the buffer, leaving data from earlier in the input file in the second half.

I'm surprised by the lack of response to my bug report - maybe Josh is on vacation...

[ssamadhi97]

indeed.. more precisely it clears "half as much of the buffer as it should" because it incorrectly assumes that input_ bps = bps of the input file (16) while the actual bps of the input_ buffer is 32

so in my case it cleared the first 392 / 2 = 196 samples of the 2048 sample input buffer instead of 392 samples - note that (bps >> 3) is the same as (bps / 8),

samples 196-391 (and onwards to sample number 2047) of the input buffer remain from the previously read block of samples.

[4candles]

I've just heard from Josh Coalson (the FLAC developer) that he's now fixed this bug in the CVS version of FLAC.

I don't know anywhere that offers pre-compiled binaries of the FLAC CVS, but it also means that next official release will be OK.

Dave.

[fatoldpig]

I've just heard from Josh Coalson (the FLAC developer) that he's now fixed this bug in the CVS version of FLAC.

I don't know anywhere that offers pre-compiled binaries of the FLAC CVS, but it also means that next official release will be OK.

Dave.thanks for the update

[fatoldpig]

anymore update on this, is the new version out yet?