Flac Fingerprint vs. st5 / Flac Frontend vs. TLH ???

[feralicious]

Originally Posted by range_hood

That´s Blondie July 9, 1979, isn´t it?
These files are rotting on my harddisk since I downloaded. I asked dgkdgk (the taper and seeder) to reseed them and reported the torrent to the DaD staff. No response from the seeder, a mod over there said the torrent is ok as long as no other leecher has a problem with it.

If you listen to the problematic tracks you will hear digital errors, so please don´t trade this further without noting that in the info file.

The ffp-txt file matches the "fingerprint infomation" in the flac files, but the audio content has different checksums. Maybe the seeder flaced the files without verify?
In TLH "Verify checksum file" (ffp) does only compare the textfile to the "fingerprints" stored in the flacs. Where "Test encoded files" (same as flac.exes Verify) decodes the audio, calculates the checksums and compares that to the "fingerprint information" in the flacs.
This "Test encoded files" is comparable to a st5 checksum verification.

What I do is checking if all checksums do match and "test encoded files" (all in TLH) - additional listening to questionable files a bit closer and noting all problems. And maybe persuading the seeder, as he seems to be the taper, to reseed a fixed show.

Yep that's the one. So how come when I fix the SBEs, processing all the files, the new files are fine and the flac ffp/checksum matches the audio checksum?

And so really basic ffp and verifying just those is useless then. Why even have them? Now I know to verify the files and not just run the ffp verify. Had I not happened to do it on this show I would've just checked using the ffp and thought it was fine. That's bad because I actually spent (too much) time trying to understand all this and SBEs, just think about all those others who don't even look into it.

[range_hood]

Originally Posted by feralicious

Yep that's the one. So how come when I fix the SBEs, processing all the files, the new files are fine and the flac ffp/checksum matches the audio checksum?

On re-encoding (fixing sbes) you calculated new checksums of the actual audio content.

Originally Posted by ferelicious

And so really basic ffp and verifying just those is useless then. Why even have them? Now I know to verify the files and not just run the ffp verify. Had I not happened to do it on this show I would've just checked using the ffp and thought it was fine. That's bad because I actually spent (too much) time trying to understand all this and SBEs, just think about all those others who don't even look into it.

That´s a good point. The function should be disabled in TLH for ffp in the next release. Maybe ban *.ffp and use *.st5 as standard. TLH development seems not to be that vital.
You have to have fingerprints as a textfile, so users can compare their sources without having to download the whole show.

[AAR.oner]

nope...i've not been clear... i understand what ya'll are saying...

i received a show ina trade...the only checksum included was a ffp.txt...when i went to verify it, it failed...so i do some research and find the "real" ffp online, checked it against my copy of the show, and it verified correct...so i compare the two ffp's...seems the ffp.txt i got in the trade had been messed up [i think a few of the letters/numbers got jumbled in the fingerprint text itself]...hence my orig point:

when creating a lossless fileset [pref. flac of course ] on a show for the first time, best to create an st5 checksum, and not a ffp.txt *only*...they are more prone to "user-error" than a .st5 or even a regular .ffp...

[Five]

after briefly reading this thread, here's what I can tell you...

when a FLAC file is created, the ffp for that file is stored in the header at the time the audio is compressed.

creating a ffp merely copies this checksum to a .txt file (using FLAC frontend) or a .ffp (using TLH). the .ffp file is just like the .txt file, just with a different extension.

when a .ffp (or ffp.txt) file is verified using TLH, it takes the .ffp file and compares the checksum associated with each file with the checksum stored in the header. it doesn't actually look further than that. That's why it takes such a short time to do this function.

FLAC frontend can't verify .ffp files this way.

when you use the "test" function on FLAC frontend, it decompresses the audio to a temp file, generates a checksum and compares this with the checksum which was stored in the header at the time the file was created. TLH can also do this, using the "test encoded files" function.

okay, here's the tricky part:

when you use TLH (or shntool) to verify an .st5 file, TLH decompresses the audio to a temp file, generates fresh checksums and compares these with the checksums contained in the .st5 file and doesn't check the headers of the FLAC files at all.

also, when you fix a SBE, the file is decompressed, altered, then re-encoded to FLAC and therefore a fresh checksum is generated and inserted into the header.

so the error from flac frontend "MD5 signature mismatch" should also be reported from TLH when you use the "test encoded files" function, and it means that the checksums contained in the headers of those files don't match the checksums generated on the spot from the WAV audio decompressed to a temp file.

when you made the new .ffp it took all the checksums from the headers just like the seeder did, therefore they are the same.

then you make an .st5 and it shows the true checksums for the files, some of which have incorrect checksums stored in their headers at the original time of compression.

the files you fixed were re-encoded and new headers were written which you tested as being accurate.

so, to fully verify a FLAC file, you have to do a couple things:

1) check that the checksums contained in the headers match the checksums of the audio contained within. you can do this using either TLH "test encoded..." or FLAC frontend verify/test.

2) if there is a .ffp file, check that it matches what is contained in the headers using TLH "verify checksum" or by generating a fresh fingerprint using FLAC frontend and comparing by eye.

3) if there is a .st5 file, verify it using TLH "verify checksum file". If you want to do the same using only FLAC frontend, run a test to make sure the headers match the files, generate a ffp.txt, then compare the checksums by eye with the .st5 checksums.

is that confusing enough?

remember, I'm only on coffee #1...

[Five]

and I find it very strange that the checksums in the header don't match the audio in some files... that's a problem that happened during compression, and I've never seen this before and am not able to re-create this sort of a problem. My gf always thought I was paranoid to bother testing FLACs so thoroughly, turns out its a good idea!

[AAR.oner]

i apologize for my hijacking of this thread Fera...sorry

i'ma start a new thread re: st5 vs ffp vs md5, WANT ya'lls thoughts opinions on it tho...cheers!

[range_hood]

Originally Posted by Five

and I find it very strange that the checksums in the header don't match the audio in some files... that's a problem that happened during compression, and I've never seen this before and am not able to re-create this sort of a problem.

I only can imagine this on encoding flacs while other programs are loading the cpu; without checking "Verify" in flac frontend.

[feralicious]

Well now I'm wondering about shn files. Do you need to both verify the md5 and "test" the files? I have been since I came across this problem but it feels like overkill.

Aaron... don't worry... I never thought you had hijacked it.

range_hood... I reported that torrent and they pulled it. Of course it had already run its course and hundreds are trading it at this very moment, but hey...

[feralicious]

Originally Posted by feralicious

Well now I'm wondering about shn files. Do you need to both verify the md5 and "test" the files? I have been since I came across this problem but it feels like overkill.

Still wondering if I'm wasting my time. Can I just do the "test encoded files"? I'm going to convert them to flac anyway before I archive.

[range_hood]

Originally Posted by feralicious

Still wondering if I'm wasting my time. Can I just do the "test encoded files"? I'm going to convert them to flac anyway before I archive.

On "Test encoded files" the files are decoded to a temp directory. As you convert them to flac anyway (shn>wav>flac) the test is included in the conversion (say, it´s no test anymore ).
The md5 check is a must, to determine if the files are the same as the seeders ones.

[feralicious]

What if I'm going straight from shn>flac? I use foobar.

I swear I think I have all this down then something like this comes up and confuses me again.

So... if the md5 is a must, then why do people think you don't need them for flacs? I don't see what the difference is if "testing" isn't enough for shns why would it be for flacs?

When someone says "whole file md5" that's not the same as an st5, right? That's like an md5 of all the tracks?

[range_hood]

Originally Posted by feralicious

What if I'm going straight from shn>flac? I use foobar.

If no error shows up on reencoding, then the files are fine in terms of "properly decoding".
To compare if the audio contents match after converting you can use a batchfile (by uhclem). Sure, if you fix sector boundary errors, the files will not match anymore.
_____________
@echo off
for %%T in (*.shn) do shntool cmp %%T %%~nT.flac
pause
_____________

Or you create st5s of shns and flacs and compare by eye:
967c57a1a178f952f931a5a090252c8d [shntool] ns2002-11-18t01.shn
bff785557d40fff55dea7e5784583d31 [shntool] ns2002-11-18t02.shn
967c57a1a178f952f931a5a090252c8d [shntool] ns2002-11-18t01.flac
bff785557d40fff55dea7e5784583d31 [shntool] ns2002-11-18t02.flac

Originally Posted by feralicious

So... if the md5 is a must, then why do people think you don't need them for flacs? I don't see what the difference is if "testing" isn't enough for shns why would it be for flacs?

On flacs "test encoded files" (TraderLittleHelper) or "Verify" (FlacFrontend) the files are decoded, the checksums calculated and compared to the checksum in the header in just one step. If there´s a problem on decoding, it will show up; if the checksums do not match, it will show up.
Shns do not have checksum informations included. That´s why you need an textfile.

Originally Posted by feralicious

When someone says "whole file md5" that's not the same as an st5, right? That's like an md5 of all the tracks?

st5 is the calculated checksum of the decompressed (decoded) audio content only.
it´s the same as a flac fingerprint, just in another text format but also usable for other audio formats like shn, ape, ....

ffp:
beatles-urt-d1t01.flac:95fd6f5e85fab21a1a84e9447af3d6bd
beatles-urt-d1t02.flac:b0ae6b35acc4f4eea9783154444ee829
st5:
95fd6f5e85fab21a1a84e9447af3d6bd [shntool] beatles-urt-d1t01.flac
b0ae6b35acc4f4eea9783154444ee829 [shntool] beatles-urt-d1t02.flac

In wholefile md5s of audio files a checksum of the whole file is calculated (including headers, tags, etc.; no decoding is done).

[feralicious]

Okay, thanks range_hood.

I used to bitcompare in foobar when I converted shn>flac, but now I make st5 of the shns first, then of the flacs and visually compare. I'll try uhclem's batch file again, but last time I tried it I couldn't get it working, though I think I was writing it out myself so I'm sure I just couldn't preoperly figure out the commands.

I swear if only I didn't care about all this I could've had all my 700gb archived already! It's annoying how much slop is getting passed around that has to be fixed.