PDA

View Full Version : Flac Fingerprint vs. st5 / Flac Frontend vs. TLH ???


feralicious
2005-09-27, 07:00 PM
Okay. Here goes...

Checking this show I downloaded. I didn't notice it had ffp since it was a text file so I verified the files with Flac Frontend v.1.7.1. Half the files failed:

http://img388.imageshack.us/img388/2050/md5mismatch7sh.jpg (http://imageshack.us)

Then I notice the ffp txt file so I change the extension to .ffp and run it with TLH and everything verifies ok, says checksums match for all files.

So I make a new ffp and it matches what is in the original.

I run them through Flac Frontend again, same errors on half the files.

I make an st5 and the files that failed have a different checksum than the ffp.

So now I'm wondering what is the point of having and checking ffp if they can go through TLH with no errors? I've been checking and archiving shows that I checked with TLH and now I'm not certain that everything's okay with them. I'm a bit perturbed by it too. I guess I should have been verifying them with Flac Frontend instead.

So now what do I do with these files? How did this happen? I need to fix a couple SBEs so I guess I'll see what happens after I do that.

Comments? Suggestions? Thanks.

feralicious
2005-09-27, 07:15 PM
Okay, so I did the SBE fix with TLH and the fixed files verified fine with Flac Frontend. Then I made new ffp and st5 for these fixed files and they all matched.

Can someone explain what happened and why? I ran the SBE fix on all files even though just 2 of them had SBE, so even though they theoretically didn't change, they must have since they now verify and come out with different ffp. :confused:

I guess the moral of the story is to continue to check all st5 and ffp against each other before archiving/verify all files with Flac Frontend. I was doing that but I got lazy on the last DVD-R I just archived. Hope it's okay...

AAR.oner
2005-09-27, 08:48 PM
i'm not sure whats going on with that show fera, but this is why i don't really like ffp.txt checksums as the ONLY checksum...since its a text file, it can be manually edited easily [opens the door for issues/mishaps]...

st5 seems the best route now for all lossless files...way better than wholefile md5, and better than ffp's in my opinion

feralicious
2005-09-27, 08:53 PM
Well the ffp wasn't wrong, the files themselves were. I made a new ffp and it matched the one that came with the show. But neither matched the st5. Then after they went through the SBE fix they were fine and the ffp matched the st5.

spiritinaphoto
2005-09-28, 12:43 AM
i'm not sure whats going on with that show fera, but this is why i don't really like ffp.txt checksums as the ONLY checksum...since its a text file, it can be manually edited easily [opens the door for issues/mishaps]...

st5 seems the best route now for all lossless files...way better than wholefile md5, and better than ffp's in my opinion
Explain to me how it is that you can edit ffp's but not st5's. I thought st5's could also be opened in any garden-variety text editor and be tweaked. :hmm:

feralicious
2005-09-28, 12:52 AM
Explain to me how it is that you can edit ffp's but not st5's. I thought st5's could also be opened in any garden-variety text editor and be tweaked. :hmm:Yes they can.

spiritinaphoto
2005-09-28, 01:22 AM
Yes they can.
I knew at least the ones I made could, because opening a text editor and pasting what I got from the command-line output was how I made them. But of course, I could be making them the wrong way for all I knew.

diggrd
2005-09-28, 03:47 AM
Well the ffp wasn't wrong, the files themselves were. I made a new ffp and it matched the one that came with the show. But neither matched the st5. Then after they went through the SBE fix they were fine and the ffp matched the st5.
This torrent had a similar problem
http://bt.etree.org/details.php?id=3038

AAR.oner
2005-09-28, 08:25 AM
Explain to me how it is that you can edit ffp's but not st5's. I thought st5's could also be opened in any garden-variety text editor and be tweaked. :hmm:
you can edit any of them true...but when running an st5 with xACT or TLH, the st5 automatically runs in one of those progs when you double-click...with a ffp.txt, it automatically opens in yer text editor [atleast on my PC it does]...this simply makes it easier for an "accident" to happen and the fingerprint to get changed...does that make sense? ignore me if it doesn't--this is just one of my neurotic pet peeves and i'm still on coffee #1 :D

TheMamba
2005-09-28, 10:40 AM
you can edit any of them true...but when running an st5 with xACT or TLH, the st5 automatically runs in one of those progs when you double-click...with a ffp.txt, it automatically opens in yer text editor [atleast on my PC it does]...this simply makes it easier for an "accident" to happen and the fingerprint to get changed...does that make sense? ignore me if it doesn't--this is just one of my neurotic pet peeves and i'm still on coffee #1 :D

Coffee #2 here and it makes sense to me. :wave:

feralicious
2005-09-28, 11:50 AM
I knew at least the ones I made could, because opening a text editor and pasting what I got from the command-line output was how I made them. But of course, I could be making them the wrong way for all I knew.Can you add a command to have it make a txt file in the show's folder? That's what my shntool batch file does.

you can edit any of them true...but when running an st5 with xACT or TLH, the st5 automatically runs in one of those progs when you double-click...with a ffp.txt, it automatically opens in yer text editor [atleast on my PC it does]...this simply makes it easier for an "accident" to happen and the fingerprint to get changed...does that make sense? ignore me if it doesn't--this is just one of my neurotic pet peeves and i'm still on coffee #1 :DWARNING!
PLEASE FINISH COFFEE #2 BEFORE CONTINUING
PROCEED WITH CAUTION

I just change the extension to .ffp and voila!

AAR.oner
2005-09-28, 12:12 PM
:lol yep fera, i've just gotten a few shows before where someone [inadvertently i hope] had screwed up the ffp.txt file and i had to make new ones...like i said...neurotic...pet peeve :rolleyes:

range_hood
2005-09-28, 01:10 PM
Thatīs Blondie July 9, 1979 (http://www.dimeadozen.org/torrents-details.php?id=49926), isnīt it?
These files are rotting on my harddisk since I downloaded. I asked dgkdgk (the taper and seeder) to reseed them and reported the torrent to the DaD staff. No response from the seeder, a mod over there said the torrent is ok as long as no other leecher has a problem with it.

If you listen to the problematic tracks you will hear digital errors, so please donīt trade this further without noting that in the info file.

The ffp-txt file matches the "fingerprint infomation" in the flac files, but the audio content has different checksums. Maybe the seeder flaced the files without verify?
In TLH "Verify checksum file" (ffp) does only compare the textfile to the "fingerprints" stored in the flacs. Where "Test encoded files" (same as flac.exes Verify) decodes the audio, calculates the checksums and compares that to the "fingerprint information" in the flacs.
This "Test encoded files" is comparable to a st5 checksum verification.

What I do is checking if all checksums do match and "test encoded files" (all in TLH) - additional listening to questionable files a bit closer and noting all problems. And maybe persuading the seeder, as he seems to be the taper, to reseed a fixed show.

spiritinaphoto
2005-09-28, 01:34 PM
Can you add a command to have it make a txt file in the show's folder? That's what my shntool batch file does.
I don't use batch files--I just enter in the commands directly.

you can edit any of them true...but when running an st5 with xACT or TLH, the st5 automatically runs in one of those progs when you double-click...with a ffp.txt, it automatically opens in yer text editor [atleast on my PC it does]...this simply makes it easier for an "accident" to happen and the fingerprint to get changed...does that make sense? ignore me if it doesn't--this is just one of my neurotic pet peeves and i'm still on coffee #1 :D
I've got the perfect solution for you--it's called read-only. ;)

feralicious
2005-09-28, 01:41 PM
:lol yep fera, i've just gotten a few shows before where someone [inadvertently i hope] had screwed up the ffp.txt file and i had to make new ones...like i said...neurotic...pet peeve :rolleyes::lol
Okay, so I should have told you to have coffee #3 or 4 first. You don't have to make a new one. Just change the extension of the existing file to make it a ffp file.

feralicious
2005-09-28, 01:46 PM
Thatīs Blondie July 9, 1979 (http://www.dimeadozen.org/torrents-details.php?id=49926), isnīt it?
These files are rotting on my harddisk since I downloaded. I asked dgkdgk (the taper and seeder) to reseed them and reported the torrent to the DaD staff. No response from the seeder, a mod over there said the torrent is ok as long as no other leecher has a problem with it.

If you listen to the problematic tracks you will hear digital errors, so please donīt trade this further without noting that in the info file.

The ffp-txt file matches the "fingerprint infomation" in the flac files, but the audio content has different checksums. Maybe the seeder flaced the files without verify?
In TLH "Verify checksum file" (ffp) does only compare the textfile to the "fingerprints" stored in the flacs. Where "Test encoded files" (same as flac.exes Verify) decodes the audio, calculates the checksums and compares that to the "fingerprint information" in the flacs.
This "Test encoded files" is comparable to a st5 checksum verification.

What I do is checking if all checksums do match and "test encoded files" (all in TLH) - additional listening to questionable files a bit closer and noting all problems. And maybe persuading the seeder, as he seems to be the taper, to reseed a fixed show.Yep that's the one. So how come when I fix the SBEs, processing all the files, the new files are fine and the flac ffp/checksum matches the audio checksum?

And so really basic ffp and verifying just those is useless then. Why even have them? Now I know to verify the files and not just run the ffp verify. Had I not happened to do it on this show I would've just checked using the ffp and thought it was fine. That's bad because I actually spent (too much) time trying to understand all this and SBEs, just think about all those others who don't even look into it.

range_hood
2005-09-28, 02:42 PM
Yep that's the one. So how come when I fix the SBEs, processing all the files, the new files are fine and the flac ffp/checksum matches the audio checksum?
On re-encoding (fixing sbes) you calculated new checksums of the actual audio content.

And so really basic ffp and verifying just those is useless then. Why even have them? Now I know to verify the files and not just run the ffp verify. Had I not happened to do it on this show I would've just checked using the ffp and thought it was fine. That's bad because I actually spent (too much) time trying to understand all this and SBEs, just think about all those others who don't even look into it.Thatīs a good point. The function should be disabled in TLH for ffp in the next release. Maybe ban *.ffp and use *.st5 as standard. TLH development seems not to be that vital.
You have to have fingerprints as a textfile, so users can compare their sources without having to download the whole show.

AAR.oner
2005-09-28, 02:56 PM
nope...i've not been clear... :lol i understand what ya'll are saying...

i received a show ina trade...the only checksum included was a ffp.txt...when i went to verify it, it failed...so i do some research and find the "real" ffp online, checked it against my copy of the show, and it verified correct...so i compare the two ffp's...seems the ffp.txt i got in the trade had been messed up [i think a few of the letters/numbers got jumbled in the fingerprint text itself]...hence my orig point:

when creating a lossless fileset [pref. flac of course ;) ] on a show for the first time, best to create an st5 checksum, and not a ffp.txt *only*...they are more prone to "user-error" than a .st5 or even a regular .ffp...

Five
2005-09-29, 10:09 AM
after briefly reading this thread, here's what I can tell you...

when a FLAC file is created, the ffp for that file is stored in the header at the time the audio is compressed.

creating a ffp merely copies this checksum to a .txt file (using FLAC frontend) or a .ffp (using TLH). the .ffp file is just like the .txt file, just with a different extension.

when a .ffp (or ffp.txt) file is verified using TLH, it takes the .ffp file and compares the checksum associated with each file with the checksum stored in the header. it doesn't actually look further than that. That's why it takes such a short time to do this function.

FLAC frontend can't verify .ffp files this way.

when you use the "test" function on FLAC frontend, it decompresses the audio to a temp file, generates a checksum and compares this with the checksum which was stored in the header at the time the file was created. TLH can also do this, using the "test encoded files" function.

okay, here's the tricky part:

when you use TLH (or shntool) to verify an .st5 file, TLH decompresses the audio to a temp file, generates fresh checksums and compares these with the checksums contained in the .st5 file and doesn't check the headers of the FLAC files at all.

also, when you fix a SBE, the file is decompressed, altered, then re-encoded to FLAC and therefore a fresh checksum is generated and inserted into the header.

so the error from flac frontend "MD5 signature mismatch" should also be reported from TLH when you use the "test encoded files" function, and it means that the checksums contained in the headers of those files don't match the checksums generated on the spot from the WAV audio decompressed to a temp file.

when you made the new .ffp it took all the checksums from the headers just like the seeder did, therefore they are the same.

then you make an .st5 and it shows the true checksums for the files, some of which have incorrect checksums stored in their headers at the original time of compression.

the files you fixed were re-encoded and new headers were written which you tested as being accurate.

so, to fully verify a FLAC file, you have to do a couple things:

1) check that the checksums contained in the headers match the checksums of the audio contained within. you can do this using either TLH "test encoded..." or FLAC frontend verify/test.

2) if there is a .ffp file, check that it matches what is contained in the headers using TLH "verify checksum" or by generating a fresh fingerprint using FLAC frontend and comparing by eye.

3) if there is a .st5 file, verify it using TLH "verify checksum file". If you want to do the same using only FLAC frontend, run a test to make sure the headers match the files, generate a ffp.txt, then compare the checksums by eye with the .st5 checksums.

is that confusing enough? :lol

remember, I'm only on coffee #1...

Five
2005-09-29, 10:11 AM
and I find it very strange that the checksums in the header don't match the audio in some files... that's a problem that happened during compression, and I've never seen this before and am not able to re-create this sort of a problem. My gf always thought I was paranoid to bother testing FLACs so thoroughly, turns out its a good idea!

AAR.oner
2005-09-29, 10:52 AM
i apologize for my hijacking of this thread Fera...sorry :wave:

i'ma start a new thread re: st5 vs ffp vs md5, WANT ya'lls thoughts opinions on it tho...cheers!

range_hood
2005-09-29, 03:40 PM
and I find it very strange that the checksums in the header don't match the audio in some files... that's a problem that happened during compression, and I've never seen this before and am not able to re-create this sort of a problem.
I only can imagine this on encoding flacs while other programs are loading the cpu; without checking "Verify" in flac frontend.

feralicious
2005-09-29, 09:47 PM
Well now I'm wondering about shn files. Do you need to both verify the md5 and "test" the files? I have been since I came across this problem but it feels like overkill.

Aaron... don't worry... I never thought you had hijacked it.

range_hood... I reported that torrent and they pulled it. Of course it had already run its course and hundreds are trading it at this very moment, but hey...

feralicious
2005-10-01, 01:32 PM
Well now I'm wondering about shn files. Do you need to both verify the md5 and "test" the files? I have been since I came across this problem but it feels like overkill.Still wondering if I'm wasting my time. Can I just do the "test encoded files"? I'm going to convert them to flac anyway before I archive.

range_hood
2005-10-01, 01:58 PM
Still wondering if I'm wasting my time. Can I just do the "test encoded files"? I'm going to convert them to flac anyway before I archive.On "Test encoded files" the files are decoded to a temp directory. As you convert them to flac anyway (shn>wav>flac) the test is included in the conversion (say, itīs no test anymore :)).
The md5 check is a must, to determine if the files are the same as the seeders ones.

feralicious
2005-10-01, 02:08 PM
What if I'm going straight from shn>flac? I use foobar.

I swear I think I have all this down then something like this comes up and confuses me again.

So... if the md5 is a must, then why do people think you don't need them for flacs? I don't see what the difference is if "testing" isn't enough for shns why would it be for flacs?

When someone says "whole file md5" that's not the same as an st5, right? That's like an md5 of all the tracks?

range_hood
2005-10-01, 04:26 PM
What if I'm going straight from shn>flac? I use foobar.If no error shows up on reencoding, then the files are fine in terms of "properly decoding".
To compare if the audio contents match after converting you can use a batchfile (by uhclem). Sure, if you fix sector boundary errors, the files will not match anymore.
_____________
@echo off
for %%T in (*.shn) do shntool cmp %%T %%~nT.flac
pause
_____________

Or you create st5s of shns and flacs and compare by eye:
967c57a1a178f952f931a5a090252c8d [shntool] ns2002-11-18t01.shn
bff785557d40fff55dea7e5784583d31 [shntool] ns2002-11-18t02.shn
967c57a1a178f952f931a5a090252c8d [shntool] ns2002-11-18t01.flac
bff785557d40fff55dea7e5784583d31 [shntool] ns2002-11-18t02.flac

So... if the md5 is a must, then why do people think you don't need them for flacs? I don't see what the difference is if "testing" isn't enough for shns why would it be for flacs?On flacs "test encoded files" (TraderLittleHelper) or "Verify" (FlacFrontend) the files are decoded, the checksums calculated and compared to the checksum in the header in just one step. If thereīs a problem on decoding, it will show up; if the checksums do not match, it will show up.
Shns do not have checksum informations included. Thatīs why you need an textfile.

When someone says "whole file md5" that's not the same as an st5, right? That's like an md5 of all the tracks?st5 is the calculated checksum of the decompressed (decoded) audio content only.
itīs the same as a flac fingerprint, just in another text format but also usable for other audio formats like shn, ape, ....

ffp:
beatles-urt-d1t01.flac:95fd6f5e85fab21a1a84e9447af3d6bd
beatles-urt-d1t02.flac:b0ae6b35acc4f4eea9783154444ee829
st5:
95fd6f5e85fab21a1a84e9447af3d6bd [shntool] beatles-urt-d1t01.flac
b0ae6b35acc4f4eea9783154444ee829 [shntool] beatles-urt-d1t02.flac

In wholefile md5s of audio files a checksum of the whole file is calculated (including headers, tags, etc.; no decoding is done).

feralicious
2005-10-01, 07:01 PM
Okay, thanks range_hood.

I used to bitcompare in foobar when I converted shn>flac, but now I make st5 of the shns first, then of the flacs and visually compare. I'll try uhclem's batch file again, but last time I tried it I couldn't get it working, though I think I was writing it out myself so I'm sure I just couldn't preoperly figure out the commands.

I swear if only I didn't care about all this I could've had all my 700gb archived already! It's annoying how much slop is getting passed around that has to be fixed.