PDA

View Full Version : Checksums Demystified (.st5, .ffp and .md5)


Five
2006-02-09, 04:49 PM
TTD Requirements:
SHN/APE: .st5 fingerprints posted in announce thread, should also be included with the show. Since SHN format is not capable of self-testing, some form of checksums must be included.
FLAC: .st5/.ffp fingerprints posted in announce thread, should also be included with the show.
DVD: .md5 checksums posted in announce thread and included with the show.

Checksums (.st5, .ffp and .md5)

Checksums are used for the purpose of verifying the integrity and identity of a digital copy of audio. There are many kinds of checksums but only a few that are commonly used by music traders:

.md5 wholefile checksum. verifies that files are identical in every single way, and is therefore sensitive to compression setting, file format et al.

.st5 aka SHNtool md5. this is a checksum taken from the decompressed audio only. the advantage to this is that it works with any lossless codec at any compression setting. This is the preferred checksum type to be included with FLAC, SHN or APE files.

.ffp aka FLAC fingerprints. These checksums are virtually identical to .st5. some differences are explained below, under "Verifying a checksum file" and "Creating a .ffp file".

For audio, I highly recommend using FLAC and including .st5 and .ffp no wholefile .md5 necessary.

For Mac use xAct. Tutorials here:
Making ShnTool md5s using xACT (http://www.thetradersden.org/forums/showthread.php?p=300188#post300188)
http://u2lynne.sandsmuseum.com/checksums.html

For PC use TLH:
http://tlh.easytree.org/


TLH checksums tutorial

Testing lossless files:

This function runs a simple test to see if the files will decode properly.

1. Go to Test encoded files > Add

2. Use Files of type to select the lossless file type.

3. Select all and hit open.

http://img221.echo.cx/img221/646/test18ia.th.gif (http://img221.echo.cx/my.php?image=test18ia.gif)

4. Check that the information displayed in the list is correct.

5. Press Test

6. The result will take a few minutes to display.

http://img221.echo.cx/img221/1386/test25xi.th.gif (http://img221.echo.cx/my.php?image=test25xi.gif)


Verifying a checksum file:

TLH can verify .st5, .ffp and .md5 files. You can simply double-click the checksum file to begin testing, or:

1. Go to Verify checksum file > Add

2. Use Files of type to select checksum type. Load the checksum file.

3. Press Test.

4. This will take a few minutes.

ffp compares the checksums in the FLAC header without checking the audio.

http://img288.echo.cx/img288/2185/verifyffp16wb.th.gif (http://img288.echo.cx/my.php?image=verifyffp16wb.gif) http://img288.echo.cx/img288/4466/verifyffp24rv.th.gif (http://img288.echo.cx/my.php?image=verifyffp24rv.gif) http://img288.echo.cx/img288/2671/verifyffp36hk.th.gif (http://img288.echo.cx/my.php?image=verifyffp36hk.gif)

.md5 checks for any change whatsoever in the whole file (useful for DVD).

.st5 checks the decompressed audio only.

http://img288.echo.cx/img288/2999/verifyst513hk.th.gif (http://img288.echo.cx/my.php?image=verifyst513hk.gif) http://img244.echo.cx/img244/1438/verifyst526nm.th.gif (http://img244.echo.cx/my.php?image=verifyst526nm.gif)

md5 for DVD cannot find the files, what do I do?

To verify a playable video DVD with checksums which are on your hd, generate md5 checksums (see below) directly from the DVD and then test those against the files which are on your hd.

Another problem that sometimes occurs is that the paths stored in the .md5 file do not tell TLH to look inside the VIDEO_TS folder. So if the .md5 file for your DVD looks like this:

8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS.BUP
8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS.IFO
e40bd45c55f783d519c2a5c9aa088e98 *VIDEO_TS.VOB
e11aaf0c14501bbf15d776a34218c205 *VTS_01_0.BUP
e11aaf0c14501bbf15d776a34218c205 *VTS_01_0.IFO
da29301176f55de40d79be5cba560d06 *VTS_01_0.VOB
99ba41ff0fec88e3def109b6cb824dcb *VTS_01_1.VOB
6c20fa2d2232106310d61dce30cd3e29 *VTS_01_2.VOB


you should (a) make a backup of the original .md5 then (b) add VIDEO_TS/ to the path (open, edit & save using notepad), and test again. The modified .md5 file for the example posted above would look like this:

8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS/VIDEO_TS.BUP
8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS/VIDEO_TS.IFO
e40bd45c55f783d519c2a5c9aa088e98 *VIDEO_TS/VIDEO_TS.VOB
e11aaf0c14501bbf15d776a34218c205 *VIDEO_TS/VTS_01_0.BUP
e11aaf0c14501bbf15d776a34218c205 *VIDEO_TS/VTS_01_0.IFO
da29301176f55de40d79be5cba560d06 *VIDEO_TS/VTS_01_0.VOB
99ba41ff0fec88e3def109b6cb824dcb *VIDEO_TS/VTS_01_1.VOB
6c20fa2d2232106310d61dce30cd3e29 *VIDEO_TS/VTS_01_2.VOB


Be very careful not to alter the 32-digit checksums or filenames, just add "VIDEO_TS/" (no quotes) as above.

Advanced: Another possibility is to burn a standalone-playable data DVD which contains info, artwork (if any) and .md5 checksums inside a folder called EXTRAS_TS. To do this, create three folders in your compilation:

AUDIO_TS
EXTRAS_TS
VIDEO_TS

The VIDEO_TS folder contains the obvious, the AUDIO_TS folder must be there and remain absolutely empty, and the EXTRAS_TS with the art/info/checksums inside. When the .md5 file is placed inside of an EXTRAS_TS folder, the path must be formatted as:

8d50cba158b40a75dd32cada8964aef7 *../VIDEO_TS/VIDEO_TS.BUP
8d50cba158b40a75dd32cada8964aef7 *../VIDEO_TS/VIDEO_TS.IFO
e40bd45c55f783d519c2a5c9aa088e98 *../VIDEO_TS/VIDEO_TS.VOB
e11aaf0c14501bbf15d776a34218c205 *../VIDEO_TS/VTS_01_0.BUP
e11aaf0c14501bbf15d776a34218c205 *../VIDEO_TS/VTS_01_0.IFO
da29301176f55de40d79be5cba560d06 *../VIDEO_TS/VTS_01_0.VOB
99ba41ff0fec88e3def109b6cb824dcb *../VIDEO_TS/VTS_01_1.VOB
6c20fa2d2232106310d61dce30cd3e29 *../VIDEO_TS/VTS_01_2.VOB


If you wish you can substitute \ for /. The end result is a DVD which is playable in most standalones & also includes all the stuff which is normally included in a trade. Another big advantage is that you can double-click the .md5 which is inside the EXTRAS_TS folder and TLH will test all of the files on that DVD. Just be very careful that you get everything correct the same way I explained it above or it will not work.

thanks to fatoldpig and yoho for suggesting this stuff!

Running a len (length) check:

This gives some helpful details about the set's compatability with audio cdr format. This is primarily used to check for SBEs (Sector Boundary Errors), and that the files are 16bit/44.1kHz.

For more detailed information about the meaning of the data, check here:
http://www.etree.org/shnutils/shntool/support/doc/shntool.txt

1. Go to Audio file Details > Add

2. Use Files of type to select the lossless file type.

3. Select all and hit open.

http://img244.echo.cx/img244/2027/len17wl.th.gif (http://img244.echo.cx/my.php?image=len17wl.gif)

4. Check that the information displayed in the list is correct.

5. Press Show details

http://img244.echo.cx/img244/1362/len20zm.th.gif (http://img244.echo.cx/my.php?image=len20zm.gif)

6. Copy and paste into your info .txt file.

length expanded size cdr WAVE problems filename
2:15.26 23875196 --- -- ---xx gl2005-03-12t01.flac
5:22.52 56923148 --- -- ---xx gl2005-03-12t02.flac
4:14.46 44913836 --- -- ---xx gl2005-03-12t03.flac
6:17.55 66632204 --- -- ---xx gl2005-03-12t04.flac
3:40.49 38923292 --- -- ---xx gl2005-03-12t05.flac
2:03.55 21826604 --- -- ---xx gl2005-03-12t06.flac
23:54.58 253094280 B (totals for 6 files, 0.4980 overall compression ratio)


Creating a .st5 file:

Test the files and run a len check before you do this.

st5 is a new name for something known as a SHNtool md5. it is a checksum for verifying only decompressed audio from a lossless file. .st5 is identical to .ffp except formatting of the text.

1. Go to Create checksum file > Browse

2. Select the folder containing the lossless files you want to fingerprint. Press Add use Files of type to change the correct format. Select all the files you want to fingerprint and press Open

3. From the dropdown menu on the right, select md5 fp then hit Create

http://img28.echo.cx/img28/568/createst514gb.th.gif (http://img28.echo.cx/my.php?image=createst514gb.gif)

4. The checksums will take a few minutes to calculate.

http://img28.echo.cx/img28/3979/createst528ow.th.gif (http://img28.echo.cx/my.php?image=createst528ow.gif)

5. Save your .st5 to the same folder.

http://img28.echo.cx/img28/4815/createst534xv.th.gif (http://img28.echo.cx/my.php?image=createst534xv.gif)

c0437a74a414351fb1fb4e4cece99e06 [shntool] gl2005-03-12t01.flac
1822d349a6be042a49af9bb0085e093c [shntool] gl2005-03-12t02.flac
5028343d2820679c23a99cc0e2f6e1f5 [shntool] gl2005-03-12t03.flac
c485a617b3af2e4267753d334b709fa2 [shntool] gl2005-03-12t04.flac
fd6e94f4c89685e8b41e4df2c073fb48 [shntool] gl2005-03-12t05.flac
2c3d244df2a8cb4aab370e26e4f3f393 [shntool] gl2005-03-12t06.flac

Creating a .ffp file:

Test the files and run a len check before you do this.

1. Go to Create checksum file > Browse

2. Select the folder containing the FLACs you want to fingerprint. Press Add use Files of type to change format to FLAC. Select all the files you want to fingerprint and press Open

3. From the dropdown menu on the right, select flac fp then hit Create

http://img257.echo.cx/img257/875/createffp13xu.th.gif (http://img257.echo.cx/my.php?image=createffp13xu.gif)

4. Since the ffp checksums were calculated during initial coding they can be very quickly extracted from the headers and put into a list. FLAC files should also be tested to make sure they can be decoded properly (see "Testing lossless files" above).

http://img257.echo.cx/img257/5468/createffp21sz.th.gif (http://img257.echo.cx/my.php?image=createffp21sz.gif)

5. Save your .ffp to the same folder.

http://img257.echo.cx/img257/7285/createffp31dm.th.gif (http://img257.echo.cx/my.php?image=createffp31dm.gif)

ffp is generated by FLAC itself. during encoding a checksum is generated from the decoded audio only for each file and stored in the header. the checksums are the same as a .st5 (aka SHNtool md5) just the formatting of the .ffp is slightly different.

gl2005-03-12t01.flac:c0437a74a414351fb1fb4e4cece99e06
gl2005-03-12t02.flac:1822d349a6be042a49af9bb0085e093c
gl2005-03-12t03.flac:5028343d2820679c23a99cc0e2f6e1f5
gl2005-03-12t04.flac:c485a617b3af2e4267753d334b709fa2
gl2005-03-12t05.flac:fd6e94f4c89685e8b41e4df2c073fb48
gl2005-03-12t06.flac:2c3d244df2a8cb4aab370e26e4f3f393

Creating a .md5 file for a DVD:

.md5 checksums are only used for video at TTD.

http://img432.imageshack.us/img432/4931/md5dvd15gd.th.gif (http://img432.imageshack.us/my.php?image=md5dvd15gd.gif)

1. Go to Create checksum file tab.

2. Press Browse. Select the folder containing the VIDEO_TS folder of the DVD you want to create checksums for. It is best not to select the VIDEO_TS folder, but instead the folder which contains the artwork, info and VIDEO_TS folder.

3. Press Add then use Files of type to change format to all. Double-click on the VIDEO_TS folder, select all the files inside that folder then press Open

4. From the dropdown menu on the right, select md5 sig (default).

5. Review that the files and folder are all correct, then hit Create.

6. Save your .md5 to the folder which contains the artwork (if you have), info and VIDEO_TS folder and not inside the VIDEO_TS folder. In the example, you would want to save to a folder called 1971.09.09 Hampton.

8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS/VIDEO_TS.BUP
8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS/VIDEO_TS.IFO
e40bd45c55f783d519c2a5c9aa088e98 *VIDEO_TS/VIDEO_TS.VOB
e11aaf0c14501bbf15d776a34218c205 *VIDEO_TS/VTS_01_0.BUP
e11aaf0c14501bbf15d776a34218c205 *VIDEO_TS/VTS_01_0.IFO
da29301176f55de40d79be5cba560d06 *VIDEO_TS/VTS_01_0.VOB
99ba41ff0fec88e3def109b6cb824dcb *VIDEO_TS/VTS_01_1.VOB
6c20fa2d2232106310d61dce30cd3e29 *VIDEO_TS/VTS_01_2.VOB

Five
2006-02-09, 04:50 PM
this is going to get put into the FAQ sometime soon.

please let me know if you find any type-o's/fuckups!

U2Lynne
2006-02-09, 05:28 PM
A couple notes from the Checksums section (comments in bold):

.st5 aka SHNtool md5. this is a checksum taken from the decompressed audio only. the advantage to this is that it works with any lossless codec at any compression setting. This is the preferred checksum type to be included with shn or ape files.

.ffp aka FLAC fingerprints. These checksums are virtually identical to .st5. some differences are explained below, under "Verifying a checksum file" and "Creating a .ffp file".

For audio, I highly recommend using FLAC and including .st5 and .ffp no wholefile .md5 necessary. Except the 32-digit checksum is going to be exactly the same in the st5 and ffp file. There is really no reason to include both types of checksums for fingerprints, just one type.


Here is a quick tutorial on making shntool md5s on a Mac using xACT:

1. Locate files on your computer.
http://www.thetradersden.org/images/thumb/xactshntoolmd51.jpg (http://www.thetradersden.org/images/xactshntoolmd51.jpg)

2. Open xACT to the checksum tab and drag the files into the window.
http://www.thetradersden.org/images/thumb/xactshntoolmd52.jpg (http://www.thetradersden.org/images/xactshntoolmd52.jpg)

3. Hit the Checksum button.
http://www.thetradersden.org/images/thumb/xactshntoolmd53.jpg (http://www.thetradersden.org/images/xactshntoolmd53.jpg)

4. Hit the shntool md5 button (verify is the default).
http://www.thetradersden.org/images/thumb/xactshntoolmd54.jpg (http://www.thetradersden.org/images/xactshntoolmd54.jpg)

5. It prompts you to save it somewhere, find the same folder.
http://www.thetradersden.org/images/thumb/xactshntoolmd55.jpg (http://www.thetradersden.org/images/xactshntoolmd55.jpg)

6. Name the md5 you are creating properly and hit Save (a more common extension these days is ".st5")
http://www.thetradersden.org/images/thumb/xactshntoolmd56.jpg (http://www.thetradersden.org/images/xactshntoolmd56.jpg)

7. It makes the shntool md5s (it will take a couple of minutes).
http://www.thetradersden.org/images/thumb/xactshntoolmd57.jpg (http://www.thetradersden.org/images/xactshntoolmd57.jpg)

bubbybooshay
2006-02-09, 05:36 PM
nice job five :clap:

AAR.oner
2006-02-09, 06:41 PM
:banger2: :cheers:

Five
2006-02-11, 03:18 PM
A couple notes from the Checksums section (comments in bold):

.st5 aka SHNtool md5. this is a checksum taken from the decompressed audio only. the advantage to this is that it works with any lossless codec at any compression setting. This is the preferred checksum type to be included with shn or ape files.

.ffp aka FLAC fingerprints. These checksums are virtually identical to .st5. some differences are explained below, under "Verifying a checksum file" and "Creating a .ffp file".

For audio, I highly recommend using FLAC and including .st5 and .ffp no wholefile .md5 necessary. Except the 32-digit checksum is going to be exactly the same in the st5 and ffp file. There is really no reason to include both types of checksums for fingerprints, just one type.

thanks

I added in "This is the preferred checksum type to be included with FLAC, SHN or APE files." in the st5 section.

The main reason is that when you double-click a .ffp for testing using TLH, it extracts the checksums hidden in each FLAC file's header and compares that to what is in the .ffp file to see if it matches (takes about three seconds). Using the .ffp this way for testing does not test the audio content whatsoever (which could be corrupt after a successful .ffp verification). A second stage of testing is necessary in this case, it is "test" mode, which decompresses the audio to a temp file and generates a fresh hash from this and compares that to the the checksum hidden in the header. So, only after the two-stage test the files are verified. Double-clicking on a .st5 for verification using TLH results in the audio content of the files to be tested being decompressed to a temp file (temporarily) while a fresh hash is generated and compared to the corresponding checksums contained in the .st5 file. So the .st5 test is only one step and verifies both (a) that the files will decompress properly (as with "test" mode) and (b) that the checksums contained in the .st5 file match hashes freshly generated from that decompressed audio. When a .st5 verifies 100% okay everything is tested except the header isn't looked at, thus the .ffp. The .ffp is admittedly a little superfluous, but since it is only 1k and takes about three seconds to generate and verify its not really much to ask.

oww, my brain!

AAR.oner
2006-03-19, 08:02 AM
so, to burn a data disc that is standalone compatable using the EXTRAS_TS method, would select your disc type as DVD-ROM (ISO), DVD-ROM (UDF), or DVD-ROM (ISO/UDF)?

[...and the answer is: DVD-ROM(UDF) ]

GregY
2006-03-19, 08:14 AM
There's also the excellent md5check perl script (and compiled version) that's available here:

http://etree-scripts.sourceforge.net/

I installed cygwin on my Win XP machine and use scripts and command line stuff to do all my lossless operations. Much cleaner than dealing with GUIs for me.

Five
2006-03-19, 02:16 PM
I never could get md5check running for some reason, but have heard it is really great, you just need a bit more of a grasp of commandline than I have! If somebody wants to post instructions for that it would be appreciated.

so, to burn a data disc that is standalone compatable using the EXTRAS_TS method, would select your disc type as DVD-ROM (ISO), DVD-ROM (UDF), or DVD-ROM (ISO/UDF)?
Just use the default that Nero selects when you boot up, I believe that's "DVD-ROM (ISO)", the setting you would use for burning normal non-video stuff. It works great with my DVD player and it is sweet being able to get all the info & artwork right off the same playable disc. Also great to just double-click the md5 there and the DVD self-tests.

U2Lynne
2006-03-19, 02:23 PM
On a Mac, when using Toast, if I want to add an EXTRAS_TS folder, I go to Data > DVD-ROM (UDF).

Five
2006-03-19, 02:27 PM
ack... okay, maybe its DVD-ROM (UDF) on PC, too. it is the default when you start up Nero. My DVD burner is on my other computer, I can check it later and post back.

Five
2006-04-17, 05:08 AM
hey

it is DVD-ROM (ISO) mode in Nero that I am using for the EXTRAS_TS method... works great on comp and standalone :thumbsup

AAR.oner
2006-04-17, 07:33 AM
hey

it is DVD-ROM (ISO) mode in Nero that I am using for the EXTRAS_TS method... works great on comp and standalone :thumbsup
cheers! i'll give that one a try then Five

karmakat
2006-04-17, 08:02 AM
:clap: Good job Five. About six months too late (for my personal use)...:run:

mike1061
2006-05-17, 12:20 AM
Great job Jamie.
Mike

cmaz
2006-05-17, 04:48 PM
i have a question regarding the difference between verifying and testing with TLH. If i verify the files, do i also need to test them? If the files verify, won't they neccessarily decode properly? And what about the check box "full ffp vrf"? When, where, and why do i choose each of these options?

When do i verify, test, and/or choose "full ffp"?

Thanks!

aristote
2007-05-16, 02:07 PM
:thumbsup

Five
2007-05-16, 10:23 PM
i have a question regarding the difference between verifying and testing with TLH. If i verify the files, do i also need to test them? If the files verify, won't they neccessarily decode properly? And what about the check box "full ffp vrf"? When, where, and why do i choose each of these options?

When do i verify, test, and/or choose "full ffp"?

Thanks!
since this tutorial was written there have been some new features added to TLH... if you test with the "full" feature enabled (as it is by default) TLH will check the files every way possible and only that one test/verification is necessary.

shovelhead
2008-04-05, 02:19 PM
WAY COOL!!!! THANK YOU SO VERY MUCH FOR YOUR TIME IN POSTING & GENEROSITY IN SHARING!!! WAY COOL!!!! THANKS AGAIN!!!! :cool::cool::cool::cool:

salmonella
2012-06-27, 11:53 PM
Is there any chance of updating the initial post in this thread? I'm trying to make a SHNTOOL md5 with xACT, however, the interface looks completely different. The example given is not helpful 6 years later.

I sincerely hope nobody takes this as complaining on my part. I know you all want this to be as helpful as possible.

Thank you.

scb
2012-07-01, 02:27 PM
in xACT 2.x, on the Checksum tab, click "Create Checksum File" and choose the "St5" option

AAR.oner
2012-07-01, 04:48 PM
what scb said ^^


select the files you wanna include in Finder, drag to the window on left in xACT [screen below], then select the options circled in red...click "Checksum" button and let it do its thing

Five
2012-07-09, 03:13 AM
Is there any chance of updating the initial post in this thread? I'm trying to make a SHNTOOL md5 with xACT, however, the interface looks completely different. The example given is not helpful 6 years later.

I sincerely hope nobody takes this as complaining on my part. I know you all want this to be as helpful as possible.

Thank you.

wow, 6yrs... I will d/l the new TLH sometime in coming months and update

SteelStage
2015-08-11, 01:56 AM
hey, read the FAQ's and having issues creating uploaded torrent

Have created ffp and st5 with TLH, can't figure out the shntool nor how to get it working

Getting this

The fingerprints provided are not shntool st5s or flac fingerprints

any more help would be appreciated.

jabulon
2015-08-11, 04:18 AM
hey, read the FAQ's and having issues creating uploaded torrent

Have created ffp and st5 with TLH, can't figure out the shntool nor how to get it working

Getting this

The fingerprints provided are not shntool st5s or flac fingerprints

any more help would be appreciated.

Are you talking about audio or video? I assume audio, because you talk about flac. Windows?


ps edit: forget about the shntool, when creating the torrent file just add the ffp and/or st5 that is good enough.

SteelStage
2015-08-11, 08:59 AM
yes audio, flac and yes windows.

when i create the torrent (with utorrent engine), then attach and load it tells me the message i tried above. Maybe my torrent is bad, when i create the torrent, i direct the folder with all files, do i need to add all the files?

jabulon
2015-08-11, 10:24 AM
Try this: Make the torrent file with TLH as well.

open Torrent select create torrent file.
add directory
check private torrent
tracker/announce (select ttd tracker)
save torrent file (set the torrent location)
create

SteelStage
2015-08-11, 10:26 AM
THanks I'll def try that and let u know

SteelStage
2015-08-11, 12:54 PM
Tried TLH for torrent,

getting this error upon upload now: "Forget something? How about some fingerprints, please."

In the file directory i have ffp and st5 files, is that correct?

jabulon
2015-08-11, 01:02 PM
You have to copy and past the ffp (the text - open the ffp-file with text editor /notepad) in the second big box on the upload page

SteelStage
2015-08-11, 02:01 PM
I'm good now thanks for you help man. I will seed some shows

jabulon
2015-08-11, 02:03 PM
:thumbsup

SteelStage
2015-08-11, 02:16 PM
one more thing, the initial seed, that seems to be stuck, any guidance on that?

I did download from the TTD page it was linked in url, then i added it to utorrent as initial seed...see it in download status 99.9 and not in seed page, but it progressed to 'finding peers' this has always baffled me doing this once before

so i think i am stuck at step 11 in audio torrent seed FAQs

jabulon
2015-08-11, 02:30 PM
The 99.9% etc comes when you have changed something after making the torrent. Delete this torrent. Make a new torrent (make sure all files are closed and saved when making the torrent. Then follow the upload procedure. I'll delete your old upload page.

SteelStage
2015-08-11, 02:37 PM
thanks i'll do that

SteelStage
2015-08-11, 02:54 PM
and we're good. Much appreciation and patience jabulon

jabulon
2015-08-11, 02:56 PM
:thumbsup