View Single Post
  #1  
Old 2006-02-09, 03:49 PM
Five's Avatar
Five Five is offline
 
Join Date: Oct 2004
Location: Canada
Checksums Demystified (.st5, .ffp and .md5)

TTD Requirements:
SHN/APE: .st5 fingerprints posted in announce thread, should also be included with the show. Since SHN format is not capable of self-testing, some form of checksums must be included.
FLAC: .st5/.ffp fingerprints posted in announce thread, should also be included with the show.
DVD: .md5 checksums posted in announce thread and included with the show.

Checksums (.st5, .ffp and .md5)

Checksums are used for the purpose of verifying the integrity and identity of a digital copy of audio. There are many kinds of checksums but only a few that are commonly used by music traders:

.md5 wholefile checksum. verifies that files are identical in every single way, and is therefore sensitive to compression setting, file format et al.

.st5 aka SHNtool md5. this is a checksum taken from the decompressed audio only. the advantage to this is that it works with any lossless codec at any compression setting. This is the preferred checksum type to be included with FLAC, SHN or APE files.

.ffp aka FLAC fingerprints. These checksums are virtually identical to .st5. some differences are explained below, under "Verifying a checksum file" and "Creating a .ffp file".

For audio, I highly recommend using FLAC and including .st5 and .ffp no wholefile .md5 necessary.

For Mac use xAct. Tutorials here:
Making ShnTool md5s using xACT
http://u2lynne.sandsmuseum.com/checksums.html

For PC use TLH:
http://tlh.easytree.org/


TLH checksums tutorial

Testing lossless files:

This function runs a simple test to see if the files will decode properly.

1. Go to Test encoded files > Add

2. Use Files of type to select the lossless file type.

3. Select all and hit open.



4. Check that the information displayed in the list is correct.

5. Press Test

6. The result will take a few minutes to display.




Verifying a checksum file:

TLH can verify .st5, .ffp and .md5 files. You can simply double-click the checksum file to begin testing, or:

1. Go to Verify checksum file > Add

2. Use Files of type to select checksum type. Load the checksum file.

3. Press Test.

4. This will take a few minutes.

ffp compares the checksums in the FLAC header without checking the audio.



.md5 checks for any change whatsoever in the whole file (useful for DVD).

.st5 checks the decompressed audio only.



md5 for DVD cannot find the files, what do I do?

To verify a playable video DVD with checksums which are on your hd, generate md5 checksums (see below) directly from the DVD and then test those against the files which are on your hd.

Another problem that sometimes occurs is that the paths stored in the .md5 file do not tell TLH to look inside the VIDEO_TS folder. So if the .md5 file for your DVD looks like this:

Code:
8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS.BUP
8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS.IFO
e40bd45c55f783d519c2a5c9aa088e98 *VIDEO_TS.VOB
e11aaf0c14501bbf15d776a34218c205 *VTS_01_0.BUP
e11aaf0c14501bbf15d776a34218c205 *VTS_01_0.IFO
da29301176f55de40d79be5cba560d06 *VTS_01_0.VOB
99ba41ff0fec88e3def109b6cb824dcb *VTS_01_1.VOB
6c20fa2d2232106310d61dce30cd3e29 *VTS_01_2.VOB
you should (a) make a backup of the original .md5 then (b) add VIDEO_TS/ to the path (open, edit & save using notepad), and test again. The modified .md5 file for the example posted above would look like this:

Code:
8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS/VIDEO_TS.BUP
8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS/VIDEO_TS.IFO
e40bd45c55f783d519c2a5c9aa088e98 *VIDEO_TS/VIDEO_TS.VOB
e11aaf0c14501bbf15d776a34218c205 *VIDEO_TS/VTS_01_0.BUP
e11aaf0c14501bbf15d776a34218c205 *VIDEO_TS/VTS_01_0.IFO
da29301176f55de40d79be5cba560d06 *VIDEO_TS/VTS_01_0.VOB
99ba41ff0fec88e3def109b6cb824dcb *VIDEO_TS/VTS_01_1.VOB
6c20fa2d2232106310d61dce30cd3e29 *VIDEO_TS/VTS_01_2.VOB
Be very careful not to alter the 32-digit checksums or filenames, just add "VIDEO_TS/" (no quotes) as above.

Advanced: Another possibility is to burn a standalone-playable data DVD which contains info, artwork (if any) and .md5 checksums inside a folder called EXTRAS_TS. To do this, create three folders in your compilation:

AUDIO_TS
EXTRAS_TS
VIDEO_TS

The VIDEO_TS folder contains the obvious, the AUDIO_TS folder must be there and remain absolutely empty, and the EXTRAS_TS with the art/info/checksums inside. When the .md5 file is placed inside of an EXTRAS_TS folder, the path must be formatted as:

Code:
8d50cba158b40a75dd32cada8964aef7 *../VIDEO_TS/VIDEO_TS.BUP
8d50cba158b40a75dd32cada8964aef7 *../VIDEO_TS/VIDEO_TS.IFO
e40bd45c55f783d519c2a5c9aa088e98 *../VIDEO_TS/VIDEO_TS.VOB
e11aaf0c14501bbf15d776a34218c205 *../VIDEO_TS/VTS_01_0.BUP
e11aaf0c14501bbf15d776a34218c205 *../VIDEO_TS/VTS_01_0.IFO
da29301176f55de40d79be5cba560d06 *../VIDEO_TS/VTS_01_0.VOB
99ba41ff0fec88e3def109b6cb824dcb *../VIDEO_TS/VTS_01_1.VOB
6c20fa2d2232106310d61dce30cd3e29 *../VIDEO_TS/VTS_01_2.VOB
If you wish you can substitute \ for /. The end result is a DVD which is playable in most standalones & also includes all the stuff which is normally included in a trade. Another big advantage is that you can double-click the .md5 which is inside the EXTRAS_TS folder and TLH will test all of the files on that DVD. Just be very careful that you get everything correct the same way I explained it above or it will not work.

thanks to fatoldpig and yoho for suggesting this stuff!

Running a len (length) check:

This gives some helpful details about the set's compatability with audio cdr format. This is primarily used to check for SBEs (Sector Boundary Errors), and that the files are 16bit/44.1kHz.

For more detailed information about the meaning of the data, check here:
http://www.etree.org/shnutils/shntoo...oc/shntool.txt

1. Go to Audio file Details > Add

2. Use Files of type to select the lossless file type.

3. Select all and hit open.



4. Check that the information displayed in the list is correct.

5. Press Show details



6. Copy and paste into your info .txt file.

Code:
    length     expanded size   cdr  WAVE problems filename
     2:15.26       23875196    ---   --   ---xx   gl2005-03-12t01.flac
     5:22.52       56923148    ---   --   ---xx   gl2005-03-12t02.flac
     4:14.46       44913836    ---   --   ---xx   gl2005-03-12t03.flac
     6:17.55       66632204    ---   --   ---xx   gl2005-03-12t04.flac
     3:40.49       38923292    ---   --   ---xx   gl2005-03-12t05.flac
     2:03.55       21826604    ---   --   ---xx   gl2005-03-12t06.flac
    23:54.58      253094280 B                     (totals for 6 files, 0.4980 overall compression ratio)

Creating a .st5 file:

Test the files and run a len check before you do this.

st5 is a new name for something known as a SHNtool md5. it is a checksum for verifying only decompressed audio from a lossless file. .st5 is identical to .ffp except formatting of the text.

1. Go to Create checksum file > Browse

2. Select the folder containing the lossless files you want to fingerprint. Press Add use Files of type to change the correct format. Select all the files you want to fingerprint and press Open

3. From the dropdown menu on the right, select md5 fp then hit Create



4. The checksums will take a few minutes to calculate.



5. Save your .st5 to the same folder.



Code:
c0437a74a414351fb1fb4e4cece99e06  [shntool]  gl2005-03-12t01.flac
1822d349a6be042a49af9bb0085e093c  [shntool]  gl2005-03-12t02.flac
5028343d2820679c23a99cc0e2f6e1f5  [shntool]  gl2005-03-12t03.flac
c485a617b3af2e4267753d334b709fa2  [shntool]  gl2005-03-12t04.flac
fd6e94f4c89685e8b41e4df2c073fb48  [shntool]  gl2005-03-12t05.flac
2c3d244df2a8cb4aab370e26e4f3f393  [shntool]  gl2005-03-12t06.flac
Creating a .ffp file:

Test the files and run a len check before you do this.

1. Go to Create checksum file > Browse

2. Select the folder containing the FLACs you want to fingerprint. Press Add use Files of type to change format to FLAC. Select all the files you want to fingerprint and press Open

3. From the dropdown menu on the right, select flac fp then hit Create



4. Since the ffp checksums were calculated during initial coding they can be very quickly extracted from the headers and put into a list. FLAC files should also be tested to make sure they can be decoded properly (see "Testing lossless files" above).



5. Save your .ffp to the same folder.



ffp is generated by FLAC itself. during encoding a checksum is generated from the decoded audio only for each file and stored in the header. the checksums are the same as a .st5 (aka SHNtool md5) just the formatting of the .ffp is slightly different.

Code:
gl2005-03-12t01.flac:c0437a74a414351fb1fb4e4cece99e06
gl2005-03-12t02.flac:1822d349a6be042a49af9bb0085e093c
gl2005-03-12t03.flac:5028343d2820679c23a99cc0e2f6e1f5
gl2005-03-12t04.flac:c485a617b3af2e4267753d334b709fa2
gl2005-03-12t05.flac:fd6e94f4c89685e8b41e4df2c073fb48
gl2005-03-12t06.flac:2c3d244df2a8cb4aab370e26e4f3f393
Creating a .md5 file for a DVD:

.md5 checksums are only used for video at TTD.



1. Go to Create checksum file tab.

2. Press Browse. Select the folder containing the VIDEO_TS folder of the DVD you want to create checksums for. It is best not to select the VIDEO_TS folder, but instead the folder which contains the artwork, info and VIDEO_TS folder.

3. Press Add then use Files of type to change format to all. Double-click on the VIDEO_TS folder, select all the files inside that folder then press Open

4. From the dropdown menu on the right, select md5 sig (default).

5. Review that the files and folder are all correct, then hit Create.

6. Save your .md5 to the folder which contains the artwork (if you have), info and VIDEO_TS folder and not inside the VIDEO_TS folder. In the example, you would want to save to a folder called 1971.09.09 Hampton.

Code:
8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS/VIDEO_TS.BUP
8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS/VIDEO_TS.IFO
e40bd45c55f783d519c2a5c9aa088e98 *VIDEO_TS/VIDEO_TS.VOB
e11aaf0c14501bbf15d776a34218c205 *VIDEO_TS/VTS_01_0.BUP
e11aaf0c14501bbf15d776a34218c205 *VIDEO_TS/VTS_01_0.IFO
da29301176f55de40d79be5cba560d06 *VIDEO_TS/VTS_01_0.VOB
99ba41ff0fec88e3def109b6cb824dcb *VIDEO_TS/VTS_01_1.VOB
6c20fa2d2232106310d61dce30cd3e29 *VIDEO_TS/VTS_01_2.VOB
__________________
Checksums Demystified | ask for help in Technobabble

thetradersden.org | ttd recommended free software/freeware webring
shntool tlh eac foobar2000 spek audacity cdwave vlc

Quote:
Originally posted by oxymoron
Here you are in a place of permanent madness, be careful!
Reply With Quote Reply with Nested Quotes