Checksums Demystified (.st5, .ffp and .md5)

[Five]

TTD Requirements:
SHN/APE: .st5 fingerprints posted in announce thread, should also be included with the show. Since SHN format is not capable of self-testing, some form of checksums must be included.
FLAC: .st5/.ffp fingerprints posted in announce thread, should also be included with the show.
DVD: .md5 checksums posted in announce thread and included with the show.

Checksums (.st5, .ffp and .md5)

Checksums are used for the purpose of verifying the integrity and identity of a digital copy of audio. There are many kinds of checksums but only a few that are commonly used by music traders:

.md5 wholefile checksum. verifies that files are identical in every single way, and is therefore sensitive to compression setting, file format et al.

.st5 aka SHNtool md5. this is a checksum taken from the decompressed audio only. the advantage to this is that it works with any lossless codec at any compression setting. This is the preferred checksum type to be included with FLAC, SHN or APE files.

.ffp aka FLAC fingerprints. These checksums are virtually identical to .st5. some differences are explained below, under "Verifying a checksum file" and "Creating a .ffp file".

For audio, I highly recommend using FLAC and including .st5 and .ffp no wholefile .md5 necessary.

For Mac use xAct. Tutorials here:
Making ShnTool md5s using xACT
http://u2lynne.sandsmuseum.com/checksums.html

For PC use TLH:
http://tlh.easytree.org/


TLH checksums tutorial

Testing lossless files:

This function runs a simple test to see if the files will decode properly.

1. Go to Test encoded files > Add

2. Use Files of type to select the lossless file type.

3. Select all and hit open.



4. Check that the information displayed in the list is correct.

5. Press Test

6. The result will take a few minutes to display.




Verifying a checksum file:

TLH can verify .st5, .ffp and .md5 files. You can simply double-click the checksum file to begin testing, or:

1. Go to Verify checksum file > Add

2. Use Files of type to select checksum type. Load the checksum file.

3. Press Test.

4. This will take a few minutes.

ffp compares the checksums in the FLAC header without checking the audio.



.md5 checks for any change whatsoever in the whole file (useful for DVD).

.st5 checks the decompressed audio only.



md5 for DVD cannot find the files, what do I do?

To verify a playable video DVD with checksums which are on your hd, generate md5 checksums (see below) directly from the DVD and then test those against the files which are on your hd.

Another problem that sometimes occurs is that the paths stored in the .md5 file do not tell TLH to look inside the VIDEO_TS folder. So if the .md5 file for your DVD looks like this:

8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS.BUP
8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS.IFO
e40bd45c55f783d519c2a5c9aa088e98 *VIDEO_TS.VOB
e11aaf0c14501bbf15d776a34218c205 *VTS_01_0.BUP
e11aaf0c14501bbf15d776a34218c205 *VTS_01_0.IFO
da29301176f55de40d79be5cba560d06 *VTS_01_0.VOB
99ba41ff0fec88e3def109b6cb824dcb *VTS_01_1.VOB
6c20fa2d2232106310d61dce30cd3e29 *VTS_01_2.VOB

you should (a) make a backup of the original .md5 then (b) add VIDEO_TS/ to the path (open, edit & save using notepad), and test again. The modified .md5 file for the example posted above would look like this:

8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS/VIDEO_TS.BUP
8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS/VIDEO_TS.IFO
e40bd45c55f783d519c2a5c9aa088e98 *VIDEO_TS/VIDEO_TS.VOB
e11aaf0c14501bbf15d776a34218c205 *VIDEO_TS/VTS_01_0.BUP
e11aaf0c14501bbf15d776a34218c205 *VIDEO_TS/VTS_01_0.IFO
da29301176f55de40d79be5cba560d06 *VIDEO_TS/VTS_01_0.VOB
99ba41ff0fec88e3def109b6cb824dcb *VIDEO_TS/VTS_01_1.VOB
6c20fa2d2232106310d61dce30cd3e29 *VIDEO_TS/VTS_01_2.VOB

Be very careful not to alter the 32-digit checksums or filenames, just add "VIDEO_TS/" (no quotes) as above.

Advanced: Another possibility is to burn a standalone-playable data DVD which contains info, artwork (if any) and .md5 checksums inside a folder called EXTRAS_TS. To do this, create three folders in your compilation:

AUDIO_TS
EXTRAS_TS
VIDEO_TS

The VIDEO_TS folder contains the obvious, the AUDIO_TS folder must be there and remain absolutely empty, and the EXTRAS_TS with the art/info/checksums inside. When the .md5 file is placed inside of an EXTRAS_TS folder, the path must be formatted as:

8d50cba158b40a75dd32cada8964aef7 *../VIDEO_TS/VIDEO_TS.BUP
8d50cba158b40a75dd32cada8964aef7 *../VIDEO_TS/VIDEO_TS.IFO
e40bd45c55f783d519c2a5c9aa088e98 *../VIDEO_TS/VIDEO_TS.VOB
e11aaf0c14501bbf15d776a34218c205 *../VIDEO_TS/VTS_01_0.BUP
e11aaf0c14501bbf15d776a34218c205 *../VIDEO_TS/VTS_01_0.IFO
da29301176f55de40d79be5cba560d06 *../VIDEO_TS/VTS_01_0.VOB
99ba41ff0fec88e3def109b6cb824dcb *../VIDEO_TS/VTS_01_1.VOB
6c20fa2d2232106310d61dce30cd3e29 *../VIDEO_TS/VTS_01_2.VOB

If you wish you can substitute \ for /. The end result is a DVD which is playable in most standalones & also includes all the stuff which is normally included in a trade. Another big advantage is that you can double-click the .md5 which is inside the EXTRAS_TS folder and TLH will test all of the files on that DVD. Just be very careful that you get everything correct the same way I explained it above or it will not work.

thanks to fatoldpig and yoho for suggesting this stuff!

Running a len (length) check:

This gives some helpful details about the set's compatability with audio cdr format. This is primarily used to check for SBEs (Sector Boundary Errors), and that the files are 16bit/44.1kHz.

For more detailed information about the meaning of the data, check here:
http://www.etree.org/shnutils/shntoo...oc/shntool.txt

1. Go to Audio file Details > Add

2. Use Files of type to select the lossless file type.

3. Select all and hit open.



4. Check that the information displayed in the list is correct.

5. Press Show details



6. Copy and paste into your info .txt file.

    length     expanded size   cdr  WAVE problems filename
     2:15.26       23875196    ---   --   ---xx   gl2005-03-12t01.flac
     5:22.52       56923148    ---   --   ---xx   gl2005-03-12t02.flac
     4:14.46       44913836    ---   --   ---xx   gl2005-03-12t03.flac
     6:17.55       66632204    ---   --   ---xx   gl2005-03-12t04.flac
     3:40.49       38923292    ---   --   ---xx   gl2005-03-12t05.flac
     2:03.55       21826604    ---   --   ---xx   gl2005-03-12t06.flac
    23:54.58      253094280 B                     (totals for 6 files, 0.4980 overall compression ratio)

Creating a .st5 file:

Test the files and run a len check before you do this.

st5 is a new name for something known as a SHNtool md5. it is a checksum for verifying only decompressed audio from a lossless file. .st5 is identical to .ffp except formatting of the text.

1. Go to Create checksum file > Browse

2. Select the folder containing the lossless files you want to fingerprint. Press Add use Files of type to change the correct format. Select all the files you want to fingerprint and press Open

3. From the dropdown menu on the right, select md5 fp then hit Create



4. The checksums will take a few minutes to calculate.



5. Save your .st5 to the same folder.

c0437a74a414351fb1fb4e4cece99e06  [shntool]  gl2005-03-12t01.flac
1822d349a6be042a49af9bb0085e093c  [shntool]  gl2005-03-12t02.flac
5028343d2820679c23a99cc0e2f6e1f5  [shntool]  gl2005-03-12t03.flac
c485a617b3af2e4267753d334b709fa2  [shntool]  gl2005-03-12t04.flac
fd6e94f4c89685e8b41e4df2c073fb48  [shntool]  gl2005-03-12t05.flac
2c3d244df2a8cb4aab370e26e4f3f393  [shntool]  gl2005-03-12t06.flac

Creating a .ffp file:

Test the files and run a len check before you do this.

1. Go to Create checksum file > Browse

2. Select the folder containing the FLACs you want to fingerprint. Press Add use Files of type to change format to FLAC. Select all the files you want to fingerprint and press Open

3. From the dropdown menu on the right, select flac fp then hit Create



4. Since the ffp checksums were calculated during initial coding they can be very quickly extracted from the headers and put into a list. FLAC files should also be tested to make sure they can be decoded properly (see "Testing lossless files" above).



5. Save your .ffp to the same folder.



ffp is generated by FLAC itself. during encoding a checksum is generated from the decoded audio only for each file and stored in the header. the checksums are the same as a .st5 (aka SHNtool md5) just the formatting of the .ffp is slightly different.

gl2005-03-12t01.flac:c0437a74a414351fb1fb4e4cece99e06
gl2005-03-12t02.flac:1822d349a6be042a49af9bb0085e093c
gl2005-03-12t03.flac:5028343d2820679c23a99cc0e2f6e1f5
gl2005-03-12t04.flac:c485a617b3af2e4267753d334b709fa2
gl2005-03-12t05.flac:fd6e94f4c89685e8b41e4df2c073fb48
gl2005-03-12t06.flac:2c3d244df2a8cb4aab370e26e4f3f393

Creating a .md5 file for a DVD:

.md5 checksums are only used for video at TTD.



1. Go to Create checksum file tab.

2. Press Browse. Select the folder containing the VIDEO_TS folder of the DVD you want to create checksums for. It is best not to select the VIDEO_TS folder, but instead the folder which contains the artwork, info and VIDEO_TS folder.

3. Press Add then use Files of type to change format to all. Double-click on the VIDEO_TS folder, select all the files inside that folder then press Open

4. From the dropdown menu on the right, select md5 sig (default).

5. Review that the files and folder are all correct, then hit Create.

6. Save your .md5 to the folder which contains the artwork (if you have), info and VIDEO_TS folder and not inside the VIDEO_TS folder. In the example, you would want to save to a folder called 1971.09.09 Hampton.

8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS/VIDEO_TS.BUP
8d50cba158b40a75dd32cada8964aef7 *VIDEO_TS/VIDEO_TS.IFO
e40bd45c55f783d519c2a5c9aa088e98 *VIDEO_TS/VIDEO_TS.VOB
e11aaf0c14501bbf15d776a34218c205 *VIDEO_TS/VTS_01_0.BUP
e11aaf0c14501bbf15d776a34218c205 *VIDEO_TS/VTS_01_0.IFO
da29301176f55de40d79be5cba560d06 *VIDEO_TS/VTS_01_0.VOB
99ba41ff0fec88e3def109b6cb824dcb *VIDEO_TS/VTS_01_1.VOB
6c20fa2d2232106310d61dce30cd3e29 *VIDEO_TS/VTS_01_2.VOB

[Five]

this is going to get put into the FAQ sometime soon.

please let me know if you find any type-o's/fuckups!

[U2Lynne]

A couple notes from the Checksums section (comments in bold):

.st5 aka SHNtool md5. this is a checksum taken from the decompressed audio only. the advantage to this is that it works with any lossless codec at any compression setting. This is the preferred checksum type to be included with shn or ape files.

.ffp aka FLAC fingerprints. These checksums are virtually identical to .st5. some differences are explained below, under "Verifying a checksum file" and "Creating a .ffp file".

For audio, I highly recommend using FLAC and including .st5 and .ffp no wholefile .md5 necessary. Except the 32-digit checksum is going to be exactly the same in the st5 and ffp file. There is really no reason to include both types of checksums for fingerprints, just one type.


Here is a quick tutorial on making shntool md5s on a Mac using xACT:

1. Locate files on your computer.


2. Open xACT to the checksum tab and drag the files into the window.


3. Hit the Checksum button.


4. Hit the shntool md5 button (verify is the default).


5. It prompts you to save it somewhere, find the same folder.


6. Name the md5 you are creating properly and hit Save (a more common extension these days is ".st5")


7. It makes the shntool md5s (it will take a couple of minutes).

[bubbybooshay]

nice job five

[AAR.oner]

[Five]

Originally Posted by U2Lynne

A couple notes from the Checksums section (comments in bold):

.st5 aka SHNtool md5. this is a checksum taken from the decompressed audio only. the advantage to this is that it works with any lossless codec at any compression setting. This is the preferred checksum type to be included with shn or ape files.

.ffp aka FLAC fingerprints. These checksums are virtually identical to .st5. some differences are explained below, under "Verifying a checksum file" and "Creating a .ffp file".

For audio, I highly recommend using FLAC and including .st5 and .ffp no wholefile .md5 necessary. Except the 32-digit checksum is going to be exactly the same in the st5 and ffp file. There is really no reason to include both types of checksums for fingerprints, just one type.

thanks

I added in "This is the preferred checksum type to be included with FLAC, SHN or APE files." in the st5 section.

The main reason is that when you double-click a .ffp for testing using TLH, it extracts the checksums hidden in each FLAC file's header and compares that to what is in the .ffp file to see if it matches (takes about three seconds). Using the .ffp this way for testing does not test the audio content whatsoever (which could be corrupt after a successful .ffp verification). A second stage of testing is necessary in this case, it is "test" mode, which decompresses the audio to a temp file and generates a fresh hash from this and compares that to the the checksum hidden in the header. So, only after the two-stage test the files are verified. Double-clicking on a .st5 for verification using TLH results in the audio content of the files to be tested being decompressed to a temp file (temporarily) while a fresh hash is generated and compared to the corresponding checksums contained in the .st5 file. So the .st5 test is only one step and verifies both (a) that the files will decompress properly (as with "test" mode) and (b) that the checksums contained in the .st5 file match hashes freshly generated from that decompressed audio. When a .st5 verifies 100% okay everything is tested except the header isn't looked at, thus the .ffp. The .ffp is admittedly a little superfluous, but since it is only 1k and takes about three seconds to generate and verify its not really much to ask.

oww, my brain!

[AAR.oner]

so, to burn a data disc that is standalone compatable using the EXTRAS_TS method, would select your disc type as DVD-ROM (ISO), DVD-ROM (UDF), or DVD-ROM (ISO/UDF)?

[...and the answer is: DVD-ROM(UDF) ]

[GregY]

There's also the excellent md5check perl script (and compiled version) that's available here:

http://etree-scripts.sourceforge.net/

I installed cygwin on my Win XP machine and use scripts and command line stuff to do all my lossless operations. Much cleaner than dealing with GUIs for me.

[Five]

I never could get md5check running for some reason, but have heard it is really great, you just need a bit more of a grasp of commandline than I have! If somebody wants to post instructions for that it would be appreciated.

Originally Posted by AAR.oner

so, to burn a data disc that is standalone compatable using the EXTRAS_TS method, would select your disc type as DVD-ROM (ISO), DVD-ROM (UDF), or DVD-ROM (ISO/UDF)?

Just use the default that Nero selects when you boot up, I believe that's "DVD-ROM (ISO)", the setting you would use for burning normal non-video stuff. It works great with my DVD player and it is sweet being able to get all the info & artwork right off the same playable disc. Also great to just double-click the md5 there and the DVD self-tests.

[U2Lynne]

On a Mac, when using Toast, if I want to add an EXTRAS_TS folder, I go to Data > DVD-ROM (UDF).

[Five]

ack... okay, maybe its DVD-ROM (UDF) on PC, too. it is the default when you start up Nero. My DVD burner is on my other computer, I can check it later and post back.

[Five]

hey

it is DVD-ROM (ISO) mode in Nero that I am using for the EXTRAS_TS method... works great on comp and standalone

[AAR.oner]

Originally Posted by Five

hey

it is DVD-ROM (ISO) mode in Nero that I am using for the EXTRAS_TS method... works great on comp and standalone

cheers! i'll give that one a try then Five

[karmakat]

Good job Five. About six months too late (for my personal use)...:run:

[mike1061]

Great job Jamie.
Mike