FLAC, SHN, Conversions, etc...A virtual potpourri [Archive]

TheMamba

2004-11-22, 08:20 PM

OK....so I have been downloading and listening to shows for a handful of months now and want to start exploring the more technical side of everything that is being discussed here and was discussed at STG....so here goes.

I know very little about FLAC frontend or how to use it properly. Is there a general How To Guide / Manual somewhere that I could read? I have some questions now....and the answers will only spur more questions. If there's a manual out there somewhere, it will at least give me a starting spot...

SHN vs. FLAC....I've read that FLAC seems to be the "better" of the two. For me, it's simply because I have a FLAC plug in for Winamp, I prefer FLAC (I just stumbled across foobar2000 so that may change...). Is it a bad thing for the qualilty of the data to convert from SHN to FLAC? i.e. if I wanted to take all of my SHN shows and convert them to FLAC, would I just be shooting myself in the foot and end up with un-tradable stuff?

If it is acceptable to convert from SHN to FLAC, how should I go about doing this?

I have been using FLAC frontend to test and decode FLAC files (and multi front end for SHN). I'll skip all of the Replaygain, and SBE questions as I saw some of that floating around here and in the FAQ.

The Tag button looks like a gem though. How the hell do I use it though? I have tried putting in values and then decoding to WAV, but it doesn't seem to work. I have an iPod and use iTunes quite a bit (WinXP). When I drag over stuff, I hate having to manually enter in the Artist, Album, etc for shows that I have downloaded.

I just took 4 Tool FLAC files for the hell of it and dragged them into FLAC frontend. Clicked Tag Conf. Auto Detect checked. Filled out Artist, Album, year, etc. Converted to WAV. Open up iTunes to play them and none of the album, artist, etc. shows up in my playlist / library??????

Any help will be much appreciated. :tunes:

Gizby

2004-11-22, 08:41 PM

If it is acceptable to convert from SHN to FLAC, how should I go about doing this?
This much I can tell you not to do. While it is not going to damage or alter the audio (at least it shouldn't), it is generally frowned upon and I believe such conversions are not allowed on this site. It also makes for a somewhat confusing lineage, leaving people to guess exactly why someone would do such a thing.

RainDawg

2004-11-23, 06:45 AM

Yes, I can help....you're asking quite a few questions here, and there's a LOT of information that you'll need to understand before moving on. The good thing is that you've already discovered the answer to the question, but likely don't reaize it yet!

The answer, as usual, is foobar2000. It can do direct conversions from SHN > FLAC and also do something called BitVerify, where it will confirm that the FLAC you created is acoustically identical to the SHN file it was sourced from: THIS IS A VERY IMPORTANT STEP FOR SHN > FLAC CONVERSION!

foobar2000 is also the answer to all your tagging needs as well, but let me discuss that in another post....

As for converting, it is acceptable and encourage to convert SHN to FLAC, as the former is an outdated and obsolete codec. FLAC has numerous advantages, and you can read a little more about the major differences in the FAQ's audio types and descriptions entry:
http://www.thetradersden.org/forums/faq.php?faq=audio#faq_filetypes

Now, foobar2000 is the complete solution to PLAYING your files, but you may not realize that it comes with an enormous amount of plugins that help you do everything else with those files too. First, make sure you've got the Special Installer version 0.8.3 (the most complete and up to date version) from the foobar2000 blogiste (foobar2000.blogsite.org).

When you install this program, make sure you include all of the plugins (some may not be useful to you, but they are all fun to play with and learn what options are out there). This will make sure that you have the diskwriter plugin, which will allow you to do direct file conversions.

After that, it's quite simple. Drag a bunch of SHN files into the playlist window, select the, right click, and say "Convert". A window should pop-up asking you what to convert to. Pick FLAC and the compression level (all of them are lossless, but the higher compression values will take more time to perform). It will take several minutes to convert. Afterwards, clear the playlist, and then drag all of the original SHN files and the new FLAC files into the playlist window. Select all of both formats, right click, and say "BitCompare". It will then run through and check that for each SHN file, there FLAC file of the same name is acoustically identical. Assuming everthing went well, it'll report that the files are identical, and you've done it correctly!

Always include in your lineage how you did to the files what you did. The above example would include:

SHN > foobar2000 0.8.3 (convered and bitverified identical) > FLAC

Note: this will NOT look at, change, or correct any sector boundary issues. If you have SBEs that you need to correct, this needs to be done with shntool. If you run a conversion with FLAC Frontend, there IS an option to correct SBEs, but if the files are in the wrong order in the original window, you're going to have screw up the entire set.

OK...I'll make another post for tagging....

RainDawg

2004-11-23, 06:52 AM

foobar2000's special installer also includes a plugin called masstagger, which is incredibly useful. First of all, it will know based on the file type you are trying to tage what the best type of file tag to use is (there are APE, Vorbis, id3, and id3v2 tags out there). For FLAC, it will use Vorbis tags.

Again, select a list of files, right click, and run masstagger. Take some time and read through all of the ways you can edit the tags. Then save a script that you'll use often so that you can bring it up every time you want to tag a set. I always batch set Artist, Album, and Date. I then have a input block (it allows one line for each file, sequencially) where I copy/paste the track titles from the info file, and it automatically tags each file with the right song title.

All in all, tagging a set of FLAC files after download takes me a total of 15 seconds with masstagger.

Bottom line: Get foobar2000, get used to it, get over the utilitarian design, and start delving into the many features it has. Make sure you use the ColumnsUI graphical interface, and then go ahead and setup your own formatting scripts. A little bit of time spent learning how to configure this awesome program will be rewarded with years of being far more efficient with your lossless filetrading endeavors....

RainDawg

2004-11-23, 06:54 AM

So...you've got about 2-3 hours of stuff to play with here. Don't get frustrated, once you work all this out you'll be very happy with the results. It's not hard, it just takes some time to get used to it all.

If you need any further help with foobar2000, PM me or make a new thread and someone will give you a hand.

TheMamba

2004-11-23, 11:26 AM

RainDawg,

Thanks a lot for enlightening me a little bit. This should give me plenty to chew on for the next couple of days.

If I run into any questions, will definitely take you up on the PM / new post offer. :thumbsup

Thanks!

Greengoat

2004-11-23, 11:51 AM

www.dbpoweramp.com/dmc.htm also works for converting various formats. If you get the powerpack you can edit tags. I don't since I never have enough time to enter all the tag info by hand.

RainDawg

2004-11-23, 12:07 PM

Goat,
I used dbPowerAmp for a while, but had some strange things happen to files that I couldn't explain. I can't say for sure that what was the culprit, but I found some files that had short sections of the end cut off (fractions of a second) so that the tracks didn't flow seamlessly.

I also did some direct SHN > FLAC conversions where a large spike was inserted at the beginning of each of the FLAC files, cause a loud and obnoxious click at the track transition....these flaws would have cause the files to fail a bitverification, btu dbPowerAmp doesn't have that feature.

I don't recommend this program, and if you do use it, I'd offer the opinion that you should use foobar or shntool and do a bitverification (or comparison for shntool) to ensure that your files are indeed acoustically identical. I runined a few sets by not doing so, and regret it now....

Gizby

2004-11-23, 02:07 PM

Well then, disregard what I said above. I clearly have no idea what I am talking about anymore.

dorrcoq

2004-11-23, 02:22 PM

does foobar2000 support APE files, and SHN or FLAC > APE conversions?

RainDawg

2004-11-23, 03:20 PM

does foobar2000 support APE files, and SHN or FLAC > APE conversions?

Yup...interestingly enough, it allows SHN > FLAC, SHN > APE, but does include an SHN encoder with the Special Installer. So, you can convert from SHN to anything, but not back into SHN without installing an outside plugin. Methinks someone has wised up to the fact that SHN is an outdated codec, and is gently prodding people into looking at other options.

Another nice option is that it can take your FLAC or APE files and compress them to MP3 or OGG...I used this for storing favorite shows on my hard drive so I don't have to dig through DVDs to get at them. Transfers the tags over from the original FLAC/APE/SHN as well, so no typing is needed.

RainDawg

2004-11-23, 03:24 PM

This much I can tell you not to do. While it is not going to damage or alter the audio (at least it shouldn't), it is generally frowned upon and I believe such conversions are not allowed on this site. It also makes for a somewhat confusing lineage, leaving people to guess exactly why someone would do such a thing.
This site is concerned with maintaining audio integrity, not with maintain large archives of outdated codecs. The belief that people can learn to convert to the more useful FLAC format and doublecheck their work is something new in the trading commuity, and an ideal we wanted to base our site on. There are some people who would have liked to make this an all-FLAC site, but a degree of prudence prevailed and we decided to simply suggest it, but not make it mandatory.

In order to keep things verifiable, we have also asked that all seeds contain shntool md5s and/or FLAC fingerprints (identical) so that the checksum becomes format-independant. Once there is a fingerprint of the audio archived, anyone can convert to any current or future lossless codec, and the fingerprint will remain identical.

Yes...we're looking towards the future and trying not to get stuck in the past....

Gizby

2004-11-23, 04:01 PM

Sorry for offering the premature advice. I've just been used to the idea of keeping shows in their original formats. But I'm glad that we are all fortunate enough to be on such a progressive site with people knowledgable and helpful enough to maintain such a level of quality. I certainly wouldn't have heard anywhere else about using the shntool checksum in place of others. And I never was a big fan of SHNs.

Also, thanks for the foobar reference. It's the first media player I've used (besides EAC) that has gapless CD playback.

Greengoat

2004-11-23, 04:32 PM

Never tried winamp and playing a cd in gapless. It works with files when they are enqueued using the gapless option.

I've never heard anyproblems on any of the cds I listen to, and I use db to convert everything except mkw files. I've converted almost exclusively with it over the past two years(yep, even in the mp3 days) and never had a problem. All the lossless concerts that I've converted, burned to disc, sounded just fine.

I'll decode a file that came with a wav md5 checksum. I realize this is not the audio, but it's a start.

Greengoat

2004-11-23, 05:06 PM

Just double checked one show.

Steve Earle
2001-06-13
"The Connection"
WBUR Studios
Boston, Massachusetts

Originally broadcast on NPR

Lineage: FM > ? > CDR > EAC > Wav > FLAC-Level 8

Personnel:

Jackie Lyden: Host
Steve Earle: Guest

Size: 118MB

Running Time: 47:51

Notes:

This is a single track wav. The interview features Steve Earle reading passages from his book of short stories titled "Doghouse Roses", discussing songwriting, activism, the death penalty, taking phone calls and answering questions from listeners. All in all this is an excellent addition to any Earle fan's collection.

The following text was taken from http://archives.theconnection.org/archive/2001/06/0613b.shtml

--------------------------------------------------

It came with a wav md5, I ran it and it cleared 100%. Next it came with an shntool file, the numbers matched up exactly. This is the shtool text that came with the show.

length expanded size cdr WAVE problems filename
47:52.34 506700812 --- -- ----- se2001-06-13npr.wav
47:52.34 506700812 B (total for 1 file, 1.0000 overall compression ratio)

------------------------------------------------------------------
Next I will attempt to convert back to audio and run the md5 and check the flac finger prints(ffp)....

Greengoat

2004-11-23, 05:53 PM

Ok ran db and converted back down to flac. It failed the md5. So I took a look at the original flac file and the new one. They were different sizes. So I looked at the compression ratio, and the original was 4:1, while the new one was 3:1. Unfortunatly I had db set on "High" compression for flac. It only has low, medium and high settings, no 1-8. But the new file was only a little over 2megs larger, not bad for coming down from a 494mb wav file.

I was going to try and use flac frontend to encode it at 8, to see if that would bring the file back down to 121megs, but then remembered the ffp. Aslong as the new file passes the fingerprints, it is fine. So I ran the new file through flac frontend and they were the same.

Here is the output

se2001-06-13npr.flac:b6a265525c1e33ca70fac17d26ef7c10

RainDawg

2004-11-23, 07:39 PM

Well, the problems I experienced with dbPowerAmp were with files done in a large batch, not a single file at a time. I'm not sure what caused it to freak out, but it could have been sector boundary errors, which this file does not contain according to the shntool output you posted. They could have also addressed issues with it in recent versions, it's been a while since I've used it, and I knew a lot less about this stuff than I did before.

Maybe you could do a similar test with a set that is NOT SB aligned, and see what the results are?

Again, I'm not sure what the problem with that software was, and back then I really didn't have the knowledge to test it...I just know that somehow the files got screwed up after using it (it could have been user error, of course ;))

JayDub

2004-11-23, 09:21 PM

Is there any way to change the compression level (from 5) when converting from .wav to .flac?

I just installed foobar2000, and like you mentioned above, my only complaint is aesthetic... I like the way winamp is on my eyes much more than the way foobar2000 is, but heck, if you say it's better, then I believe it is, so I'm going to give it a shot.

Also, I meant to ask earlier... can you give a link to show me how to do a verified rip? I thought I saw it around sometime, but I can't find it now. I set EAC up all proper, and I want to make sure. :thumbsup

RainDawg

2004-11-23, 09:36 PM

Sure thing JD....

This is the FAQ entry:
http://www.thetradersden.org/forums/faq.php?faq=ripping#faq_eac

which just gives a little blurb about verified extraction, but mostly leads you to the coaster factory:
http://users.pandora.be/satcp/eac07.htm

The coaster factory contains a plethora of information, and I recommend reading through the other parts of that site while you have time.

Once again, if you need anything, just ask.

Greengoat

2004-11-23, 09:57 PM

Two things, one, yeah it could have been a problem with earlier versions of the app. Ant two, now I need to go and find one that isn't sector aligned. But I have the feeling over the 1 1/2 years that I ran across at least one of those.

RainDawg

2004-11-24, 12:48 PM

But I have the feeling over the 1 1/2 years that I ran across at least one of those.
Uh-huh...I'd say more than one. Let me know how it goes, I am very intested to hear how this software test works out as I've shunned it long ago, but never did a proper test on it myself to confirm my suspicions ;). Thanks Goat.

Greengoat

2004-11-24, 02:25 PM

Leaving town tonight for the weekend. Never got around to finishing the test. I looked through a couple shows but saw nothing wrong. 2 down, hundreds more to go.

Is there an easier way to see if the sectors boundries are offset? I know take a look at my out of date list and let me know if there are any that you know had problems with in the past. http://db.etree.org/greengoat

Thanks, I really like using db, the right click option is just so handy. But I want to make sure it's doing what it's supposed to do and not damaging the files.

RainDawg

2004-11-24, 03:17 PM

OK...I just ran downloaded dbPowerAmp and ran it through a battery of tests. I made up a sample batch of .wav files, some with SBEs and some without. I then compressed to SHN and FLAC, and both outputted identical shntool md5s as the original WAV. I then compressed the SHNs directly to FLAC in batch, and back. Again, identical.

I honestly can't figure out where I got those errors I experienced before, but I am content right to say that dbPowerAmp appears up to the job.

I still recommend not deleting the input files until you run a BitVerify with foobar2000 or shntool -comp check to ensure that the result files are acoustically identical to the input files. ALWAYS double check your work. I cant stress this point enough..

I would continue to use dbPowerAmp if you like it, and if you run a comparison of the audio fingerprint on both files and it results the same, you know your files are the same and can move on an delete the originals.

I know I experienced some flaky behavior with this a while back, but I honestly can't repeat it and will say for now that I think this is sound software. If you ever do experience a changed fingerprint as the result of dbPowerAmp, please let me know and we'll try to pinpoint exactly how and why is messed up. If not, I'll assume I screwed up somehow back then and leave it at that...

RainDawg

2004-11-24, 03:21 PM

PS I just did a meansurement of the dbPowerAmp FLAC conversion, and it takes about twice as long to convert a set from WAV > FLAC using "high" as it does to convert using the command-line codec on "8". Whatever db uses as it's graphical frontend must be inserting some very inefficient overhead into the process. This isn't really a problem, just a bit of an annoyance....as long as the files are the same, it doesn't matter to me how long it takes to convert.

SHN > FLAC direct conersion takes a LOT longer than the same thing with foobar2000. I still like the easy right-click features db has, so will probably use it anyway,

TheMamba

2004-11-24, 06:44 PM

Is there any way to change the compression level (from 5) when converting from .wav to .flac?

Poked around a bit and figured it out I think....

With foobar open, click on the Foobar2000 drop down and select Preferences.

Select Diskwriter under the components heading.

On the right-hand side there is a slot called Output presets. In that area, click on the Add New button. Select FLAC from the dropdown. Hit OK and then change the compression level to whatever you want. The only thing I'm unsure of is the Add Padding value, so I left it at the default #.

Now you've created a new preset.

RainDawg

2004-11-24, 06:50 PM

The only thing I'm unsure of is the Add Padding value, so I left it at the default #.
The padding is a certain amount of space left at the beginning of the file, before the audio portion, for adding tags (aka Vorbis comments). You can force this to be something other than the defauly 4096, but it's not recommended. What creating this default space does is allow people to add and edit tags without having to rewrite the entire file....

Unles you're putting a sick amount of information (pages of text for a single track) the 4096 should be plenty. Of couse, it's awfully rude to decrease this value as people who download your seeds WILL want the freedom to add whatever tags they choose.

So, yeah, leave it at the default.

tall_leaping

2004-11-25, 10:40 AM

thanks for this discussion guys, it has been informative. i am just getting used to foobar (altho i dont like the ColumnsUI, i like the tabbed playlists- maybe i need to play with it more), and now maybe i'll check out dbPA, since foobar is NOT that userfriendly...but i do like the SHN>FLAC direct option...

ah..decisions, decisions...

JayDub

2004-11-25, 10:52 AM

foobar2000 tends to crash my machine. :shrug:

Greengoat

2004-11-26, 10:18 AM

Thanks for checking into db Dawg. I did notice right away how fast flac frontend is on converting, and pretty much everything else. I really like flac frontend, made some ffp for some of the flac shows I have, just for my own reference. But is it okay to make ffp of older shows and include this fingerprint with the show? Just curious, since there is a slight possibility, as remote as it may be, that my flac files do not match the original. Or is flac so new that making a ffp now, is better than not making one at all, assuming the ffp just wasn't included in a burn sometime down the road from the original.

And :thumbsup to this discussion. GG still learning, and reading the other techno stuff as we go.

RainDawg

2004-11-26, 10:20 AM

foobar2000 tends to crash my machine.
:wtf: wow...never heard of that happening before. What are you doing that makes it crash? Usually, it's far more stable than most other packages.

RainDawg

2004-11-26, 10:34 AM

But is it okay to make ffp of older shows and include this fingerprint with the show? Just curious, since there is a slight possibility, as remote as it may be, that my flac files do not match the original. Or is flac so new that making a ffp now, is better than not making one at all, assuming the ffp just wasn't included in a burn sometime down the road from the original.

Yes, it's not only OK but it's required for FLAC seeds here to have an ffp for inclusion with the seed. Note that creating an ffp is just taking the checksum text that is contained within the FLAC's header....it is calculated at encoding, not when you create an ffp. So the only way this value would not match what is contained in the file is if the file itself is corrupted, which can be checked with Frontend's "Test" command.

It is very possible that the FLAC files you have are not identical to the original, but the only way this would happen is if someone took the originals, converted to wav, and screwed around with them before converting back to FLAC. If this is the case, it should be noted in the lineage that the fingerprints of the files you have are different than the original seed because the audio was changed.

The ideology here is that you get the accurate lineage info, and then make the ffp for posting on the site. That way, at any point in the future, someone can check back and say "yes, I have an identical version to what was posted to TTD on xxxx-xx-xx". If we are able to attain a reputation (as etree has done) as only hosting fingerrpints of good seeds, than this will be enough for future traders to trust the material they have as well as check information and verify they have an identical one.

Five

2004-11-26, 08:55 PM

Okay, here I am doing my 1st conversion using foobar. I had to tinker around a bit, first thing I did was make a new preset with compression level 8 and the recommended 4096 bytes of padding. This I did by going to foobar2000 > preferences > diskwriter > Output Presets > Add new Then I set it to keep the filename and folder the same as the source (see screenshot)...

Five

2004-11-26, 08:59 PM

Then I used this Bit-compare tracks function. I like what it does, but is there some way to make the output easier to read? I was squinting for five minutes trying to find the bits that say "No differences in decoded data found." I eventually copied the output to a notepad .txt file and used the search function while counting how many time I had tapped on the "Find next" button. For each file I'm getting an output like this:

INFO (foo_bitcompare) : Comparing:
INFO (foo_bitcompare) : location: "file://C:\Documents and Settings\Grace\Desktop\jamiestuff\Police - 1977 BBC Rock Hour\Track 10.shn" (0)
INFO (foo_bitcompare) : location: "file://C:\Documents and Settings\Grace\Desktop\jamiestuff\Police - 1977 BBC Rock Hour\Track 10.flac" (0)
INFO (foo_shn) : cannot find / read external metadata tag file
INFO (foo_shn) : internal seektable will be used.
INFO (foo_bitcompare) : No differences in decoded data found.
INFO (foo_bitcompare) : Finished successfully.

Five

2004-11-26, 09:00 PM

Bottom line: Get foobar2000, get used to it, get over the utilitarian design, and start delving into the many features it has. Make sure you use the ColumnsUI graphical interface, and then go ahead and setup your own formatting scripts. A little bit of time spent learning how to configure this awesome program will be rewarded with years of being far more efficient with your lossless filetrading endeavors....
*grumbles* yes, foobar is a useful tho not beautiful tool.

TheMamba

2004-11-26, 10:35 PM

*grumbles* yes, foobar is a useful tho not beautiful tool.

:lol:

spiritinaphoto

2004-11-27, 03:24 AM

I have the dBpowerAMP Music Converter on my computer, but I don't really use it much anymore, because the conversions are slow as hell with it. Only reason I keep it around is because it provides killer stats on holding your mouse over the file for all filetypes it has codecs for. Tell me it isn't cool to see this upon holding your mouse over one of your audio files:

http://img123.exs.cx/img123/9265/mouseover.png

Oh, if anyone wants to make foobar2000 look prettier, yet has problems using any of the GUI replacement plugins for foobar, I recommend installing a program like WindowBlinds or StyleXP to skin your OS, and pick some skin you like.

starman714

2004-11-27, 09:28 AM

I never knew foobar could do all of that , and a lot of 'that' is
well over my head still. I just d/l and installed it a few days ago
at someone else's recommendation... I just really wanted to say
thanks for the question and the in-depth answer ; I'm saving these
2 pages as a referance guide.

Five

2004-11-27, 09:59 AM

that is a great feature of dbpoweramp, I never knew about that! This is much better than the similar mp3infp (http://win32lab.com/fsw/mp3infp/index.html) which I've been using for the past couple years.

Yes, this thread has been quite educational. This will most likely show up somewhere at TTD as a slightly more organized tutorial in the near future.

RainDawg

2004-11-27, 10:58 AM

*grumbles* yes, foobar is a useful tho not beautiful tool.
Well, I agree with you that the output from BitVerify is not pretty...I was a little dissapointed when I saw that I head to read all that. The good thing, Fiver, is that it just means no one's been able to program it, so you can step and write a plugin to use BitVerify and output to a nice pretty window ;).

Seriously, people must understand that this is an open source, community built player that is not even to v1 yet. It's a little rough around the edges but the functionality is awesome, and once you get used to it, not so hard to use.

Five

2004-11-27, 11:06 AM

yeah, no arguments. I wish I could program a better window. I also realized that there's an option "beep on errors"... now all I've got to do is induce it to make an error and check that it does indeed beep then I can have peace of mind that no beep means perfect verify.

uhclem

2004-12-28, 12:01 PM

I have been using FLAC frontend to test and decode FLAC files (and multi front end for SHN). I'll skip all of the Replaygain, and SBE questions as I saw some of that floating around here and in the FAQ.
If you are already using Multi-Frontend for SHN then use it for FLAC too. There seems to be a common misconception out there that you need FLAC Frontend too, but you don't. FLAC Frontend is best if you are making FLAC files from WAV files as a seeder. If you are just a lumpen downloader of FLAC files, Multi-frontend is all you will ever need. You can also use it to convert from SHN to FLAC.

In order to test your FLAC files prior to conversion get MD5check available at my web page (http://home.ripway.com/2004-12/223394/files/).