Author Topic: Using FLAC with large hard disks - audio glitch  (Read 7325 times)

0 Members and 1 Guest are viewing this topic.

Offline S80_UK

  • Global Moderator
  • Veteran.
  • *****
  • Posts: 392
  • Volvo S80 D5 with VW Phatbox
    • Volvo Phatbox Installation
Using FLAC with large hard disks - audio glitch
« on: January 12, 2006, 06:11:18 pm »
Has anyone had problems when using FLAC with large hard disks in the DMS?

I used to run my original 20 gig drive wilth about 50 CDs of music, all in FLAC, and had no problems.  I now have an 80 gig drive with about 210 CDs loaded (again, all but a couple of them use FLAC).  Every so often I was getting a short break in the audio just after the start of a new track.  I went back to the other drive to check everything still worked (it did).

After some digging around, I found that the cluster size on the 80 gig drive was 32k bytes instead of 16k as used on the smaller drive.  By changing the cluster size down to this smaller size, the problem went away - almost.  So I went further, and reduced the cluster size again down to 8k bytes.  The break in the audio did not happen any more.  Just to be certain, I reformated the PHTDTA partition and went back up to 32K and then to 16K and then back to 8K.  It was the same story. Only with 8K byte clusters was the problem not noticable.  

I can only imagine that the higher data requirements of the FLAC decoder are tripping up the buffering that is done by the software on the Phatbox when it tries to read other information from the disk just after it has started a new track.  

I don't have any low level knowledge of the software structure, so some of this is speculative, but it may that the box is reading a play list (perhaps to determine the details of the next track to play), or part of the FAT, or something else?  If it is something like the playlist then the large clusters would mean that it would read a lot of data from the disk when only very little is really needed for this task.

Why do I not see the problem on the smaller drive?  Maybe because of the 16K clusters, and maybe because of the simpler directory structure and smaller FAT tables (just a hunch).  

I suspect that this problem would normally only affect the FLAC decoder.  Other decoders run at half of the data rate or less (sometimes much less), and so the buffering that the Phatbox software does would more easily cover any interruption to flow of the music data.

Does anyone have any other ideas?  

Mine is a patched VW box running Volvo firmware.  Hard disk is a Western Digital Scorpio WD800UE - a 5400 rpm drive with 2MB cache.   (I don't think the problem is specificaly related to these facts, although it could just be I suppose).

Regards,

Les.




Offline SteveC

  • Moderator
  • Getting the hang of things.
  • *****
  • Posts: 79
Re: Using FLAC with large hard disks - audio glitc
« Reply #1 on: January 12, 2006, 07:22:31 pm »
The solution seems to be counter-intuitive to the problem.

The normal equation usually goes: Larger files get better performance with larger clusters. What happens is that it takes fewer read operations (round-trips to the drive) to read the file - more time is spent actually transferring the data, and less time is spent asking for it. It also (usually) helps reduce file fragmentation, in that each time you reposition the drive head, you get a substantial chunk of the file - at least 32K, as opposed to the possibility of having to reposition the heads every 8K.

Also, the cluster size has no bearing on the application's buffering of data. While the drive reads the full 32K, the application may have only asked for 10K of it (or 1 byte), and it just gets the data it asked for, not the full cluster (unless the application is working at the lowest level for data access - unlikely, but possible).

So under normal circumstances, 32K or even 16K clusters will be more performant for FLAC than 8K or 4K clusters. This is especially true with laptop drives where spindle speeds (latency) and head speeds (seek) are slower than "normal" drives. Since it takes longer to reposition the heads, and longer for the data to "arrive" at the heads as the drive spins, fewer seeks (larger clusters) are the optimal configuration.

However, there may be some odd issues due to smaller caches on the drives, and the way the Phatbox caches/reads ahead that conspire to make all of this moot. This could especially be true with FLAC, where less of the song fills the buffers, so that there is less tollerance for delays in getting data from the disk. If reading the "wasted" data in a larger cluster slows down reading the data from the next song, and all of this is combined with multitasking the player, display, etc., then I could theoretically see a problem with using large clusters.

IMO, if the solution works, then that's great, and good to know. But it's odd that the solution runs counter to what logic would suggest.
« Last Edit: January 12, 2006, 07:24:23 pm by SteveC »

Offline S80_UK

  • Global Moderator
  • Veteran.
  • *****
  • Posts: 392
  • Volvo S80 D5 with VW Phatbox
    • Volvo Phatbox Installation
Re: Using FLAC with large hard disks - audio glitc
« Reply #2 on: January 12, 2006, 08:09:17 pm »
Hi SteveC,

I agree with you this seems very strange.  I have in the past worked on low level disk hardware and software (embedded, not PC, not Linux) and couldn't really make much sense of this either.

I will keep monitoring, and report if I find anything else.

Les.

Offline judb

  • Administrator
  • Veteran.
  • *****
  • Posts: 1329
  • ph4t l3wtz
Re: Using FLAC with large hard disks - audio glitc
« Reply #3 on: January 12, 2006, 09:27:52 pm »
I wonder if the larger cluster size has an impact on the drive seeks / reads crossing physical boundries of the drive making two read operations a requirement and slowing down the transfer rate?  Although I would think this would only happen every N number of reads based off the cluster size in the file system and the physical disk geometry.  

I'd be curious to see if we can find a higher performance filesystem setup for the phatbox.  or perhaps there are issues with the FAT32 support in the kernel that cause slower performance with larger filesystems that a kernel update might help.... hmmm

Offline S80_UK

  • Global Moderator
  • Veteran.
  • *****
  • Posts: 392
  • Volvo S80 D5 with VW Phatbox
    • Volvo Phatbox Installation
Re: Using FLAC with large hard disks - audio glitc
« Reply #4 on: January 12, 2006, 11:10:36 pm »
That's an interesting idea.  The problem has never been completely consistent.  A track that glitched one time might be OK on another pass (even after a long time).  The various buffers on the drive and then in the box could mean that the problem would only show at odd times due to different boundary alignments.

Now whether a different filing system is the answer, I don't know.  My first thoughts were to wonder if this can be tackled from the other end, by rebuilding the applications to get better real-time responsiveness.  This was on the assumption that the current buffering scheme did not allow operation to be as close to real-time as audio demands.  But that's a big step. I read that Sam has been playing with the kernel I think.  Maybe that's the area to look first (http://forum.phathack.com/cgi-bin/yabb2/YaBB.pl?num=1135307646).

Sam - if you're watching - any ideas?

Les.


Offline SteveC

  • Moderator
  • Getting the hang of things.
  • *****
  • Posts: 79
Re: Using FLAC with large hard disks - audio glitc
« Reply #5 on: January 13, 2006, 02:02:19 am »
After more consideration, I really think that this is mostly due to the Phatbox's limited CPU power and RAM. Decoding FLAC requires the CPU to perform more operations than decoding MP3 (IIRC, MP3 decoding is done in hardware, while FLAC is decoded in software). Therefore the decoding of FLAC is impacted more by "maintenence" activities (retrieving the next block of data, garbage collection, headunit management, etc.).

Add in the fact that the FLAC data takes more RAM than an equivalent time period of MP3 data (prior to decoding - the RAM needed after decoding is the same, assuming that the output is buffered instead of being transferred directly to the DAC stage). This leads me to assume that either they have allocated larger buffers for FLAC decoding (impacting memory useage for other areas), or smaller time-intervals are in RAM at a time (less tollerance for delays due to other processes and drive access).

Either way, FLAC requires much more drive activity and buffer management than MP3, as well as more CPU power. Considering that Phatnoise has frequently admitted that the hardware is being taken to the ragged edge, I think you are just running into the limits of the hardware. The fact that it is glitching between songs, when it is doing the most work (reading tags, updating the HU, buffering, garbage collection from the last song, etc.) really points to this explanation.

Another issue not mentioned is the fact that your 80G drive is holding more than 4 times the music, and thus has a much larger set of databases to manage, which consume either more RAM or more disk access to read/search/etc.

I beleive that you will probably have to accept this as a fact of life.

Of course, none of this explains why smaller clusters _helps_ rather than hurts performance. That's still bass-ackwards.

EDIT: One thought occurs to me. What if they allocate read buffers based upon a fixed number of clusters, instead of some fixed size? This could explain the behavior of smaller clusters improving performance. Just a thought.
« Last Edit: January 13, 2006, 04:22:03 am by SteveC »

Offline S80_UK

  • Global Moderator
  • Veteran.
  • *****
  • Posts: 392
  • Volvo S80 D5 with VW Phatbox
    • Volvo Phatbox Installation
Re: Using FLAC with large hard disks - audio glitc
« Reply #6 on: January 13, 2006, 07:06:25 am »
Actually, there is no MP3 hardware decoder in the Phatbox - the MP3 is all software (IIRC, MP3 should run happily on the ARM7 core at the speed used in the PB).  And FLAC should require less math than MP3, WMA etc to decode, since it is basically a limited set of integer operations to perform the lossless decompression (same reason that FLAC encoding is very fast - about 3-4 times faster than MP3 on my PC).  However, everything else is true - the amount of data being crunched is a lot higher, and I think that is at the heart of the problem.  And certainly, as I implied in my original post, this could be simply a consequence of the bigger disk meaning that tables, etc also get bigger.  And the buffering allocation or strategy could be part of the problem, but who knows.

As you say, so long as my fix works for me,  I can live with it.  The reason I raised this topic was to find out if others were affected and to propose a possible solution.  It would be great to know more about this, but ony if this is a more widespread problem, I guess.

Les.
« Last Edit: January 13, 2006, 07:07:38 am by S80_UK »

Offline SteveC

  • Moderator
  • Getting the hang of things.
  • *****
  • Posts: 79
Re: Using FLAC with large hard disks - audio glitc
« Reply #7 on: January 13, 2006, 02:29:18 pm »
Quote
Actually, there is no MP3 hardware decoder in the Phatbox - the MP3 is all software (IIRC, MP3 should run happily on the ARM7 core at the speed used in the PB).

Could have sworn they had an hardware MP3 decoder in there. Must have gotten the specs mixed up with my Audiotron's. It's been quite a while since I have done much with either of them, other than just listen to music. Thanks for the correction.

Offline SteveC

  • Moderator
  • Getting the hang of things.
  • *****
  • Posts: 79
Re: Using FLAC with large hard disks - audio glitc
« Reply #8 on: January 13, 2006, 02:39:08 pm »
Hmm. For some reason I can't edit my last post to add this, so ...

Quote
And FLAC should require less math than MP3, WMA etc to decode, since it is basically a limited set of integer operations to perform the lossless decompression (same reason that FLAC encoding is very fast - about 3-4 times faster than MP3 on my PC).

My "more operations to decode FLAC" statement was based upon having to perform operations on more data. MP3 has to do funkier math, but FLAC (I think) does more (simpler) math. As to encoding speeds, FLAC doesn't have to worry about psycho-acoustical models, joint stereo and variable/target bitrates.

Offline amb

  • Getting the hang of things.
  • **
  • Posts: 50
Re: Using FLAC with large hard disks - audio glitc
« Reply #9 on: January 14, 2006, 03:58:31 am »
On a somewhat related note, almost all my music are in ogg format on the DMS.  On my original 20gb Toshiba as well as a 80gb Hitachi Travelstar I hacked in, I was experiencing fairly frequent premature playback termination problems (i.e., the song would simply stop, and then skip to the next track).  That Hitachi drive died recently and I replaced it with a Western Digital Scorpio WD800UE 80gb, and magically the problem essentially went away.  It did happen once in over 3 weeks of use rather than the almost daily occurrence previously.

So, I surmise that the problem probably has something to do with data underflow.  Also that ogg decoding is cpu-intensive probably made matters worse, and brought the phatbox to its limit somehow.  The new WD drive probably has better data buffering than the other two (even though the Hitachi was similarly spec'ed 5400rpm, 8mb buffer, etc).  The FAT32 filesystem cluster size is the same between the two 80gb drives so that wasn't a variant...

Anyway, so far I am quite happy with the WD drive and would recommend it highly.

Offline AvidAngler

  • Newbie
  • Posts: 8
  • In Over My Head!
Re: Using FLAC with large hard disks - audio glitc
« Reply #10 on: January 15, 2006, 06:14:00 pm »
Did you buy the WD800UE with the 2MB cache or the WD800VE with 8MB cache?  Is larger cache size a desirable feature in the DMS application?

I noticed Seagate offers a comparable drive to the WD800VE in their Momentus 5400.2 ST98823A.  It is also a 80GB unit with 8MB cache but has a 5 year warranty vs 3 years for the Western.  Pricing between the two brands seems to be fairly comparable at $120-$140.

Offline amb

  • Getting the hang of things.
  • **
  • Posts: 50
Re: Using FLAC with large hard disks - audio glitc
« Reply #11 on: January 15, 2006, 07:45:15 pm »
Quote
Did you buy the WD800UE with the 2MB cache or the WD800VE with 8MB cache?
Now that you ask I am not 100% sure.  My memory tells me it's the UE but I'll have to open the DMS up to check.

Quote
Is larger cache size a desirable feature in the DMS application?
I'll let others who are more knowledgeable than I about this topic answer.

I remember some years ago, for desktop/server hard drives there used to be a distinction between those designed for "data" use vs. those for "multimedia or AV".  The difference was that the multimedia versions don't do a lengthy "thermal recal" operation which might stall a streaming media transfer.  I don't see this mentioned much any more, but I wonder if this sort of thing still happens, whether they apply to recent laptop drives, and might it be a factor in some of the phatbox playback glitches we observe.