The size or number of files on the DMS should not make much difference. I have a 250GB DMS with 29,500 tracks in around 750 playlists. Voice browsing is just as fast now as it was when I was using the original 20GB DMS. Browsing by letter never takes more than a few seconds between letters. Only when there are 100s of tracks by a particular artist, it takes slightly longer to browse to that artist, but never more than 5 seconds or so.
Drive performance could slow down somewhat if you get close to capacity, so you might want to leave a few GB free. To be safe, check the DMS drive for errors.
I suggest using the options in PMM to fix the DMS database and regenerate the voice prompts. If you previously created prompts for track titles in addition to the usual fields, don't do it this time around, and see if the performance improves.