In my experience (on K2 installations), having huge “file”-caches has negative impact when it gets flushed (e.g. when an editor saves a page in the panel) because the folder listing to delete all its contents takes a while. This degraded the UX in the panel quite a bit, so on some of my sites I decided to not use file-caching for that.
I think using memcached (or other caching tools such as apc, etc) have better performance for this; but they come with a cost (extra money; but also an extra level of complication - I couldn’t easily see what happened in my memcached database, nor debug it - but that could be due to my poor knowledge about them).
I’ve noticed K3 caches work in a different way now; they use subfolders which would probably speed up the file listings quite a bit by default. I haven’t tested this further though.