Indeed, I used this one as a case study for a plugin's custom panel view (and it worked!). You do not need to add anything to a blueprint as with fields. Only I could not yet find a way to add a button to the dropdown menu and hence need to rely on widgets as buttons/ways to the panel view. I used it for the new config panel view of the panelBar.
There would definitely be the need for something like this. Calling
$site->index()->files() gets slow quite easily, with much less pages and files then you mention.
Reasonable ideas, in my view. Good thing with this solution: you get your custom panel view, which really is like a blank slate in the panel. You can do pretty much anything you want. Include custom CSS, JS....
Given my limited time, I am not sure if I will manage to work on any of these features. Priority lies on my plugins. But of course I would be happy to see PRs. I think this proof of concept shows in a quite simple way how to create and use such a custom panel view (workaround, since it is not really easily supported, but this workaround works pretty well so far).