The problem is indeed in composer.json. When linking the panel assets folder with the media folder, Kirby generates a hash of its version here, and that version is read from Kirby’s composer.json here.
No composer.json means no verson, no hash, and an error when initializing the panel assets.
I opened an issue on GitHub for this.