To get a clearer picture: Does this only happen when trying to log in the the Panel? Or also when doing stuff in the Panel.
If the login screen is already slow, then I would assume that this has nothing to do with the size of the site because I don’t think it loads anything at that time.
However, your network tab shows something that happens after login, but then again, this view doesn’t even load any images that could potentially slow things down.
To me that sounds like a problem with the server rather than something Kirby related, but I think that’s probably something we would have to look into more closely. Feel free to get in contact with email@example.com.