Okay so I have now solved this properly.
In my nginx config I was only using try_files
IN the root instead of FROM the root, like so:
location / {
autoindex off;
try_files $uri $uri/ /index.php?$uri&$args;
}
Of cause had to be this
location ~ / {
autoindex off;
try_files $uri $uri/ /index.php?$uri&$args;
}
also location /panel
should probably be location ~ /panel
Also see https://gist.github.com/bastianallgeier/c2e1f1e519f5f2943ec4#gistcomment-1795907