No API access when user permission to panel is set to false

Yes, looks like you have to allow Panel access and then set everything else to false, compare this setup of the better-rest plugin: https://github.com/robinscholz/better-rest/blob/master/blueprints/users/betterrest.yml