I have built a website with Kirby, I am wondering that as it is file based; What is to stop a user from just downloading my whole website? Is there any security or permissions I need to add to protect my website. Furthermore, is it possible for someone to just add a new user with a password and access my panel? Hoping it is safe was just looking for some information?
Hello,
What is to stop a user from just downloading my whole website?
Basically nothing.
If it can be viewed in a web browser then it can be downloaded:
Prevent my site from being copied.
Disclaimer: If I want to see how something was achieved these days I tend to just check source. However, in the past I’ve downloaded entire websites. Never to duplicate or steal though, just to learn. 
This isn’t a ‘weakness’ of Kirby, it’s just the nature of the web.
It’s been said many times before, but the only way to absolutely ensure the contents of your site are not downloaded or copied is to not have a website. 
However, anyone doing so will only really be able to grab your (client-side) textual content/images/media/fonts/JS libraries and generated HTML/CSS. Your core server-side (.php) files should be safe.
…is it possible for someone to just add a new user with a password and access my panel?
Not without having access to the panel in the first place, coupled with permission to set up a new user account.
However, you could choose to either not upload the panel to your host or generate static pages only and upload these. Either way, no panel.
It’s safe to say that your Kirby site is safe. 
I hope this helps.
Best
John