Yep, that’s what @Thiousi wrote above. The code doesn’t check if the file actually exists.
You can use the following code to do that:
<?php if($favicon = $site->favicon()->toFile()): ?>
<link rel="icon" href="<?= $favicon->url() ?>" type="image/png">
<?php endif ?>