Actually, on second thought, since you mention it is your own plugin: if the <iframe> element used in your code to embed the video in the panel, you should be able to set a referrerpolicy="strict-origin-when-cross-origin" (or any other referrer policy, if that is still too tight) on the iframe element. That way, the loosened policy would only apply to the embedded source.
The W3C spec is not specific about precedence, but from a logical point of view, I’d assume that specificity rules apply as they do with similar things: i.e. server header gets overruled by meta tag, gets overruled by attribute on a element…