The link mark emits the link event from the command. This event is then picked up by a global event handler in the Writer component. This event handler opens the dialog and triggers another ProseMirror command to update the mark with the result of the dialog.
Since you need a custom dialog, you cannot use the link event. Instead, you can copy the this.$panel.dialog.open() code to your mark implementation directly and adapt it to your needs.
If your marks only differ by the class name, you could alternatively also adapt your parseDOM rules so each mark only grabs the instances it should. The link mark is a good example for such custom selectors.