Yeah I wasn't worried about it at the time. But it was also a chance to teach a lesson to my team about protecting everyone on the field.I had the same reaction and was going to talk about the value of punctuation!
While I appreciate that as a ump, it would precarious if the other team pushed the issue later in the game (to force a pitcher out or to prevent an offensive conference later in the inning). If you are going to do that, try to get the BU to come over to verify you aren’t doing any coaching (kind of like an injury time out). Best practice is to have the catcher go out.