The time I witnessed Michael Tilson doing just that, stopping mid-performance and not only turning back and scream at the offender, but also hitting loudly his music paper with the baton, I felt so embarrassed, even if it wasn't me. It just occurred to me that musicians think of just two sides, their side and our side, the audience side...so just for the fact of being on the other side of the stage I somehow felt as a part of the offense. It was horrible, and after that the performance just wasn't the same...some of the magic was broken.
You are right. It is hard to get back into "the zone" when there is a disturbance even for audience members. I think I am a little A.D.D. so jangling bracelets or candy wrappers distract me and make it hard to enjoy a performance, especially when people draw out unwrapping their candy thinking they are making less noise by unwrapping it slowly. Instead, they are making noise for a much longer period. I would rather they do it as fast as they can. Things that are unavoidable like coughing do not bother me as much, but things that people could have prepared for (turned off phones, unwrapped candy and popped it in ahead of time, etc) really distract me. Before Rheingold (2 hours 45 minutes with no intermission) or first act of Götterdämmerung (2 hours) I unwrap cough drops and wrap them in a handkerchief so I can access them noiselessly during the performance if my throat gets dry!
Years ago there was a commercial where a Brünnhilde-like opera singer was singing, and a cell phone went off, so she threw the spear and hit the phone with it! That made me laugh! I am sure it is very distracting for performers, and I do think the conductors get so sick of it.