You've probably heard it before; I've heard it everywhere. Sometimes you even see it in the rulebook. Some kind of rules or suggestion that all talk at the table is IC. Or to limit OoC talk. Or advice to put a lot of character into you voice when talking IC. The point is, there is a problem here.
Role-playing game mode clash is something like putting A SHOUTED WORD in all capitals and then forgetting the 'caps lock.' Without knowing, everything from there on IS SHOUTING IN CHAT. Are we asking inexperienced gamers to subconsciously know whether someone else is speaking IC or not?
Experienced gamers (or people very familiar with the people they play with) will not have a problem here. Knowing how the group (or the familiar person) speaks when in or out of character makes the process seem effortless. So much so, I'm not sure I've seen much written about it. From games I've read, there are a lot of unformed ideas on how to handle this.
One solution (a rather awkward one) would be like only using the 'shift' key. (My wife has permanently removed her 'caps lock,' by the way.) Let's say you have everyone speaking OoC, raise their hand. Everyone knows it's OoC without even thinking. I've even seen one game that gave hand signals for use in place of all OoC communication.
I haven't really got any fresh or newsworthy ideas what can be done, but just for a moment, consider this a problem of mode clash. I know some people who play in ways they describe as 'immersive' (I make no claim to understand the term; I'm quoting) have a huge problem with mode clash. I've heard it said that it would be like having a bucket of cold water dumped on you in your sleep. Since this is the way they have fun, it is really very unfortunate.
But even typical gamers suffer from this problem. Have you ever seen someone complain that they were 'thinking out loud' and not designating what their chara was about to do? Mode clash.
Now there are still other typical modes I haven't even mentioned (or know about; that's why I like talking to you). For example, a mode where everyone is quiet while the gamemaster sets the scene or when the gamemaster communicates secret information to one player (ever see someone 'taken aside?') or note passing? These are all modes that are rarely even mentioned, yet remain a part of gaming culture.
Hopefully looking into this may lead to more humane role-playing game rules (rituals). I would very much appreciate your sharing any other modes you can think of. Likewise, let's hear ways I haven't mentioned for dealing with modes and mode switching. You guys never fail to come through, I can't wait to hear your thoughts.