@dLj How we used it in the Navy too. "Over" is used why you are in an "active conversation" with one or more individuals, and are done speaking, normally expecting a response. If others are on the same frequency, unless it is a urgent or they are part of the conversation, they don't just cut in. It keeps people from trying to talk over one another.
"Out," as you say, means "I am done talking and no longer will remain in the active conversation. Hail me if you want to contact me." If you stay on the same channel is is customary to say "Out, Song of the Sea is monitoring Channel 9". If you are shifting to another channel, you might say "Out, Song of the Sea is changing to Channel 16", or whatever channel you are shifting to. This way they know you are leaving the conversation and if they want to talk to you again, they will have to hail you on the channel you are monitoring.
"Roger" means I have received your transmission and normally also is taken to mean I understand what you said (Not that I agree or not, just I understand.) If it is unclear you can say something like "Say again, or similar similar) If it is really important the Roger is followed by a repeat or summary of what you think you heard such as "Roger, understand you are changing course to Starboard and coming to course 265"
Crisp and clear communications are important especially in restricted waters or when failure to communicate clearly has ramifications. As an old an famous football coach once said "you play like you practice."
@jviss Not only do the Coasties, making the routine transmissions on 22A read from a script but they talk so fast that you can hardly understand them. It really bugs me. Why not slow down and speak clearly? Guess they have to get back to whatever else they were doing in a hurry.