So if the loser has 3 castles, the winner swaps 1 for 1 thus placing 3 of his own castles?
To the best of my knowledge the answer is yes, you swap out 3 for 3 and yes this can be a seemingly abrupt shift in power.
Of course that abrupt shift is usually preceded by multiple turns of jockeying for position on a 3 part island so the apparent abruptness is a bit more illusory. It seems rare in my experience to have a complete shift in power on an island that big with zero turns of anticipatory build-up and manipulating of the majorities in different colors. So maybe you can view it not as ending the game suddenly, more of a pinched battle right to the end.