This is the crux of a disagreement in my group. When you end your turn in the outer worlds don't you return to the outside realms? So the curse automatically ends at the end of your turn because you end up in the outside realms? Please clarify.
When you end your turn in the outside realm, you go back to the tile you entered from. To remove any effects of a card you collected, you need to go outside the house. Upper floor, ground floor and basement regions are not outside.
That's how we have played it. If not, there is no penalty as you would lose the card as soon as you returned.