I played this for the first time a week ago and after building the Temple it demanded all other players give me 4 bucks or 4 prestige for every residential building they controlled, I rocketed so far ahead it wasn't even fair and for the last half of the game the other players just fought for 2nd place.
Did we play that right and if so how can that be countered?
This is one of those "I know the temple is in the deck, so I'd better be prepared for it and hold some money for each of my residential buildings in case it comes out." You almost always want some money late in the game in order to account for the possibility of a "Give cash or prestige to the guy who built this" building. As I recall, the temple isn't the only one.
Of course, this applies only to the guys who are holding heavy shares of a particular building type. If you've only got two residential buildings, the temple isn't as dangerous as if you have 5 of them.
Also, keep in mind that you pay for each building individually, not as a lump sum. So if you have 18 bucks and 5 residential buildings, you can give the temple founder 16 bucks (accounting for 4 of your 5 buildings) and 4 prestige for the final building. You almost always want to give cash instead of prestige unless holding the cash will clearly give you a bigger advantage over the field/person building the temple. Remember also that it's not just four points...it's actually an eight point swing to the temple founder...you lose 4 and he gains it.