If my house burns down, must I rebuild it?
On most standard policies, effectively yes, because replacement cost is paid on condition that you replace. Some high-value policies allow a cash settlement of the dwelling limit without rebuilding. If having that choice matters to you, it has to be arranged before the loss rather than negotiated after it.
How is the limit set on a house with no comparables?
By a professional appraisal, ideally supporting an agreed value policy that fixes the figure in advance. That removes the post-loss argument about what the house was worth, which on a custom property is the argument most likely to happen.
What is a coinsurance penalty and should I have one?
It reduces payment on a partial loss when the limit is below a required percentage of replacement cost. High-value policies commonly dispense with it. On a standard form it can reduce an ordinary claim, so its absence is one of the specific things worth confirming here.