based on my experience with scaling instances the scale was based on the group size at creation. Anybody that joins the group during the instance would not be allowed into the instance but would have to wait for the next round. If anybody left (LD'd) then everyone had to cover for them or restart the instance. As far as open world dungeons it might be a simple matter of spawning more mobs as mobs are needed. If everyone wipes then maybe they reset to a nominal number after a short term, or maybe they stay there in ambush until regular visits whittle the number down.