We carefully took out rhe pieces to avoid the building collapsing.
After several rounds of play, Andrew had an idea of creating Domino effect. He said "if I measure the blocks exactly 1 brick apart, then it will collapse neatly. "
So we tried that. It did.