Instead of trying to start all the nodes with dynamic peer addresses set, alwasy start one, then join the rest to this one. The SetPeers in the test may be causing leadership changes and sporadic failures.