Tonight we fixed my friend’s network – see this post for history. It turns out that after changing IPs on the first server, the servers lost communication long enough for the password on one of the 2 of them to expire. This was causing replication to fail, which caused DNS to fail to update, and the 2nd DC stopped answering client requests.
Thanks to cyrilliano at Neowin Forums I was reminded of all the fun that is netdom.
I just had to run netdom resetpwd /server:dc2 /userd:FOOBAR\administrator /passwordd:admin_password
and wait 15 minutes. His post suggests disabling the KDC service, but I found it unneccessary. However, the 10 minute wait was not quite enough in my situation. Yet another reminder that patience is the first requirement for DC troubleshooting.
2018-01-24 at 12:15
I had to do this again today. Microsoft has a bigger writeup here: https://support.microsoft.com/en-us/help/325850/how-to-use-netdom-exe-to-reset-machine-account-passwords-of-a-windows