The short answer is that we did it. Five more sites up. So, um, six in total. The long answer is that it was a clusterfuck and, honestly, it required a lot more effort and time than it should have.

It seems that The Boss sent no information about our use of VLANs to the new provider. He also forgot to mention that every site (instead of just the main office and fail-over site) was going to be sending voip traffic. Which, naturally, meant that the provider hadn't planned to implement QoS at any of yesterday's sites.

I also discovered something about the MPLS setup which The Boss apparently misunderstood/communicated incorrectly. I have some serious doubts that we have enough pipe at the main office. The plus side is that if I'm right, we can jack up the bandwidth (and pay more, of course).

So there we are, The Consultant (we will call him Jacques) and I, sitting at our office in Perry. We convince the network engineer on the conference call that, yes, we need to route two VLANs. He implements that for us, but there is some confusion on his end about the VLAN capabilities of the router they installed (because they weren't expecting to support VLAN routing). The router was not handling the VLAN routing correctly so only the data side was working. With teams at two sites, we managed to get both up on the data network. The Boss left his site and headed for the third, since Jacques and I were handling all of the phone turn-ups.

As The Boss arrived at the third site, the network engineer pushed a new VLAN config to our Perry router without telling us, thinking that would solve everything. As Jacques and I were working on the SIP proxy at the time we didn't notice that we lost our data connection, and in fact could no longer reach the router on the LAN.

The network engineer, confident in his own abilities, mirrored this new configuration at the third site. When it didn't work, we all decided that it was an internal problem at that site. I went to reroute their traffic over their old connection and discovered that Perry was down. That's when the fight started.

The network engineer dug his heels in and would not accept the possibility that the problem could be in the router's config. He suddenly started insisting that our switch didn't support VLANs, which set Jacques off. What followed was a spirited debate about whether we should fix our switch or they should fix their config. Again, the engineer dug in his heels and refused to change the config, saying he wasn't going to go swapping configs on all of our routers (why don't we manage these ourselves?) just because we didn't want to accept that our equipment was bad.

Finally the project manager on their end told him to just do it. Apparently the engineer then decided to build a new VLAN config from scratch. He dropped it in without telling us, maybe he was trying to catch us on something, and suddenly our phones all registered and the data came back. Defeated, the engineer copied that config to the other two sites and we were up. They started working on QoS while we brought up site four.

Site five was a live cut. It's a newer site that had an existing Cisco system in place, so it was all-or-nothing. Since their equipment was all configured weeks ago, and we've refined things since then, we knew we had some configuration work to do before we cut. Still, Jacques and I were starving, so we stopped for some pizza (it was 5:15 and we'd worked through lunch). Jacques was so hungry, "eets macking me mad at people".

We get a few bites into our pizza and The Boss calls wondering where we were. Apparently he got there, figured he had things under control, and ordered the cut-over. He didn't understand why things didn't just work. When Jacques and I arrived it was worse than we thought. The switch had the wrong configuration (I guess Jacques had used it to test something knowing it would need to be reconfigured anyway) and had to be set up from scratch. Once that was done we were fine. We ran their analog lines into a gateway, tested, stress tested the new QoS setup on the routers, and went home.

...and that was my most boring diary ever.