. Or, it could have been a DOS attack. At any rate, it is good to have you back, Steph and Drew. I was starting to worry that you'd given up on us.

[Link]

The TLDR version of what happened: the data center that I have the servers at has a problem with their Cisco router and they refused to believe me.

What follows is the long version....

I've been renting rack space at this data center (DC) since 2017. Up until recently that time they have not had any maintenance windows that impacted customers with planned downtime. About 10 days ago I received a notification that they would be upgrading the power service. The racks are served with redundant power feeds labeled A-side and B-side. The DC indicated they would be shutting down the A power on Monday night around 12 am for up to 10 hours. Then the next night it would be the B side for a similar amount of time.

All of my servers have redundant power supplies but the networking equipment is single-source power. I knew that ADT would go offline at some point if I didn't install an automatic transfer switch for the networking gear in the rack ASAP. I ordered a CyberPower PDU24004 and installed it in the rack on Wed evening Feb 5th. The install involved me shutting off two network switches and the router that is my connection to the DC. The install went well, things were only offline for about 10 minutes on Wed. I verified everything was operational. On Thursday I went out of town to go visit family. By Thursday night pieces of ADT stopped being accessible from the internet. Yes, pieces. The website was still up but the image upload site was offline. The interviews site went offline separately. Eventually I lost remote access to everything on Saturday afternoon.

I had no idea what was going on. I assumed I damaged one of the two network switches which was causing different IP addresses on the network to be offline. I ordered replacement equipment which arrived on Monday and I got to the data center Monday night after driving back to Maine from Ohio. I arrived at the data center about 30 minutes after the planned A-side power shutdown started. All of the networking equipment in my rack was fully operational but the network link from the DC was down. No "link" indicator light on the router jack. So I realized very quickly the problems were not what I expected.

Needless to say it was a complete clusterfuck at the DC when they shut down the A-side power. All kinds of unintended consequences and surprises and dozens of irate customers on-site and I assume many more via the trouble ticket support system. Around 3:30 am I was able to get one of the facilities guys to get my link "operational" meaning the link light on the router was illuminated. However there was no data flowing over the network. I was instructed to open a trouble ticket case with the Network Operations Center (Noc). This is when the finger pointing and CYA mentality of a large organization dealing with a stressful situation kicked into high gear.

The NoC folks are not on-site. They are remote. When I opened my ticket they automatically lumped me into the power-related category even though my problems started days before the planned power outage. When I didn't get a response by 4:30 am I called them. I was told that they would not be looking into my problem until the power was fully restored at the DC and the maintenance was over. Initially they said they wouldn't help me until both the A-side and B-side maintenance windows were over which were days apart! Really an arrogant bunch of pricks.

At this point I emailed my account rep to let him know what was going on. Around 7:30 am on Tuesday he was able to elevate the case to ensure I wasn't waiting until after the B-side maintenance was completed.

When the A-side power came back just before 8am Tuesday morning I updated my case to let them know that the power was restored. About an hour later I received an update that all on-site staff had been at the DC all night and no one was available to help diagnose my situation. Those people were all around me all night and I spoke with several of them about my problems but they all took off as quickly as they could. I was told that they couldn't help me until Wednesday. At this point I was absolutely fuming.

A quick phone call with my account rep resulted in them finding someone that was underqualified to address my situation, i.e. not in their normal line of work. This guy connected his laptop to the network jack that provides the internet service for ADT and was able to successfully ping their Cisco router on the other end of the line. This "test" was what they used as a determination that the problem was on my end. I don't know what the guy did or how his laptop was configured but he was basically like "see, the network works. it's your problem now." I was completely confused by this. I spent the next 6 hours trying to figure out what could have possibly went wrong with my equipment. I eventually decided that I would go home and grab several different computers and a spare router and go back to the data center. (This is a 3 hour round-trip.) I wanted to be sure I wasn't using any gear that was in the rack and might have been mysteriously damaged in some way.

I get back to the DC around 8:00 with two laptops, a router and spare Compaq server. I configure the networking in each of these devices in the most basic dedicated IP address fashion attempting to replicate the so-called test that was performed earlier in the day. At no point was I able to get any data to pass over the network. I started to feel confident that the test was faulty and got on the phone with someone at the NoC again. This time the person was very helpful. We were on the phone until 11:30 pm. For some reason he was able to see any of the devices that I plugged into the network port. So the assumption was that the network was working and I was a moron. So I started snapping photos of the configs and sharing them. He verified everything I was doing was correct and was baffled that the circuit wasn't functional for me. He requested a network engineer be on-site at the first available time to investigate. I was told that I would be billed for the network engineer's time because as far as they could tell the circuit was functional and one of their employees allegedly tested it as functional. Of course they couldn't give me a time for when the network engineer would be on-site. Only that it would be "during business hours on Wednesday."

I arrived home from the data center around 1am Tuesday night/Wed morning. I slept a few hours and was back on the road by 6am to make the 1.5 hour drive to the DC. I was back at 7:30 so I could setup several proof-of-failure demos for the network engineer when they arrived. He arrived around 8:45. They sent out some low-end tech support guy. Barely more capable than the guy that installs cable modems in residential applications. It was obvious to me that they all thought I was full of shit and some low-level nerdling would be all that was necessary to point out my shortcomings.

I started showing him network packet captures for ARP who-has requests for the gateway IP address and not getting a response from the router it became clear to me that he was in over his head. He was just there to do what he knew how to do. Of course he asked me the typical "did you reboot your computer?" types of questions. I stayed calm. It wasn't easy. After all he was there to help me. We spent about 15 minutes going through basics like the network config I was using, IP address assignments, etc. He confirmed I was setup properly. Still managing to stay calm during this. So he decides to plug in his laptop into my circuit because he's confident that his equipment is known-good. Ooops. WTF? He doens't have internet access. Oh, yeah, his laptop has two network jacks built in. Must have plugged into the wrong port so he tries the other and it doesn't work either. At this point he realizes that I am right and there's indeed a fucking problem. However, he's not capable of determining what the problem is. He's not a network engineer. But at this point I am relieved because after almost 48 hours of butting heads with them they finally acknowledge that the issue isn't with my equipment and that I'm not a fucking moron.

He had to spend several hours on the phone with different levels of Network Operations and Network Engineering before they finally decided to switch my service to a different port on the router and ADT was back online. They closed the trouble ticket.

Around 8 hours later they re-opened the trouble ticket because apparently other customers started calling in about the exact same problem. It just took them longer to realize why they were offline. Most apparently assumed it was related to the power maintenance. It turns out that any customers connected to that Cisco router that had dropped their network link eventually lost their internet service. The root cause of the problem is the way the ARP cache is configured in the router. I'm guessing there's not enough memory allocated to the cache. ARP cache entries stayed valid as long as a link was active. Turn off your router and the link goes down and the cache entry goes stale and is purged. If there's not enough memory available in the router to add fresh ARP cache entries when the links come back up the Cisco router will not reply to "who-has" requests so the client's routers can't reach the gateway address.

Of course I shouldn't have been required to figure all this shit out for them.

Yes, I am shopping for new options.

[Link]

One would think the maxim that the customer is always right would have led your provider to get the tech guys in to prove otherwise.

So, yeah, that's an F on their report card, and I hope you can find a competent replacement ASAP.

[Link]

good luck with finding a better solution asap

[Link]

Jaysus! I read your long post. Have been in some similar situations over my career encountering networking issues which the networking support people didn't believe me and blamed it on anything else that was not their issue (servers, applications, anything but the routers, bridges, wiring, etc etc etc), so I sympathize with your ordeal. Luckily I had good relationships with some very sharp folks that I could then escalate things to get me away from the first level idiots and get things diagnosed correctly and fixed.

Again, thanks!

[Link]

I started to save my datas on a private server (with a second to save the datas from the first) back in 2019 after an attack on my cloud datas from my ex who unfortunately knew my password. It contains all my business datas but mainly private and porn vids and pics.

[Link]