Latest Events
Info may appear to be old, it is current. Fiber is very reliable and the gear we deploy is commercial grade. Our network is large, however, lack of probelms to report is due to the fact that it is "all Fiber Ethernet GigE based." No old telco topology muxing or copper circuits.
These Graphs are LIVE ! Updated every 30 mins.
Simply, Click your browsers Reload/Refresh button to update.
GigE paths running BGP (To many to present all)
If the info has spelling and grammer errors, please remember that it's engineers that post it live, without any executive review.
Nov 1 10:15 AM City Power Hit on Ground to Building - Severe - Hit Core Routers - UPS system did not flip over fast enough due to this type of failure - Core Router lost power only long enough (ms timing) to hose it's OS. LA and southern sites not effected. Much time involved in checking things out before posting. All well, again by 12:35 PM. However, Large parts of Palo Alto around California Ave. are still without power.
Oct 31 7:30AM City power in South Palo Alto Restored. Customers in this region were online immediately.
Oct 31 6:30AM City Power in South Palo Alto areas near BlockBuster Video region. Only those areas are effected. UPSes on that path lasted 1 Hour. We are investigating.
Oct 2 2:32 AM City power is out in blocks of Palo Alto in the Area of Elcamino and California Ave. Those business have no power. It has been out for a while , so those UPSes that were in place have no battery left. When the city restores power , be sure to double check all the applications on your servers to make sure they are running before calling. Thank You.
Sept 26 2008 5:53PM Telecom Italia back online.
Sept 26 2008 1:18 AM Telecom Italia having trouble with BGP on the Seabone disrupting some Transit routes. Since we have so many Transit Paths we have shut this down until futher notice.
Aug 7 2008 It appears that the new CPU card is having problems with the OLD chassis - so the upgrade did not go as expected. PAIX main router is running on the spare older version, but not in dual mode for auto backup, in the rare event a failure occurs. The good news is that some of the changes had a psoitive effect...There are no longer errors occuring on the PAIX fiber paths. We are now formulating a plan to change the chassis. We have newer ones in stock. We will keep you updated here.
Aug 7 2008 PAIX Palo Alto "Emergency", we have a major failure of a cpu card in a main router - The secondary has taking over, however, this card level is an older version and we see errors. We have newer version rev level cards in stock, but, they do not run the older software version running at PAIX. Therefore BOTH Master and Slave must be changed together. We expect upto 5 mins of outage. This will cause a BGP Flap of all sessions at PAIX. Many customers may feel nothing. While others will feel an outage as related to thier Private Circuit at PAIX and BGP flap changes of Transit routes.It will take place from 12:30 to 1:30 today. We have scripted this emergency repair. This should be short lived - 5 mins expected. Although there are no custoemr complaints, do to current error level it urgent to complete ASAP.
California Ave "Area Customers Only"- Lost City Power for several hours. City power was restored 30 mins "after" the UPSes lost battery. ONLY California Ave Area Custoemrs were effected that had no power to their buildings - all others were not effected at all as traffic by passed these routers. Once city power was restored routers immediately re-connected automatically.
June 6, 2008 10:48 AM Someone unauthorized made it into an authorized only area and mistakenly hit a power switch that took down a path from FIC in Palo Alto to LA. This caused BGP flaps as the routers re-routed. We found this immediately and turned that power back on. A meeting with staff was just held to review the importance in preventing this human error from happening again. An awkward, embarrassing for us and rare event that we can not recall happening in the 6 year history of the company.
May 10 , 2008 Santa Clara Facility ONLY @ 5:40PM Only customers in this facility were effected - several times today the core router located in Santa Clara appeared to have problems with it's ethernet blade. Restarting the blade seemed to clear this up 2 times. But no "hardware" failure was placed in logs. The third failure left a log that indicated the software image was corrupt - we immediately loaded a new image and the checksum was good. We are now running a new version of code in that router. The problem is fixed OR the logs will now provide us more detail to help resolve it.
05/07/08 7:30(ish)-7:55AM Large Denial of service attack effecting some customers in both LA and SF areas, not all. For the "most" part originating/effecting LA area routers. Black Hole put in place , shortly there after it stopped.
Apr 5 1:50 AM While studying logs it became obvious the device would in fact fail soon. Device was replaced with a spare. Those circuit customers effected are all now operating fine. NOTE: Graph system uses the failed device to reach most all routers for data gathering...This is why your graph may show a drop to zero, yet you did really not experience an outage.
Apr. 5 12:40 AM Device Had Internal Failure on Path to some Bay Area Lateral circuits. Rebooted. Exploring Debug Logs to further evaluation. Lasted 15 mins.
Mar. 28 12:20 AM Major Fiber Problem effecting many SF bay area offices on our CO rings. The path from FIC to "that" ring was broken. It was repaired before commute hour. All are now back up 7:10AM. This was the largest we in several years in terms of how many customers were effected. And the first related on this path.
While waiting, we have engineered a work around. One that would have enable the use of other MAIN HUB sites on the CO rings. These are other paths that office packets could have directly traveled to another major HUB site also on those rings. Such as the telco floor at Market Post Tower in San Jose. However, since those effected routers were not reachable due to initial configuration setups at install, changes could not be made remotely.
We are now finalizing an engineering configuration change to those effected, which would enable a direct reach this other HUB location. It will not effect your service. This additional path setup will not cost anything to those customers. When completed this will prevent this type of outage from effecting your service in the future.
Implementation will take some time and should not effect services.
Feb 20 2:28AM-2:40AM Card (blade) went bad in path to co-lo router. Changed path to GBIC on alternate blade. Problem respolved. Co-Lo customers effected for 10-20 mins. of packet loss.
Feb. 11, 2008. About 10 AM we had some short intermitant GBIC link issues, at the FIC Co-Lo space in Palo Alto. Effecting only Co-Lo'ed servers in our facility. No circuit customers were effected. These were short outages. At Noon we changed GBICs in question, when it was clear it would not stablize. There have been no issues since the completion of this repair.
Jan 27 8:30PM..LA area customers effected as CO loop lost power at One Wilshire. Up and down 2 times for 15 mins. making the outage look longer than it was. Dual UPSes were installed , yet there was still an outage. Our CTO will fly to LA to investigate.
Jan 17 8:58AM Main Co-lo router had a problem with one of its slave controllers. Causing up/down issues. The slave kept taking over and the software kept allowing it , causing several reboots. Pulled slave processor. We be installing a new slave from stock and exploring the software rev level. It should not effect service. If it does , it will be just a couple minutes for this emergency repair.
2007 Dec 2nd Sunday...Most customers all didnt feel an outage - maybe just some BGP flap issues as traffic went out San Jose and LA. Those with private paths to PAIX "did" feel it. We are sorry. There was a miss-understanding regarding the city of Palo Alto's cut and move of fiber down town. In preperation for moving a power substation. We thought it was the 2nd Sunday of December - apparently it was today the 2nd of December. However, we prepared last month so flipping the private paths over to our 3rd path was not difficult. We have not yet heard from the city regarding completion, however, every private path customer should be operational at this time. The only exception are those in the 800 High downtown complex feed directly from that substation.
Nov 15th City is now feeding Fiber Internet Center's Palo Alto facility via a seperate power feed line. We hope that this will avoid the outages we have experienced in the past.
About this page & How to read the graphs
If the info has spelling and grammer errors, please remember that it's engineers that post it live, without any executive review.
Few ISPs would ever think of providing you with a LIVE instant snapshot of thier network's connectivity. But, we want you to be aware that we have all the tools in place to track and alert us. Here we provide you with an instant glimps into a primary artery of the Fiber Internet Center (FIC) network.
These Graphs are a LIVE 5 minute intervals of the FIC primary connections. The page is updated every 30 minutes.
A disconnection is represented by a FALL-TO-BOTTOM-ZERO. Completely to zero - remember, these graphs can fall to 10meg/sec and look close to zero. Look closely as that is not a disconnect. Any level other than zero means the network path never went down. Our monitoring tools will trigger an immediate notification Alert, using intervals of less than one minute. In the rare event that you see a FALL-TO-ZERO, all the proper FIC personnel have already been automatically notified and are working on the problem.
The graphs represent a current 24 hour period. Due to occasional dramatic high volume video stream traffic a large PEAK may push down the representation of the average bandwidth, as this graph's primary function is to inform you that FIC personnel are "already" aware an outage occurred, before this graph was updated. Several people are automatically notified by both external and internal systems. 3 of those people have mobile devices. They are paged immediately upon a 60 second lapse in total primary backbone connectivity.
If you see these graphs look fine (no symoltanuous drops to zero) yet you are experiencing problems reaching your machines or offices connected via FIC. So that we may serve you better, you should first attempt to PING or TRACEROUTE to your machines and notify us with the detailed output of these results.