This morning at approx. 9AM AEDT we received monitoring alerts for multiple sites across our hosted infrastructure. Only some customers and services were impacted. All services have returned to normal and our engineers are looking into the root cause of the issue. Unfortunately the outage impacted our status page so updates couldn't be posted at the time of the incident.
We are currently experiencing degraded network performance in our Sydney location. Our network engineers are actively working with our upstream providers to resolve the issue as quickly as possible.
Services are being restored and we will continue to monitor the situation. Unfortunately, situations like this will often be caused by attacks of some sort and are becoming more frequent. We will be doing a debrief with the data centre to understand how we might work around this situation moving forward and what options we have to further de-risk hosting outages for our customers. Obviously an outage of this length is far from ideal and Veeps is focused on ensuring we can provide a superior service for our customers by ensuring we have the right upsteam providers in place. Thanks for your understanding.
Our data centre has just received feedback from their upstream providers and have indicated that they are nearing resolution. Hopefully we will be able to provide a more positive update shortly.
We have received a further update from the data centre. There is currently ongoing issues with 2 large upstream providers. Our data centre is actively working with those providers to resolve the issues. Based on what we are seeing there are indications that the issue is related to a BGP (Border Gateway protocol) attack. As mentioned earlier this is similar to what we were seeing yesterday. Unfortunately this is something outside of our control, however we are in constant contact with the data centre who is managing the issues with their upstream providers. We will continue to provide ongoing updates.
Reports from the data centre indicate that they are still attempting to work around the issue. They have not provided us with a time to resolution at this stage and we are actively requesting further updates. We will continue to provide more information as soon as it's available.
We are continuing to work with our data centre provider to gain further information about the upsteam network outage that is impacting connectivity to the data centre.
The data centre itself is fully operational however one of the major network providers used by the data centre is experiencing an outage.
As a result, some but not all customers will be having difficulty connecting to their site or service based on their local provider.
There are reports that this outage is impacting multiple data centre providers across Australia.
We will continue to provide updates as soon as we receive further information.
We have identified a similar issue with the same Sydney data centre this morning with peering to certain upstream providers. Engineers are currently investigating.
Our data centre provider has confirmed that the root cause of the issue was related to an outage with an upstream network provider that impacted many services across Australia. All services have now stabilised and returned to normal operation.
Services are currently being restored. Our data centre network providers are either seeing services returning to normal or are routing around any ongoing issues.
We are unclear of the root cause of the issue but we know the issues impacted multiple data centres and network vendors. Many customers were not able to connect to their services during this time or would have experienced intermittent issues.
We will share any additional information once it becomes available.
We can confirm that all services within the data centre are operational however there are multiple upstream network issues causing connectivity issues for some customers. We will continue to monitor and provide additional updates once known.
