NETWORK STACK: Upgrades completed, OSPF & BGP sessions have reconnected. Other works are still in progress but will not effect public-facing services.
We're keeping this maintenance warning open in case of any unexpected failures.
Posted Feb 05, 2023 - 15:42 GMT
Update
EXCHANGE: Email cache flush completed.
TRIAROM DESKTOP SERVICES; Fully restored.
NETWORK STACK: UNDERGOING UPGRADES, MAY CAUSE INTERMITTENT CONNECTIVITY ISSUES.
Posted Feb 05, 2023 - 15:34 GMT
Update
EXCHANGE; Cached emails are being flushed through to the server, any emails sent during the maintenance window should start arriving shortly.
TRIAROM DESKTOP SERVICES: Service restored, performance degraded.
Posted Feb 05, 2023 - 15:21 GMT
Update
EXCHANGE: Service is now restoring. We will continue to monitor the system over the next few hours.
Posted Feb 05, 2023 - 14:09 GMT
Update
EXCHANGE: We have discovered a misconfiguration on a router that was causing painfully slow transfers of files. We have corrected this and are continuing with Exchange work.
Posted Feb 05, 2023 - 12:20 GMT
Update
EXCHANGE: We continue to work on restoring Triarom Exchange services.
NOTIFY + TRACK MY REPAIR + CUMBRIA SPA HOT TUB SERVICES: Services have resumed.
TRIAROM DESKTOP SERVICES: Awaiting restart - planned at 1200.
Posted Feb 05, 2023 - 11:07 GMT
Update
EXCHANGE: We continue our efforts to restore Exchange service, preparing to fallback on reinstating the old physical host - we will have an update within an hour.
All other service: remain on track for service restoration by 1400
Posted Feb 05, 2023 - 09:35 GMT
Update
EXCHANGE: The copy operation has completed successfully, and the systems have been reconfigured. We're working hard on restoring service to the Exchange services from the public internet.
Thank you again for your patience.
Posted Feb 05, 2023 - 00:18 GMT
Update
EXCHANGE: The copy operation for our inboxes database is drawing to a close. We will begin validation, configuration and service restoration within the next 20 minutes.
We are not providing an ETA for this step as the infrastructure is undergoing a "cold-start" - which can lead to longer initalisation times.
Posted Feb 04, 2023 - 23:51 GMT
Update
EXCHANGE: We are unfortunately unable to provide an ETA on Exchange service restoration - the copy operation is taking much longer than expected due to large inboxes. We are still aiming to restore service before midnight.
All other target restoration times remain.
Posted Feb 04, 2023 - 21:20 GMT
Update
Outage date:
EXCHANGE CUSTOMERS: We are now targeting an EXCHANGE service restoration time of 22:00 due to delays in disk migration with some large mailboxes.
ALL OTHER SERVICES: A full restoration time of 12:00 05/02/23 is still set. Some services will become available before this time.
Thank you for your patience.
Posted Feb 04, 2023 - 20:39 GMT
Update
Our core is back online. Services will be restored throughout the evening and leading into tomorrow morning.
We expect full restoration by 12pm 05/02/23.
EXCHANGE CUSTOMERS: Triarom Exchange is "in-flight" migrating into our newly updated service cluster. We plan to have Exchange service restored by 9pm tonight (04/02/23) - but will advise on updates to this.
Thank you for your patience.
Posted Feb 04, 2023 - 17:40 GMT
Update
UPDATE: The core reconfiguration and rebuild of our network infrastructure is complete - we're working on bringing services back online one-by-one.
Posted Feb 04, 2023 - 16:38 GMT
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Feb 04, 2023 - 10:01 GMT
Scheduled
NOTE: Apologies for any previously sent notifications, this maintenance was scheduled for the wrong days. It is this weekend, 4th and 5th of February.
NOTE 2: We have started preparation work for this maintenance window, which may cause a slight degradation in performance to many services, this will be remediated following the maintenance.
We will be performing maintenance on our server clusters between Saturday the 4th of February 10am and Sunday the 5th at 5pm. This maintenance work expands our system capacity, performance and will allow us to diagnose and resolve faults faster.
During this maintenance window we will:
- Add more compute capacity to our server cluster - Add more memory capacity to our server cluster - Physically clean our servers that have been running for over a year. - Replace the thermal conduction material on the server processors. - Update documentation to allow faster fault investigation. - Evaluate VLAN trunk/access ports - Perform tests of our battery backup & power cleaning systems. - Install host software updates - Reevaluate our backup solutions. - Rehearse a "cold-start" of our infrastructure. - Re-address many of our hosts in preparation for future internet access diversity upgrades. - Decommission and virtualise two of now-redundant our servers, to reduce our carbon footprint. - Restack our server cluster to follow updated documentation.
EXPECTED OUTAGES: - TDS Network Stack - Intermittent outages between the window - Triarom Cloud Desktops - Outage for the entire window. - Triarom Cloud Desktop Storage Platform - Out for the entire window. - Triarom Exchange - Intermittent outages, with a maximum of a 2 hour outage window. - Triarom Hosted UniFi - Outage for the entire window. - Triarom Computers Support Platform - Intermittent outages. - Triarom Remote Network Monitoring - Outage for the entire window.
If you have any questions or concerns about this work, please don't hesitate to contact us.