XEN/KVM OS Upgrade & security patch (Resolved)
- Priority - High
- Affecting Server - VPS Master
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 05/22/2022
EVENT START TIME: 00:01 EDT
EVENT END DATE: 05/22/2022
EVENT END TIME: 06:00 EST
EVENT LOCATION: Midwest NOC
EXPECTED DOWN TIME: 10 -45 minutes
EVENT DETAILS:
Maintenance XEN Server OS Upgrade & security patch
Site Affected: XEN Nodes 5
Maintenance KVM Server OS Upgrade & security patch
Site Affected: KVM Nodes 1
Maintenance OPENVZ Server OS Upgrade & security patch
Site Affected: OVZ Node Nodes 3
Update /05/22/2022 1257 EDT
All Upgrades complete No Issues
- Date - 05/21/2022 22:00 - 05/22/2022 18:49
- Last Updated - 05/21/2022 22:58
Xen 6 Maintenance (Resolved)
- Priority - High
- Affecting Server - Las Vegas NOC
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 03/16/2022
EVENT START TIME: 12:45 EST
EVENT END DATE: 03/16/2022
EVENT END TIME: 14:00 EST
EVENT LOCATION: Las Vegas NOC
EXPECTED DOWN TIME: 10 -20 minutes
EVENT 3 DETAILS:
Maintenance XEN 6 Retirement
Our XEN 6 Hardware Node is being replaced by newer hardware we will be migrating all VPS’s on XEN 6 In Las Vegas to XEN 2 Las Vegas, down time will be approximately 10-20 minutes.
Upgrades complete no issues
- Date - 03/14/2022 21:50 - 03/23/2022 08:42
- Last Updated - 03/23/2022 08:43
VPS Node OS upgrade (Resolved)
- Priority - High
- Affecting Server - VPS Master
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 04/17/2021
EVENT START TIME: 00:01 EST
EVENT END DATE: 04/21/2021
EVENT END TIME: 23:59 EST
EVENT LOCATION: Midwest NOC
EXPECTED DOWN TIME: 2 - 30 minutes
EVENT 3 DETAILS:
We will be upgrading our node OS to AlmaLinux 8 & Openvz 7.
Site Affected: XEN Nodes All, KVM nodes ALL, OpenVZ nodes all.
Las Vegas XEN nodes will be scheduled at a later date.
We will be cutting all VPS servers over to Newer Node Operating systems expected downtime will be 2 to 30 Min depending on the size of your VPS.
-----------------------------------------------
Update 04/20 18:00 EST All XEN VPS nodes have been completed
-------------------------------------------------
Update 04/21/ The XEN nodes took longer than expected extending the maintenance window to end of the day 4/21
-----------------------------------
Update 04/21/ All KVM Nodes have been completed
---------------------------------------------------------
Update 04/21/ All OPENVZ Nodes have been completed
--------------------------------------------------------
Update 04/22 one of the Open VZ nodes is failing to boot we are repairing it
--------------------
Update 4/23 all upgrades complete
- Date - 04/21/2021 04:30 - 04/28/2021 16:21
- Last Updated - 04/28/2021 16:21
Server XEN 3 Failing Post on OS boot (Resolved)
- Priority - Critical
- Affecting Server - VPS Master
-
· Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 09/04/2020
EVENT START TIME: 10:51 PDT
EVENT LOCATION: SB NOC (Midwest)
Server XEN 3 Failing Post on OS boot
EVENT 1 DETAILS:
Server Xen 3 went into a kernel Panic & is failing to Post for Boot
The NOC tech is on scene & is investigating
Update 12:15 PDT
One of the memory sticks Froze on reboot Error was cleared & Ram/memory reseated & server is on line
Update 2:20 PDT
We will be monitoring the system for errors and replacing the affected DIMM if we get an Error
- Date - 10/04/2020 10:51 - 01/22/2021 23:08
- Last Updated - 10/04/2020 15:34
Transit Provider Maintenance (Resolved)
- Priority - Low
- Affecting Server - Las Vegas NOC
-
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 03/16/2020
EVENT START TIME: 2300:00 PDT
EVENT END DATE: 03/17/2020
EVENT END TIME: 03:00 PDT
EVENT LOCATION: Hurricane Electric POP Las Vegas
EVENT DETAILS:
Lowest Host/Empire Technology LLC has been informed by Hurricane Electric will be performing Maintenance: Router software upgrade to enhance security and stability
Associated circuit(s): Las Vegas 1 & 2
Thank You,
Datacenter Operations Team
- Date - 03/16/2020 23:00 - 10/04/2020 11:29
- Last Updated - 03/17/2020 01:04
Open VZ 1 Maint (Resolved)
- Priority - High
- Affecting Server - VPS Master
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 12/25/2019
EVENT START TIME: 00:01 PST
EVENT END DATE: 12/25/2019
EVENT END TIME: 01:00 PST
EVENT LOCATION: SB NOC Open VZ 1 (Midwest)
EVENT DETAILS: Reboot To sync up with backup server
EXPECTED DOWN TIME: 5 MIN - 15 MIN
EVENT DETAILS: Open VZ 1
Maintenance: we will be reinstalling the OS Kernel as the current Kernel is not syncing up with our backup server. This will require 2 reboots
---
Update 12/25/2019 06:09 PST
Maintenance complete No issues
- Date - 12/25/2019 00:01 - 01/06/2020 09:30
- Last Updated - 12/25/2019 06:10
partial network Outage (Resolved)
- Priority - Medium
-
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 09/10/2019
EVENT START TIME: 05:24 PST
EVENT END TIME: 06:44 PDT
EVENT LOCATION: South Bend NOC
EVENT 1 DETAILS:
emergency Maint: Our Primary distribution switch in our South Bend Data center had a partial failure. We have dispatched a NOC technician to troubleshoot why specific ports remain dead even after resetting them.
===============================================
Update 06:20 PDT
after Further Investigation Our Ups Power on FEED #2: Went into an overload condition. It was receiving Irregular unclean Power from the utility company (Indiana Michigan Power Company) 110.32 % of normal. Which sent our UPS: into Bypass mode (Battery power) we remained on Battery power but the fault did not clear & our ups shut down as a safety precaution to not overload equipment.
===============================================
Update 06:44 PST
We have reset the circuit and have requested an electrician be dispatched to take a closer look and adjust as necessary. All Network anomalies were resolved at this time (Xen 3’s network took a bit longer to come up but is on line again). All servers & network issues should be resolved if you are still having issues Please open a support ticket so we can take a closer look.
===============================================
Update 07:20 PST
we will be monitoring the circuit for the next 24 hours and send out further updates as needed
- Date - 09/10/2019 05:24 - 09/13/2019 10:54
- Last Updated - 09/10/2019 09:13
Router(s) Upgrade/Replacement (Resolved)
- Priority - Medium
-
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 06/24/2019
EVENT START TIME: 00:10 EDT
EVENT END DATE: 06/28/2019
EVENT END TIME: 04:00 EDT
EVENT LOCATION: SB NOC (Midwest)
EVENT DETAILS: Router(s) Upgrade/Replacement
EXPECTED DOWN TIME: 15-60 Seconds
EVENT DETAILS:
The week of Jun 23, 2019 we will be replacing our aging routers in our Midwest data center with new Cisco Routers to accommodate more capacity & resource usage.
We are expecting 0 down time as we will be hot swapping each Router one at a time leaving at least one network link active. There will be a 15-60 second outage as the routes converge when your server/rack is cut over.
Note: BGP customers with Redundant Links you will get 2 separate emails for each Link (this includes private BGP sessions). Please verify that we are sending the correct routes. Single homed BGP customers please verify that your session is receiving the correct routes (default route for most of you).
===========================================
Update 06/28/2019
All upgrades are complete: routing adjusted to optimal, Redundant Failover tested
============================================
Update 07/3/2019
Adjusted IPV6 Routing for VPS servers only (autoconfig disabled it enabled itself during cut over)
=============================================
Update 07/15/2018
Moved IPV6 IBGP temporary link off old Primary router & router decommissioned
- Date - 06/24/2019 00:10 - 08/30/2019 05:22
- Last Updated - 07/23/2019 06:52
CVE-2019-0211 Patch (Resolved)
- Priority - High
- Affecting System - Apache Web Server
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 4/4/20198
EVENT START TIME: 06:10 PDT
EVENT END DATE: NA
EVENT END TIME: NA
EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)
EVENT DETAILS: CVE-2019-0211, According to the Apache team, less-privileged Apache child processes (such as cgi) can execute malicious code with the privileges of the parent process.
Because on most Unix systems Apache httpd runs under the root user, any threat actor who has planted a malicious CGI script on an Apache server can use CVE-2019-0211 to take over the underlying system running the Apache httpd process, and inherently control the entire machine.
WARNING: Apache HTTP 2.4.17 to 2.4.38 Local Root Exploit
Apache HTTP 2.4.17 to 2.4.38 is vulnerable to a local root exploit when mod_prefork, mod_worker and mod_event are used:
https://httpd.apache.org/security/vu...lities_24.html
We are hearing reports of exploit(s) already being produced and strongly recommend that everyone update to Apache HTTP 2.4.39 as soon as possible - especially in shared hosting environments!
https://www.zdnet.com/article/apache...-environments/
CloudLinux
https://cloudlinux.com/cloudlinux-os-blog/entry/beta-ea-apache24-2-4-39-1-cloudlinux-major-security-update
Maintenance Apache: 4.39 patch
EXPECTED DOWN TIME: NA
EVENT DETAILS:
Server OS Upgrade & security patch
Site Affected: All Managed Dedicated & VPS Servers Running the Apache web server.
We will be verifying all managed cpanel servers running Apache have been updated.
Directadmin managed servers we will be patching all servers.
cPanel cloudlinux managed servers we will be patching via the testing repo.
Per usual this event can be tracked @ https://secure.empire-hosting.net/serverstatus.php
--------------------------------------------------------
For all unmanaged clients & some managed clients Please see your respective repo if this vulnerability affects you. See below for Control panel specific information.
cPanel CentOS 6/7 if you are running easyapache 4 with automatic updates your server will be updated automatically.
cPanel if you are running easyapache3 or an unsupported End OF Life Operating system there is no patch available.
cPanel Cloudlinux 6/7 Patch is available via the testing repo
Directadmin Centos 6/7 patch is available via custombuild.
Directadmin Centos an unsupported End OF Life Operating system there is no patch available.
Litespeed web server is unaffected.
================
Update 4/4/2019 0730 PDT
All Shared Hosting servers have been Patched/updated
=======================================
Update 4/4/2019 0800 PDT
All Managed DirectAdmin servers have been Patched/updated
=============================================
Update 4/4/2019 11:00 PDT
All Managed cPanel servers verified & updated if needed
=============================================
Update 4/4/219 11:00 PDT
All Managed Cpanel/CloudLinux servers Patched/updated
- Date - 04/04/2019 06:00 - 04/09/2019 06:55
- Last Updated - 04/04/2019 10:57
All Managed Dedicated VPS Servers CloudLinux (Resolved)
- Priority - High
- Affecting System - Systems running CloudLinux OS
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 11/24/2018
EVENT START TIME: 00:10 EST
EVENT END DATE: 11/24/2018
EVENT END TIME: 04:00 EST
EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)
EVENT DETAILS: CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'
Maintenance OS Kernel: 'Foreshadow & Foreshadow-NG'Patch
EXPECTED DOWN TIME: 5-10 Min
EVENT 1 DETAILS:
Server OS Upgrade & security patch
Site Affected: All Managed Dedicated & VPS Servers Running the CloudLinux operating system .
Note: if you are an unmanaged client Patch your OS at your leisure
NOTE: Warning if you are running Kernelcare rebootless kernel (licensed Version) it will not patch for CVE-2018-3615/3620/3646 it requires an up2date kernel & reboot.
---------------------------------
Update 07:00 EST
Maintenance Complete No issues
- Date - 11/23/2018 00:10 - 11/24/2018 20:00
- Last Updated - 11/24/2018 05:56
OS Upgrade & security patch (Resolved)
- Priority - High
- Affecting Other - Shared Hosting Servers
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 11/24/2018
EVENT START TIME: 00:10 EST
EVENT END DATE: 11/24/2018
EVENT END TIME: 03:00 EST
EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)
EVENT DETAILS: Maintenance OS Kernel: OS Upgrade & security patch
EXPECTED DOWN TIME: 5-10 Min
EVENT 1 DETAILS:
Server OS Upgrade & security patch
Site Affected: All Shared Hosting servers
---------------------------------
Update 07:00 EST
Maintenance Complete No issues
- Date - 11/24/2018 00:10 - 11/24/2018 20:00
- Last Updated - 11/24/2018 05:56
Upstream IP Transit Edge Router Maintenance (Resolved)
- Priority - Low
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 11/07/2018
EVENT START TIME: 000:00 EST
EVENT END DATE: 11/07/2018
EVENT END TIME: 06:00 EST
EVENT LOCATION: SB NOC (Midwest)
This will be a no impact issue we will be turning down the circuit at 10:00 EST 11/06/2018 until completed. Below is a summery of maintenance notice sent to us.
========
As a valued customer, Cogent is committed to keeping you informed about any changes in the status of your service with us. This email is to alert you regarding maintenance we will be performing on our network:
Start time: 12:00 AM (local time) 11/7/2018
End time: 06:00 AM (local time) 11/7/2018
Work order number: NC840-80
Order ID(s) impacted: XXXXXXX
Expected Outage/Downtime: 45-60 minutes
Cogent customers receiving service in the greater South Bend, IN metro area will be affected by this maintenance. Only the Cogent Order ID(s) listed above will be impacted. The purpose of this work is to apply and activate software module upgrades on code running on the edge routers per the manufacturer's recommendations.
=============================================
Maint complete NO issues
- Date - 11/06/2018 10:00 - 11/09/2018 09:10
- Last Updated - 11/07/2018 17:44
Maintenance OS Kernel: (Resolved)
- Priority - Low
- Affecting System - All Managed Dedicated Servers
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 09/15/2018
EVENT START TIME: 13:00 PDT
EVENT END DATE: 9/16/2018
EVENT END TIME: 04:00 PDT
EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)
EVENT DETAILS: CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'
Maintenance OS Kernel: 'Foreshadow & Foreshadow-NG'Patch
EXPECTED DOWN TIME: 5-10 Min
EVENT 1 DETAILS:
Server OS Upgrade & security patch
Site Affected: All Managed Dedicated Servers.
Note: if you are an unmanaged client Patch your OS at your leisure
Note2: all RedHat variants Have Patches available with the exception of CloudLinux OS
Note3: Debian 8 No patch released
NOTE4: Warning if you are running Kernelcare rebootless kernel (licensed Version) it will not patch for CVE-2018-3615/3620/3646 it requires an up2date kernel & reboot.
NOTE5: If you are running CloudLinux OS, they have not pushed a patch yet all CloudLinux customers will be scheduled on a later date.
Update 05:53 PDT
All Servers Patched no issues
- Date - 09/15/2018 10:12 - 09/18/2018 16:07
- Last Updated - 09/16/2018 10:28
Maintenance OS Kernel: (Resolved)
- Priority - Low
- Affecting System - All Managed VPS Servers
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 09/15/2018
EVENT START TIME: 13:00 PDT
EVENT END DATE: 9/16/2018
EVENT END TIME: 01:00 PDT
EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)
EVENT DETAILS: CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'
Maintenance OS Kernel: 'Foreshadow & Foreshadow-NG'Patch
EXPECTED DOWN TIME: 0 (new OS kernel & software patch will be loaded on Node reboot see below)
EVENT 1 DETAILS:
Server OS Upgrade & security patch
Site Affected: All Managed VPS XEN & KVM Servers, ( OpenVZ NA).
Note: if you are an unmanaged client Patch your OS at your leisure
Note2: all RedHat variants Have Patches available with the exception of CloudLinux OS
Note3: Debian 8 No patch released
EXPECTED DOWN TIME: 5MIN - 15 MIN
EVENT 2 DETAILS:
Our XEN/KVM/OpenVZ Maintenance Virtuozzo Server OS Upgrade & security patch
Site Affected: VPS Servers All Nodes
Update O5:55 PDT
Updates complete No issues
- Date - 09/15/2018 10:14 - 09/18/2018 16:07
- Last Updated - 09/16/2018 10:28
OS Upgrade & security patch (Resolved)
- Priority - Low
- Affecting System - Shared Hosting Servers
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 09/15/2018
EVENT START TIME: 13:00 PDT
EVENT END DATE: 9/16/2018
EVENT END TIME: 01:00 PDT
EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)
Maintenance OS Kernel: OS Upgrade & security patch
EXPECTED DOWN TIME: 5-10 min
EVENT 1 DETAILS:
Server OS Upgrade & security patch
Site Affected: All Shared Hosting servers
Update 05:56 PDT
Updates complete No issues
- Date - 09/15/2018 10:09 - 09/18/2018 16:07
- Last Updated - 09/16/2018 10:28
June 21 RTO and IPV6 Circuit Turn up (Resolved)
- Priority - Low
-
Description of Outage:
On Thursday , June 21, 2018 at approximately 06:59 PM PST, Lowest Host Empire Technology LLC was experiencing routing drops of our Level3 IPv6 Network in our Midwest Data Center. In an effort to Repair the disruption our Network Feed was moved to a New Router & port. Due to a failure of our uptream’s internal core routing configuration with our router, it caused our core2.SBII Router to drop large portions of our IP announcements entering our network. This caused degraded network performance, packet loss, and the inability to reach our network.
Our core2.SBII Router handles our upstream BGP sessions. Which were ultimately unreachable due to the failed configuration on our primary feed, this condition lasted until our upstream tuned down their link.
Resolution:
The Faulty configuration was fixed by our upstream peer. Configuration was restored, and our routing protocols began operating normally at this time.
Network Status IPv4 100% Primary and secondary Feeds
Network Status IPV6 Inbound 100% Outbound 50% degraded Level3 is not routing our outbound packets and they are still trying to find a resolution.
-------------------------------------------------------------------------------------
In progress:
Turnning up a new IPV6 Link
Status Cable being installed
---------------------------------------------------------------------------------------
Update June 23, 2018
Link has been established awaiting on seesion information
----------------------------------------------------------------------------------------
Update June 23, 2018
Network Status IPV6 100% Primary and secondary Feeds
Level3 IPV6 routing issue resolved at this time
- Date - 06/22/2018 10:56 - 08/11/2018 20:57
- Last Updated - 06/23/2018 10:41
VPS Server Meltdown & Spectre Patch (Resolved)
- Priority - High
- Affecting Server - VPS Master
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 1/28/2018
EVENT START TIME: 08:30 PST
EVENT END DATE: TBA
EVENT END TIME: TBA
EVENT LOCATION: ALL
EXPECTED DOWN TIME: 5 -10 minutes Possible multiple patches.
EVENT DETAILS: Meltdown
Maintenance CPU Vulnerabilities: Meltdown & Spectre Patch
Site Affected: All VPS servers
We will be patching all Managed dedicated VPS servers for the vulnerabilities above. Currently All managed dedicated clients can be patched with the exception of Cloud Linux & kernelcare (awaiting patch). If you are an unmanaged client please check with your OS vendor if they have a patch ready. Please refer to the security notice email dated 1-14 for the current status of the OS vendors we use.
Currently Redhat/Centos 6 & 7 is the only OS that has patched for CVE-2017-5753 & CVE-2017-5754 Spectre 1 & Meltdown 3, waiting on CPU manufactures to release patches for CVE-2017-5715 Spectre II.
As of today
Centos 6/7 Scientific Linux 6/7 Patched Spectre 1 & Meltdown 3
Cloudlinux 6/7 waiting for Patches
Kernelcare ALL waiting for Patches
Ubuntu 14/16 Meltdown 3 only
Debian 8/9 Meltdown 3 only
Centos 5 EOL it won’t be patched upgrade to centos 7
Debian 6 EOL it won’t be patched upgrade to Debian 9
Gentoo 2011 EOL upgrade to 12.1 2018 patch
===============================================================
WARNING WARNING WARNING !!!
Do not patch your XEN VPS Currently the Latest kernels for XEN are unstable and may result in your server not booting. If you have upgraded your OS kernel on Xen please set the boot back to your previous OS kernel.
At this time we may do a forced upgrade on all VPS server managed or unmanaged node(s) wide. For Centos, Scientific Linux, CloudLinux we will defiantly. For Ubuntu and Debian it is still undecided. We may push our own sources mirror with Patched Kernels if they do not patch their OS kernels in a timely manner.
For those running EOL OS’s centos 5, Debian 6, and Gentoo we are going to try building a custom Kernel and pushing the patch
===============================================================
Virtualization types
OpenVZ: Node only patch client OS does not require any Patch.
KVM: Node and Client OS.
XEN: Node and Client OS, still waiting on a stable Patch for XEN VPS servers see warning above.
====================================================================
Update 1-28-2018 21:35:43 PST
We will be patching All KVM servers Node software
We will be patching all OpenVZ Node software
for Meltdown & Spectre Patch
======================================================================
Update 1-28-2018 11:38:43 PST
Upgrades to KVM OpenVZ complete no issues
======================================================================
Update 2-4 2018 09:12:00 PST
Upgraded all Managed VPS Servers on KVM.
Did a Forced upgrade on all unmanaged servers running CentOS 7 & CentOS 6 x86_64.
===========================================================
Update 2-8 2018 04:22:00 PST
We will Be patching the all of the XEN nodes for the Meldown Spectre Vulnerabilities Expected Dowtime 10 MIN
============================================================
Upadate 2-14 2018 10:24 PST
All Xen Nodes have Been Patched No issues
============================================================
Upadate 3-4 2018 9:20 PST
All Xen & all KVM servers have been forcfulliy patched for Meldown Spectre
============================================================
Upadate 3-4-2018 09:35 PST
Just realized we did not Patch KVM 3 & Openvz 1 for spectre Meltdown
we will be applying the patch and rebooting the node
expect 10 min of down time
========================================================
Closing all servers patched that can be
- Date - 01/28/2018 08:16 - 04/03/2018 02:41
- Last Updated - 04/03/2018 02:41
shared Sever Meltdown Spectre (Resolved)
- Priority - High
- Affecting Server - rc1
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 02/14/2018
EVENT START TIME: 08:59 PST
EVENT END TIME: 16:00 PST
EVENT DETAILS: Emergency Maintenance all Shared Hosting servers
We will be patching all of our shared hosting servers for the CPU Vulnerabilities: Meltdown & Spectre.
For reference see security Notice email sent out On Jan 16 2018,
We will be rebooting each shared hosting server expect 5-10 min of down time while we apply the patch.
====================================================================
Update 02/14/2018 10:22 PST
All updates complete no Issues
- Date - 02/14/2018 09:02 - 02/17/2018 08:59
- Last Updated - 02/14/2018 10:26
Dedi Server : Meltdown & Spectre Patch (Resolved)
- Priority - Low
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 1/16/2018
EVENT START TIME: 08:30 PST
EVENT END DATE: TBA
EVENT END TIME: TBA
EVENT LOCATION: ALL
EXPECTED DOWN TIME: 5 -10 minutes
EVENT DETAILS:
Maintenance CPU Vulnerabilities: Meltdown & Spectre Patch
Site Affected: All managed dedicated servers
We will be patching all Managed dedicated server s for the vulnerabilities above. Currently All managed dedicated clients can be patched with the exception of Cloud Linux & kernelcare (awaiting patch). If you are an unmanaged client please check with your OS vendor if they have a patch ready. Please refer to the security notice email dated 1-14 for the current status of the OS vendors we use.
========================================================
Update Jan 17 2018 09:17 PST
Update All Centos Managed dedicated server Patched with Partial support ( awaiting Intel patches)
Update Still waiting on CloudLinux patches
=========================================================
Update Feb 2 2018 09:33 PST
Update CloudLinux 7 Kernelcare 7 Patches have been released all managed servers have been patched
Update Still waiting on CloudLinux 6 Kernelcare 6
===========================================================
Update Feb 4 2018 08:00 PST
Update All CloudLinux 6 all managed dedicated servers have been patched with the CloudLinux 6 Hybrid kernel.
Update all managed dedicated have been patched down grading this issue to low.
- Date - 01/16/2018 08:30 - 02/11/2018 21:05
- Last Updated - 02/04/2018 09:17
Dedicated Server Clients: Bandwidth graphs (Resolved)
- Priority - Low
- Affecting Other - Dedicated Server Clients: Bandwidth graphs
-
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 01/03/2017
EVENT START TIME: 05:24 PST
EVENT END TIME: TBA
Due to the fact that our Edge router failed in Las Vegas our Data center Bandwidth Graphs have Become too out of sync. Restoring from backup did resolve the issue they would run for about 30 minutes and then would drop all logging from logging errors. We will be rebuilding all clients BW monitoring.
=============================================================
Update Jan 3 14:20 PST
We have be rebuilt all client BW monitoring.
=============================================================
Update Jan 4 12:43 PST
We have upgraded our monitoring software to the latest version unfortunately our entire Bandwidth display in our dedicated server management area is not compatible with the new graph & font size of the the most up to date software. We are currently rebuilding the dedicated server manager to accommodate the changes.
- Date - 01/03/2018 05:24 - 01/11/2018 01:21
- Last Updated - 01/04/2018 12:53
Emergency Network Maintenance Las Vegas (Resolved)
- Priority - Low
-
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 01/02/2017
EVENT START TIME: 05:24 PST
EVENT END TIME: 08:44 PST
EVENT LOCATION: Las Vegas NOC
EVENT 1 DETAILS:
emergency Maint: Our IP Network announcement in our Las Vegas Data center has failed we have dispatched a NOC technician to the Datacenter to try and clear the Network fault on or Edge router.
===============================================
Update 07:50
After Further Investigation Our Edge Router is not rebooting we are currently replacing the router Please standby for further updates.
===============================================
Update 08:44 PST
We have replaced Our Edge router all server should be on line once again if you are still having issues Please open a support ticket so we can take a closer look.
===============================================
Update 16:20 PST
we are downgrading this Maintenance as it was completed @ 08:44 PST.
We will leave issue open for 24 more hours.
===============================================
Update Jan 3 05:20 PST
Our Data center Bandwidth Graphs have Become too out of sync. Restoring from backup did resolve the issue. We will be rebuilding all clients BW monitoring.
- Date - 01/02/2018 05:27 - 01/04/2018 12:54
- Last Updated - 01/03/2018 05:33
Data center IP transit Maintenance: (Resolved)
- Priority - Low
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 07/09/2017
EVENT START TIME: 11:51 PST
EVENT END TIME: TBA
EVENT DETAILS: Emergency IP transit Maintenance
On or about July 5th 2017
our Level3 Primary transit was having 10 sec to 5 Minute optical drops we opened a Ticket to have Level3 troubleshoot the issue
========
After further investigation by Level3 they have determined that the one of Core Routers at the Point of presence (POP) Is creating the issue.
Level3 has dispatched a technician to the data center to replace router or parts as necessary
We have turned down on our primary Level3 Feed until the repairs are complete
- Date - 07/09/2017 11:49 - 07/14/2017 16:46
- Last Updated - 07/09/2017 12:11
Moderate Packet Loss On Las Vegas Shared Serv (Resolved)
- Priority - High
- Affecting Server - rc1
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 06/30/2017
EVENT START TIME: 09:45 PDT
EVENT LOCATION: Las Vegs NOC
EVENT DETAILS:
We are experiencing Moderate network packet Loss on Or RC1 shard Hosting server in Los Vegas we are investigating the issue
===========================
We have resolved the issue we will continue to monitor the issue for any further anomalies
- Date - 06/30/2017 08:45 - 07/09/2017 11:49
- Last Updated - 06/30/2017 09:24
XEN 8 Raid Failure (Resolved)
- Priority - Critical
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 06/28/2017
EVENT START TIME: 05:44 PST
EVENT LOCATION: Las Vegas NOC
EVENT 1 DETAILS:
emergency Maint we have had a total Raid Failure on XEN 8 we are restoring servers from backup
============================================================
All vps servers restored from backup
- Date - 06/28/2017 05:47 - 06/29/2017 12:35
- Last Updated - 06/28/2017 10:01
Maintenance Cisco Edge Switch (Resolved)
- Priority - Medium
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 06/17/2017
EVENT START TIME: 00:01 EST
EVENT END DATE: 06/17/2017
EVENT END TIME: 00:06 EST
EVENT LOCATION: Midwest NOC
EXPECTED DOWN TIME: 5 minutes
EVENT DETAILS:
Maintenance Cisco Edge Switch IOS reload. We will be performing maintenance on one of our Edge switches to clear a memory fault.
Site Affected: Midwest NOC
=======================================
Maint Complete
- Date - 06/17/2017 22:01 - 06/28/2017 05:47
- Last Updated - 06/16/2017 21:08
XEN Server OS Upgrade & security patch (Resolved)
- Priority - High
- Affecting Server - VPS Master
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 05/27/2017
EVENT START TIME: 00:01 EST
EVENT END DATE: 05/27/2017
EVENT END TIME: 06:00 EST
EVENT LOCATION: Midwest NOC
EXPECTED DOWN TIME: 10 -20 minutes
EVENT 3 DETAILS:
Maintenance XEN Server OS Upgrade & security patch
Site Affected: XEN Nodes 3, 7, 5
===================
Updates completed XEN 7 was cancleled will be resceduled at a later date
- Date - 05/27/2017 00:01 - 06/10/2017 17:33
- Last Updated - 05/26/2017 22:44
XEN Server OS Upgrade & security patch (Resolved)
- Priority - High
- Affecting Server - VPS Master
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 05/6/2017
EVENT START TIME: 00:01 EST
EVENT END DATE: 05/6/2017
EVENT END TIME: 06:00 EST
EVENT LOCATION: Las Vegas NOC (West Coast)
EXPECTED DOWN TIME: 10 -30 minutes
EVENT 1 DETAILS:
Non emergency Drive(s) Replacement. On Xen 2 (retiring server)
EVENT 2 DETAILS: Repair Of OS install on Xen 8, Our VPS control panel is not syncing with the OS.
EXPECTED DOWN TIME: 10 -30 minutes Note: Please do not reboot your server if you are on this node please open a ticket until maintenance is completed.
EVENT 3 DETAILS:
Maintenance XEN Server OS Upgrade & security patch
Site Affected: XEN Nodes 2, 8, 6
------------------------------------------------
Update: all node upgrades have been completed except XEN 2
we have extended the maintenance action to Sunday May 7 for xen 2
- Date - 05/05/2017 10:01 - 05/15/2017 05:50
- Last Updated - 05/07/2017 09:08
Xen Network Bridge crash (Resolved)
- Priority - Low
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 4/19/2017
EVENT START TIME: 12:00 PDT
EVENT END DATE: 4/19/2017
EVENT END TIME: 12:05 PDT
EVENT LOCATION: Midwest NOC (XEN3 )
EVENT DETAILS: emergency maintenance
Orur XEN 3 node Network crashed we took the time and patched the server and restasrted the network on all Virtual machines
Thank You Server Operations Team.
==========================
- Date - 04/19/2017 12:00 - 04/20/2017 07:37
- Last Updated - 04/19/2017 12:09
Virtuozzo Server Maint (Resolved)
- Priority - High
- Affecting Other - SB NOC (Midwest)
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 4/2/2017
EVENT START TIME: 12:00 PDT
EVENT END DATE: 4/2/2017
EVENT END TIME: 17:00 PDT
EVENT LOCATION: Midwest NOC (VPS 2 )
EXPECTED DOWN TIME: 5-15 Min .
EVENT DETAILS: emergency maintenance
We will be retiring are our Openvz node 2 and moving data to new hardware 4/2/2017 12:00 PM PDT - 4/2/2017 1700:00 PM PDT
Thank You Server Operations Team.
==========================
Maint complete No issues
- Date - 04/02/2017 12:00 - 04/04/2017 08:09
- Last Updated - 04/04/2017 08:09
Decommission Xen 1 (Resolved)
- Priority - High
- Affecting System - XEN 1
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 3/27/2017
EVENT START TIME: 05:00 PDT
EVENT END DATE: 4/2/2017
EVENT END TIME: 17:00 PDT
EVENT LOCATION: Midwest NOC (XEN 1)
EXPECTED DOWN TIME: 5-20 Min .
EVENT DETAILS:
We will be retiring are our oldest XEN node and moving data to new hardware 3/27/2017 05:00 AM PDT - 4/2/2017 1700:00 PM PDT
Thank You Server Operations Team
===============================
update
This maintenance has been completed No issues
- Date - 03/27/2017 05:00 - 04/02/2017 10:55
- Last Updated - 04/01/2017 09:43
DDOS Attack Las Vegas Shared hosting server. (Resolved)
- Priority - Low
- Affecting Server - rc1
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 2/27/2015
EVENT START TIME: 10:25 PST
EVENT LOCATION:Las Vegas NOC
EVENT DETAILS:
One of our shared hosting servers ins under a layer7 DDOS attack RC1 attackers are flooding the inbound network connection
We are currently mitigating the the attack at this time:
Note: we will be making some modifications to the network on this server it will be unreachable until we stop the attack
Note: some of you who are are receiving the attack your domains will be redirected and you will be notified in a separate Email
============================
Update
We have successfully mitigated the attack
For Now
We are currently Running a File system check
As the file system was damaged during the attack
========================================
Update
We were able to mitigate the attack in fast fashion
But do to the heavy volume of the attack and the server crashing several times during the attack
The server was Forced into a manual Maintenance cycle and the file system had to be manually repaired.
The server is back on line we will continue to monitor and mitigate the lingering attacks.
- Date - 02/27/2017 10:25 - 03/05/2017 09:09
- Last Updated - 02/27/2017 18:42
Upgrade Easyapache 4 (Resolved)
- Priority - Medium
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 1/29/2017
EVENT START TIME: 05:00 PST
EVENT END DATE: 2/28/2017
EVENT END TIME: 17:00 PST
EVENT LOCATION: Shared Hosting servers (ALL)
EXPECTED DOWN TIME: 0 there will be a cut over to the new Apache configuration which may cause a few sporadic restarts.
EVENT DETAILS:
We will be performing server maintenance: on all of Shared Hosting Servers Reason: Easyapache 3 is going End of life and to keep our clients up to date with the latest security and software we must make this upgrade to Easyapache 4. Maintenance Window: 1/29/2017 05:00 AM - 2/28/2017 1700:00 PM PST
Thank You Server Operations Team
=============
Update All Las Vegas shared hosting servers updated
=============
Update all Midwest shared Hosting servers updated
- Date - 01/29/2017 05:04 - 02/02/2017 09:28
- Last Updated - 01/31/2017 06:12
XEN Server OS Upgrade & security patch (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 12/23/2016
EVENT START TIME: 00:01 EST
EVENT END DATE: 12/23/2016
EVENT END TIME: 22:00 EST
EVENT LOCATION: Las Vegas NOC (West Coast)
EXPECTED DOWN TIME: 10 -30 minutes
EVENT 1 DETAILS:
Non emergency Drive Replacement.
EVENT 2 DETAILS: Start Date 12/23/2016 00:01 EST
Maintenance XEN Server OS Upgrade & security patch
Site Affected: XEN Nodes 2, 8, 1
===================================
Updates complete 2, 1
Updates Complete with the exception of xen 8
we are tweaking the drive sets for performace
- Date - 12/22/2016 10:01 - 01/12/2017 05:26
- Last Updated - 12/27/2016 00:21
KVM Servers Hard Drive upgrade (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 10/15/2016
EVENT START TIME: 00:01 EDT
EVENT END DATE: 10/15/2016
EVENT END TIME: 06:00 EDT
EVENT LOCATION: SB NOC (Midwest)
EXPECTED DOWN TIME: 5-25 MIN
EVENT DETAILS:
We will be performing server maintenance: on all of KVM Nodes. Reason: we found that our KVM Nodes SSD drives have degraded in sustained write speed over time. We are going upgrade to a different manufacture & model that has proven to be much better in terms of performance & longevity with our current Raid card models.
Thank You Server Operations Team
==========================================
We are a bit behind schedule but we are about 95% complete
- Date - 10/14/2016 10:01 - 10/19/2016 16:01
- Last Updated - 10/15/2016 06:58
Midwest Data center expansion (Resolved)
- Priority - High
-
AUG 1 2016, we have expanded our Midwest Data center.
- Yes we have done it again.
- All Midwest data center clients we have expanded our Midwest Data center.
- We have added additional Rack/floor Space and increased our network capacity by a Factor of 1.5.
- During the first week Aug 2016 some servers will require relocation we will send out a notification if your server requires relocation.
We will be starting our expansion on Aug 1 2016, and sending out notifications if your server requires a move. This will be short notice relocation/reboot.
===============================================
We are about 95% complete had an Issue with one of our XEN nodes it failed to Boot operating system had to be reinstalled
==================================================
Upgrades complete
- Date - 08/01/2016 15:00 - 08/11/2016 08:04
- Last Updated - 08/11/2016 08:04
Las Vegas shared hosting server Maint (Resolved)
- Priority - Low
- Affecting Server - rc1
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 06/06/2016
EVENT START TIME: 07:59 PST
EVENT END TIME: 16:00 PST
EVENT DETAILS: Emergency Maintenance Shared Hosting server
At 07:59 PST We will be replacing a Failing Backup drive in our Rc1 Shard Hosting server.
This is a low impact event and we do not expect any Downtime if this should change we will update as required
========
The failed Hard drive has been replaced we had to reboot the server as it did not recognize the new Drive
- Date - 06/06/2016 08:06 - 06/18/2016 13:46
- Last Updated - 06/06/2016 11:26
VPS Servers Maint Patch (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 5/7/2016
EVENT START TIME: 00:01 EDT
EVENT END DATE: 5/8/2016
EVENT END TIME: 06:00 EDT
EVENT LOCATION: SB NOC (Midwest)
EXPECTED DOWN TIME: 5-15 MIN
EVENT DETAILS:
We will be performing server maintenance: on all of KVM Nodes. Reason: server security & maintenance patch.
We will be performing server maintenance: on all of OPenVZ/Viruozzo Nodes. Reason: server security & maintenance patch.
Maintenance Window: 5/7/2016 00:01 - 5/8/2016 0600:00 PM EDT
Thank You Server Operations Team
=================================================
Maint complete no issues
- Date - 05/06/2016 10:01 - 05/17/2016 16:49
- Last Updated - 05/08/2016 08:26
Degraded On Xen Node 6 (Resolved)
- Priority - Low
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 04/24/2016
EVENT START TIME: 07:59 PST
EVENT END TIME: 09:20 PST
EVENT DETAILS: Emergency Maintenance XEN 6
At 07:59 PST a hard drive popped out of our Raid array and caused a lock up
The Technician on Duty immediately noticed the error and rebooted the raid is currently rebuilding
========
We will monitor the rebuilding of the array and close this issue when it is complete
=============================================================
Raid rebuuild is complete
- Date - 04/24/2016 07:59 - 04/29/2016 10:46
- Last Updated - 04/24/2016 13:29
VPS Servers Maint Patch (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 4/17/2015
EVENT START TIME: 00:01 EDT
EVENT END DATE: 4/18/2015
EVENT END TIME: 06:00 EDT
EVENT LOCATION: SB NOC (Midwest)
EXPECTED DOWN TIME: 5-15 MIN 2 Instances
EVENT DETAILS:
We will be performing server maintenance: on all of KVM Nodes. Reason: server security & maintenance patch.
We will be performing server maintenance: on all of OPenVZ/Viruozzo Nodes. Reason: server security & maintenance patch.
Maintenance Window: 4/17/2015 00:01 - 4/18/2015 0600:00 PM EDT
Thank You Server Operations Team
====================================
This maint window has been canceled
- Date - 04/16/2016 21:01 - 04/19/2016 10:19
- Last Updated - 04/17/2016 07:24
Upgrading our shared Hosting servers to the l (Resolved)
- Priority - Low
- Affecting Other - SB NOC (Midwest) Las Vegas NOC (West Coast)
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 3/05/2016
EVENT START TIME: 13:00 PST
EVENT END DATE: 03/19/2016
EVENT END TIME: 06:00 PST
EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)
EXPECTED DOWN TIME: 1-5 minutes when we cut the network over
EVENT DETAILS:
Maintenance Scheduled for Midwest & West Coast Data centers
Maintenance window: 5 March 2016 13:00 PST to 19 Mar 2016 06:00 PST
Maintenance Window Duration: 2 weeks
Maintenance Upgrading all shared Hosting servers to the latest OS Centos 7
Site Affected: Shared Hosting servers ALL
Expected Down time 1-5 minutes
Risk Classification: LOW
Empire Hosting Master Ticket #: 20160305
===========================
Upgrades complete No issues
- Date - 03/05/2016 11:04 - 03/15/2016 07:32
- Last Updated - 03/15/2016 07:32
informational Notice XEN 2 (Resolved)
- Priority - Low
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 02/23/2016
EVENT START TIME: 21:12 PST
EVENT END TIME: NA
EVENT DETAILS: informational Notice XEN 2
At 21:12 PST a client on Xen Node 2 reported that their server was not coming up after a reboot
After further investigation it was found that the Network Bridge lost connection (the server was running but no network).
We have reconfigured the Bridge and restarted the network those servers that we noticed that lost connection we rebooted. If you cannot connect to your servers IP please do a reboot will reconnect your network.
============================================================
No more issues closing
- Date - 02/23/2016 22:34 - 03/05/2016 10:51
- Last Updated - 03/05/2016 10:51
Network Event (Resolved)
- Priority - Critical
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 02/09/2016
EVENT START TIME: 09:00 PST
EVENT END TIME: 09:20 PST
EVENT DETAILS: Emergency Maintenance
At 09:00 PST we had a Network Event in our Mid West Data center
We have investigated to or Level3 transit was null routing all of our IP routes to un-routable inbound and outbound
========
After further investigation we had to power cycle the edge router and bring up our backup Peers manually.
We are currently on our primary Level3 Feed monitoring the situation Network restored @ 09:20 PST
=================================================================
resolved
- Date - 02/09/2016 09:00 - 02/10/2016 13:55
- Last Updated - 02/10/2016 13:55
KVM Emergency Maintenance (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 12/26/2015
EVENT START TIME: 4:04 PST
EVENT DETAILS: Emergency Maintenance
KVM 1
We have had reports of VM’s failing to boot
We have investigated it to a software glitch on the we are currently migrating the affected VMS to another node
========
After further investigation the affected VM's file systems went read only and would not stay on line long enough to to run a FSCK.
corrected the issue by moving disk partitions to another Node and ran manual FSCK's
=================================================================
resolved
- Date - 01/26/2016 16:05 - 01/27/2016 01:50
- Last Updated - 01/27/2016 01:50
Emergency Maintenance XEN2 (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 1/22/2016
EVENT START TIME: 8:17 PDT
EVENT end TIME: 8:21 PDT
EVENT DETAILS: Emergency Maintenance
Xen 2
We have had reports of some client VPS servers hanging on reboot
We have restarted the node to clear the error
========
After further investigation it was found 2 Virtual servers were casing the issue 1 the first VPS that was hung required a file system check and another VPS was abusing the disk IO it has been suspended.
- Date - 01/22/2016 20:17 - 01/25/2016 03:37
- Last Updated - 01/22/2016 21:29
Shared hosting servers exploited (Resolved)
- Priority - Low
- Affecting System - Shared Hosting Server
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 12/25/2015
EVENT START TIME: 2:00 PDT
EVENT DETAILS: Emergency Maintenance
Hared Hosting Server(s)
We have had reports of some client sites being redirected to a Hacked site page
We are currently have investigated the hacked sites and have determined that that 2 redirects have been place in the server one in the public html and one in the Web server configuration the apache configuration appears to be exploited. We are currently restoring from Backup.
========
Update Server has been restored form Backup
Old server is still intact removed from the network
Changing Status to Low as everything is comming back on line
==========================================================
All Data restored server running without issue
===================
Note This issues is 100% resolved if you were affected we have preparred a ROH please open a ticket
and request a copy
- Date - 12/25/2015 02:00 - 12/28/2015 20:52
- Last Updated - 12/29/2015 09:55
Planned partial power outage: (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 11/3/2015
EVENT START TIME: 22:00 EST
EVENT END DATE: 11/4/2015
EVENT END TIME: 03:00 EST
EVENT LOCATION: SB NOC (Midwest)
EXPECTED DOWN TIME: 15-45 Minutes
EVENT DETAILS:
Planed Partial Power Outage: We will be replacing a defective Power distribution unit on circuit A1A. During this planed maintenance your server will be inaccessible. The extended window is to allow for the scheduling of the electrician.
Please contact us at https://secure.empire-hosting.net/submitticket.php if you have any questions or concerns.
Thank You Network Operations Team
====================================
Maintenance is underway
================================
Maintentence complelete
No Issues
- Date - 11/03/2015 18:00 - 11/22/2015 15:02
- Last Updated - 11/03/2015 23:34
Planned power outage: (Resolved)
- Priority - Low
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 10/31/2015
EVENT START TIME: 08:00 EDT
EVENT END DATE: 10/31/2015
EVENT END TIME: 12:00 EDT
EVENT LOCATION: SB NOC (Midwest)
EXPECTED DOWN TIME: 0
EVENT DETAILS:
The power company had just informed us that there is a planned power outage from 8am to 12pm Saturday. Our UPSs and generators will be ready to take over the load. However some clients only have b power. There's a possibly some of these systems will bounce as the main and backup power switches.
AEP did not give us any details on why there will be an outage other than a being a maintenance window.
Please contact us at https://secure.empire-hosting.net/submitticket.php if you have any questions or concerns.
Thank You Network Operations Team
=======================================================
No issues remained on Generator power until utility company restored power
Closing
- Date - 10/31/2015 05:00 - 11/01/2015 10:13
- Last Updated - 11/01/2015 10:12
DDOS Attack Midwest Shared hosting server. (Resolved)
- Priority - Low
- Affecting Server - Columbia
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 10/21/2015
EVENT START TIME: 15:06 PDT
EVENT END TIME: 15:20 PDT
EVENT LOCATION:Midwest NOC
EVENT DETAILS:
One of our shared hosting servers ins under a layer7 DDOS attack RC2/Columbia attackers are flooding the inbound network connection
We are currently mitigating the the attack at this time:
Note: we will be making some modifications to the network on this server it will be unreachable for about 5 minutes
Note: some of you who are are reviving the attack your domains will be redirected and you will be notified in aseparate Email
============================
the network has been adjusted
========================
The attack has been mitigated if your web site is down you are one of the unlucky ones that was under a intense flood
Please see the ticket we opened for you on your behalf
====================================
Resolved
- Date - 10/21/2015 15:58 - 10/22/2015 22:42
- Last Updated - 10/22/2015 22:42
KVM 3 Emergency Maintenance (Resolved)
- Priority - Critical
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 8/26/2015
EVENT START TIME: 09:20 PDT
EVENT LOCATION: SB NOC (Midwest)
EVENT DETAILS: Emergency Maintenance
VPS Server Node KVM 3
Is currently having some issues the server is stuck at the boot menu
We are currently investigating
========
Update 10:46 PDT
This issue has been resolved
The server was a total Loss (would not boot) we restored all Virtual machines to another server 0 data loss. Pulling old server down to see what failed.
=================================================================================
- Date - 08/27/2015 09:20 - 09/03/2015 09:59
- Last Updated - 08/28/2015 06:48
KVM Server OS Upgrade & security patch (Resolved)
- Priority - High
- Affecting Server - VPS Master
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 8/24/2015
EVENT START TIME: 23:00 PDT
EVENT END DATE: 08/27/2015
EVENT END TIME: 06:00 PDT
EVENT LOCATION: SB NOC (Midwest)
EXPECTED DOWN TIME: 20 Minutes
EVENT DETAILS:
Maintenance Scheduled for Midwest Data centers
Maintenance window: 24 Aug 2015 23:00 PDT to 27 Aug 2015 06:00 EDT
Maintenance Window Duration: 3 Day
Maintenance KVM Server OS Upgrade & security patch
Site Affected: KVM All Nodes
Expected Down time per Node 10 - 25 Minutes
Risk Classification: high
Empire Hosting Master Ticket #: 20150824
=========================
All updates complete no issues
- Date - 08/24/2015 23:00 - 08/27/2015 13:09
- Last Updated - 08/27/2015 13:09
Virtuozzo 4.7 End of Life (Resolved)
- Priority - Critical
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 8/23/2015
EVENT START TIME: 0:15 PDT
EVENT END DATE: 9/31/2015
EVENT END TIME: TBA
EVENT LOCATION: Midwest NOC
EVENT DETAILS:
All Virtuozzo nodes:
Virtuozzo 4.7 End of Life is September 2015.
There is no replacement Parallels/Odin has discontinued the Virtuozzo line. We will be migrating our existing Virtuozzo infrastructure to Virtuozzo OpenVZ.
There will be no changes to your plans or resources the only changes will be new control panel access (more features) and the latest Virtualization file system which you may see a performance increase.
================================================
Upgrades complete no issues
- Date - 08/23/2015 06:56 - 08/26/2015 10:16
- Last Updated - 08/26/2015 10:15
DDOS Attack Midwest Shared hosting server. (Resolved)
- Priority - High
- Affecting Server - Columbia
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 8/12/2015
EVENT START TIME: 8:15 PST
EVENT LOCATION:Midwest NOC
EVENT DETAILS:
One of our shared hosting servers ins under a layer7 DDOS attack RC2/Columbia attackers are flooding the inbound network connection
We are currently investigating and mitigating the the attack at this time
Thank You network Operations Team
============================
Update
The DDos accttack has been mitigated we will leave this issue open and monitor for 12 more hours
======================================================
Resolved
- Date - 08/12/2015 20:22 - 08/13/2015 10:27
- Last Updated - 08/13/2015 10:27
Circuit 1A1 Power Fault (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 7/31/2015
EVENT START TIME: 02:45 PDT
EVENT END DATE: 7/31/2015
EVENT END TIME: TBA
EVENT LOCATION: SB NOC (Midwest)
EVENT DETAILS:
Sorry for the blanket email this only affects select servers
At approximately 2:45 PDT we lost power to one of circuits in our mid west facility some servers were power cycling until we could bring the whole system down and bring each server up 1 at a time currently power is stable and all affected servers are on line we will continue to montor the system and update if we need to perform any additional maintenance.
Thank You NOC Operations Team
- Date - 07/31/2015 02:45 - 08/02/2015 11:55
- Last Updated - 07/31/2015 03:13
XEN 9 Hard Drive replacemnt (Resolved)
- Priority - Critical
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 7/6/2015
EVENT START TIME: 10:00 PDT
EVENT END DATE: 7/6/2015
EVENT END TIME: 18:00 PDT
EVENT LOCATION: SB NOC (Midwest)
EXPECTED DOWN TIME: 5 MIN
EVENT DETAILS:
We will be performing Emergency server maintenance: XEN 9 VPS server: During a routine Inspection on this server it was found that a hard drive is failing. Being this is the 4th hard drive that has failed on this particular machine in less than a year we are going pull this server down for extended maintenance and replace the entire array. We will be pulling a live snapshot of your server and moving it to a new hardware node.
Maintenance Window: 7/6/2015 10:00 - 7/6/2015 18:00 PDT
Thank You Server Operations Team
Last Updated - 07/03/2015 16:56
=============================
Completed No issues
- Date - 07/06/2015 10:00 - 07/11/2015 09:46
- Last Updated - 07/11/2015 09:46
Shared Hosting servers Maint Patch (Resolved)
- Priority - Low
- Affecting Server - rc1
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 7/5/2015
EVENT START TIME: 00:01 EDT
EVENT END DATE: 7/5/2015
EVENT END TIME: 03:00 EDT
EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)
EXPECTED DOWN TIME: 5 MIN
EVENT DETAILS:
We will be performing server maintenance: on all of shared hosting servers. Reason: server security & maintenance patch.
Maintenance Window: 7/5/2015 00:01 - 7/5/2015 0300:00 PM EDT
Thank You Server Operations Team
===========================================================
Maint complete No issues
- Date - 07/05/2015 02:00 - 07/11/2015 09:46
- Last Updated - 07/06/2015 09:28
XEN/KVM OS Upgrade (Resolved)
- Priority - High
- Affecting Other - XEN/KVM OS Upgrade
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 5/10/2015
EVENT START TIME: 06:00 PDT
EVENT END DATE: 5/14/2015
EVENT END TIME: 08:00 PDT
EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)
EXPECTED DOWN TIME: 3MIN - 25 MIN
EVENT 1 DETAILS:
Our XEN/KVM Software vender has released some New storage features. To allow for use we will be converting all of our XEN & KVM machines to a new configuration that will allow for the use of Shared & Multi storage arrays for future expansion. Some of the vm Configurations are on an old configuration and cannot take advantage of this and the new configuration breaks our Backup configuration & NRT Migration. We will be doing most of the maintenance between 12:00 EDT - 3:00 EDT each Day.
EVENT 2 DETAILS: Start Date as posted above patch as needed per conversion
Maintenance XEN Server OS Upgrade & security patch
Site Affected: All XEN & KVM Nodes.
========================================
Upgrade completed
- Date - 05/10/2015 05:46 - 05/18/2015 07:38
- Last Updated - 05/18/2015 07:38
Backup Server(s) Maint. (Resolved)
- Priority - Low
- Affecting Other - Backup Server(s) Maint.
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 5/10/2015
EVENT START TIME: 06:00 PDT
EVENT END DATE: 5/14/2015
EVENT END TIME: 08:00 PDT
EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)
EXPECTED DOWN TIME: NA
EVENT DETAILS:
Per the release of Our XEN/KVM Software with new storage features. Our Backup servers will be off line or in a limited capacity to accommodate an update of our backup procedures. Please make a note of it
==============================
Las Vegas NOC has been completed
==============================
100% complete
- Date - 05/10/2015 06:24 - 05/15/2015 04:34
- Last Updated - 05/15/2015 04:34
Shared Hosting servers Maint (Resolved)
- Priority - Low
- Affecting Server - Columbia
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 5/10/2015
EVENT START TIME: 06:00 PDT
EVENT END DATE: 5/14/2015
EVENT END TIME: 08:00 PDT
EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)
EXPECTED DOWN TIME: 30 seconds -25 Minutes
EVENT DETAILS: We will be making some changes to our Backup system for all of our shared hosting servers. During this time our R1 soft interface will be down: Normally we would not send out an email for only 30 seconds of network interruption but there is a chance we may need to shut one or more servers down longer to accommodate the change.
============
Maint Complete no issues
- Date - 05/10/2015 06:37 - 05/13/2015 01:27
- Last Updated - 05/13/2015 01:27
Virtuozzo VPS server Maint Patch (Resolved)
- Priority - Critical
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 5/12/2015
EVENT START TIME: 10:00 PDT
EVENT END DATE: 05/14/2014
EVENT END TIME: 06:00 PDT
EVENT LOCATION: SB NOC (Midwest)
EXPECTED DOWN TIME: 20 Minutes
EVENT DETAILS:
Maintenance Scheduled for Midwest Data centers
Maintenance window: 12 May 2015 00:01 EDT to 14 May 2015 06:00 EDT
Maintenance Window Duration: 2 Day
Maintenance Virtuozzo Server OS Upgrade & security patch
Site Affected: Virtouzzo All Nodes
Expected Down time per Node 10 - 25 Minutes
Risk Classification: high
Empire Hosting Master Ticket #: 20150510
Thank You Server Operations Team
========================================
Maintence complete No issues
- Date - 05/12/2015 06:51 - 05/13/2015 01:27
- Last Updated - 05/13/2015 01:27
Shared Hosting servers Maint Patch (Resolved)
- Priority - Medium
- Affecting System - All Shared Hosting Servers
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 4/28/2015
EVENT START TIME: 23:00 PST
EVENT END DATE: 4/29/2015
EVENT END TIME: 08:00 PST
EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)
EXPECTED DOWN TIME: 5 MIN
EVENT DETAILS:
We will be performing server maintenance: on all of shared hosting servers. Reason: server security & maintenance patch.
Maintenance Window: 4/28/2015 23:00 - 4/29/2015 08:00 PM PST
Thank You Server Operations Team
========
Maint complete no issus
- Date - 04/28/2015 23:00 - 04/29/2015 06:47
- Last Updated - 04/29/2015 06:47
XEN 1 Network Maint: (Resolved)
- Priority - Critical
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our
data-centers nationwide. This does not necessarily mean that your facility or server will be affected.
All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 04/10/2015
EVENT START TIME: 11:12 PST
EVENT END DATE:04/10/2015
EVENT END TIME: 12:10 PST
EVENT LOCATION: Midwest NOC
EXPECTED DOWN TIME: 20 Min
EVENT DETAILS:
Emergency Maintenance Scheduled for Midwest Data center
Maintenance window: FRi 10 April 2014 11:20 PST
Maintenance Window Duration: 15 Min
Maintenance Network Failed On Node
Site Affected: XEN Nodes 1
Expected Down time 20 Min
Risk Classification: High
Empire Hosting Master Ticket #: 20150410
The Main Network On this server has failed.
We will be patching the server and rebooting to resolve the issue
Thank You Server Operations Team
====================
Update The network has been Restored
============================
this has been 100% resoved
- Date - 04/10/2015 11:15 - 04/10/2015 17:27
- Last Updated - 04/10/2015 14:23
Maintenance Failed Hard drive replacement (Resolved)
- Priority - Critical
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 3/13/2015
EVENT START TIME: 12:00 PDT
EVENT END DATE: 03/13/215
EVENT END TIME: 16:00PDT
EVENT LOCATION: Las Vegas NOC
EVENT DETAILS:
During a normal Maintenance inspection it was revealed that one of the Hard drive in the raid array has failed.
We will be replacing the defective hard drive which will require a reboot of the Hardware node 10-20 minutes of down time is expected
Thank You Server Operations Team
================================================================================================
Upadte Defective drive has been replaced and the Raid Aray is rebuilding
This issue will remain open until the raid rebuild is complete
==============================================
Rebuild complete closing
- Date - 03/13/2015 09:00 - 03/14/2015 09:18
- Last Updated - 03/22/2015 14:21
Network Issues Las Vegas NOC (Resolved)
- Priority - Low
-
EVENT START DATE: 3/8/2015
EVENT START TIME: 10:45 PDT
EVENT END DATE: 3/8/2015
EVENT END TIME: 10:54 PDT
EVENT LOCATION: Las Vegas NOC
EXPECTED DOWN TIME: NA
EVENT DETAILS:
Risk Classification: Low
Empire Hosting Master Ticket #: 20150308
Our Primary Level3/TW Telecom Primary NTT Primary Route is has Gone down which is causing it to Flap with our Backup Feed we have routed all traffic away to our Backup until Our upstream transit providers can resolve their their issues. This will be a No impact event.
======================================================
Update Our primary transit routes have come back on line
We are going to remain on our Backup Feed Until we get and update from our upstream
================================================================
issue resolved
- Date - 03/08/2015 10:45 - 03/08/2015 12:13
- Last Updated - 03/08/2015 11:35
Drive Maint VPS 4 (Resolved)
- Priority - Critical
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 2/14/2015
EVENT START TIME: 13:00 PST
EVENT END DATE: 2/14/2015
EVENT END TIME: 15:30 PST
EVENT LOCATION: SB NOC (Midwest)
EXPECTED DOWN TIME: 20 Min
EVENT DETAILS:
We will be doing Emergency Drive Maintenance on Virtuozzo server VPS4
The Partitions are having drive errors and require a repair
Thank You Server Operations Team
============================
Update we are moving the time up on this action to 13:00 PST
=======================================================
Hard drive partitions have been repaired No issues
- Date - 02/14/2015 13:00 - 02/14/2015 12:52
- Last Updated - 02/14/2015 12:52
DDOS attack Las Vegas Data center (Resolved)
- Priority - Critical
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 2/7/2015
EVENT START TIME: 7:25 PST
EVENT LOCATION: LV NOC (West Cost)
EVENT DETAILS:
XEN node 6 is currently Under a DDOS attack flooding the inbound network connection
We are currently investigating and mitigating the the attack at this time
Thank You network Operations Team
============================
the Attack has been Mitigated and the attacking IP's have benn Null Routed
==============================================
- Date - 02/07/2015 07:23 - 02/07/2015 21:44
- Last Updated - 02/07/2015 07:36
SSD Raid Maint on VPS nodes (Resolved)
- Priority - Low
- Affecting Other - SSD VPS Nodes
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 1/31/2015
EVENT START TIME: 15:00 PST
EVENT END DATE: 2/02/2015
EVENT END TIME: 06:00 PST
EVENT LOCATION: SB NOC (Midwest)
EXPECTED DOWN TIME: 30 Seconds
EVENT DETAILS:
We will be doing Raid Maintenance on all Hardware Raid SSD machines in our Midwest NOC
this event will a a low impact event 30 seconds of down time 2 instances while we rearrange our raid configuration to increase performance.
Thank You Server Operations Team
============================
Maint completed No issues
- Date - 01/31/2015 15:00 - 02/02/2015 17:55
- Last Updated - 02/02/2015 17:54
XEN Maint Patch & Security Update (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 1/27/2015
EVENT START TIME: 17:39 PST
EVENT END DATE: 1/28/2015 (time pushed up 2 days)
EVENT END TIME: 08:00 PST
EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)
EXPECTED DOWN TIME: 10 MIN
EVENT 1 DETAILS:
A forced update from one of has software venders made our XEND instance shut down
We will be rebooting XEN 1 XEN 6 XEN 4 Investigating the rest of the XEN Nodes.
EVENT 2 DETAILS: Start Date 1/31/2015 00:00 PST
Maintenance XEN Server OS Upgrade & security patch
Site Affected: XEN Nodes 2, 7, 8, 10, 12
==============================================================
Affected Nodes 1, 6, 4 patched No issues
===================================================================
the Tme line on this has been pushed up forced update to all of our XEN nodes were pushed by our software vender
===============================================================================
XEN Nodes 2, 7, 8, 10, 12 patched No issues
- Date - 01/27/2015 18:37 - 01/31/2015 04:18
- Last Updated - 01/28/2015 07:20
Mid west Data Center On Generator Power (Resolved)
- Priority - Medium
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
Mid west Data Center On Generator Power
EVENT START DATE: 1/25/2014
EVENT START TIME: 18:36 PST
EVENT LOCATION: Midwest NOC
EXPECTED DOWN TIME: NA
EVENT DETAILS:
major weather related event in our Mid West data center has force us onto Generator power
a few servers in on one of our power feeds power cycled them selves everything has been powered back on
if you are having issues please open a support ticket.
This issue will remain open until we go back on utility power
==============================================
utility power has been restored
Closing Issue
- Date - 01/25/2015 19:41 - 01/26/2015 11:22
- Last Updated - 01/25/2015 20:54
XEN 7 Network Maint: (Resolved)
- Priority - Low
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 1/7/2014
EVENT START TIME: 16:00 PST
EVENT END DATE: 1/8/2014
EVENT END TIME: 15:00 PST
EVENT LOCATION: Midwest NOC
EXPECTED DOWN TIME: 30 Seconds - 1 minute
EVENT DETAILS:
Emergency Maintenance Scheduled for Midwest Data center
Maintenance window: Wed 7 Jan 2014 16:00 PST to Thurs 8 Jan 15:00 PST
Maintenance Window Duration: 12 hour
Maintenance Private Network Bridge failure
Site Affected: XENNodes 7
Expected Down time 30 Secs
Risk Classification: Low
Empire Hosting Master Ticket #: 20141229
Our Private VPN that controls our Midwest Private network has failed on this server .
We will be pulling this server down for extended Maintenance rather than having clients down during this period.
We will be Live Migrating all VPS's to different Nodes
Thank You Server Operations Team
==================================================================================
This maintenance action was delayed because of a hard drive failure on another server took priority
90% complete End time has been updated
this
- Date - 01/07/2015 16:00 - 01/17/2015 10:33
- Last Updated - 01/08/2015 04:23
XEN 9 Hard Drive replacemnt (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 1/7/2014
EVENT START TIME: 16:30 PST
EVENT END DATE: 1/7/2014
EVENT END TIME: 17:30 EDT
EVENT LOCATION: Mid West NOC
EXPECTED DOWN TIME: 10 Minutes
EVENT DETAILS:
Emergency Maintenance Scheduled for Midwest Nevada Data center
Maintenance window: Wed 7 Jan 2014 16:30 PST to 17:30 PST
Maintenance Window Duration: 2 hour
Maintenance Failed Hard drive replacement
Site Affected: XENNodes 9
Expected Down time 10 15 Minutes
Risk Classification: high
Empire Hosting Master Ticket #: 2016107
During a normal Maintenance inspection it was revealed that one of the Hard drive in the raid array has failed.
We will be replacing the defective hard drive which will require a reboot of the Hardware node 10 minutes of down time is expected
Thank You Server Operations Team
===============================
UPDATE:
The Raid array is failing to build
========================
We will be migrating All VPS off of this server and pulling the server down and rebuilding the array from scratch
Expect another reboot of your servers
===========================================
Raid has been rebuilt and is functioning normally
Downgrading to Low until new backups sync
=========================================
Backups syncing closing issue
- Date - 01/07/2015 16:30 - 01/08/2015 04:17
- Last Updated - 01/08/2015 04:17
XEN 6 Hard Drive replacemnt (Resolved)
- Priority - Critical
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 12/29/2014
EVENT START TIME: 06:01 PST
EVENT END DATE: 12/29/2014
EVENT END TIME: 06:00 EDT
EVENT LOCATION: Las Vegas NOC (West Coast)
EXPECTED DOWN TIME: 10 Minutes
EVENT DETAILS:
Emergency Maintenance Scheduled for Las Vegas Nevada Data centers
Maintenance window: Monday 29 Dec 2014 06:00 PST to 12:00 PST
Maintenance Window Duration: 6 hour
Maintenance Failed Hard drive replacement
Site Affected: XENNodes 6
Expected Down time 10 15 Minutes
Risk Classification: high
Empire Hosting Master Ticket #: 20141229
During a normal Maintenance inspection it was revealed that one of the Hard drive in the raid array has failed.
We will be replacing the defective hard drive which will require a reboot of the Hardware node 5 minutes of down time is expected
Thank You Server Operations Team
=============================
closed no issues
- Date - 12/29/2014 06:20 - 01/02/2015 10:49
- Last Updated - 01/02/2015 10:49
CVE-2014-9322 Patch (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 12/19/2014
EVENT START TIME: 00:01 EDT
EVENT END DATE: 12/19/2014
EVENT END TIME: 06:00 EDT
EVENT LOCATION: SB NOC (Midwest) Las Vegas NOC (West Coast)
EXPECTED DOWN TIME: 10 Minutes
EVENT DETAILS:
Maintenance Scheduled for Midwest & West Coast Data centers
Maintenance window: Friday 19 Dec 2014 00:01 EDTto 06:00 EDT
Maintenance Window Duration: 6 hour
Maintenance Shared Hosting Servers OS Upgrade & CVE-2014-9322 patch
Site Affected: Shared Hosting Servers ALL
Expected Down time 10 15 Minutes
Maintenance Scheduled for Midwest & West Coast Data centers
Maintenance window: Friday 19 Dec 2014 00:01 EDTto 23:59 EDT
Maintenance Window Duration: 24 hour
Maintenance Managed XEN Servers OS Upgrade & CVE-2014-9322 patch
Site Affected: Shared Hosting Servers ALL
Expected Down time 10 15 Minutes
Maintenance Scheduled for Midwest Data centers
Maintenance window: Friday 19 Dec 2014 00:01 EDTto 06:00 EDT
Maintenance Window Duration: 6 hour
Maintenance Managed Virtuozzo Servers OS Upgrade & CVE-2014-9322 patch
Site Affected: Shared Hosting Servers ALL
Expected Down time 10 15 Minutes
Maintenance Scheduled for Midwest & West Coast Data centers
Maintenance window: Friday 19 Dec 2014 00:01 EDTto 23.59 EDT
Maintenance Window Duration: 24 hour
Maintenance Managed Dedicated Servers OS Upgrade & CVE-2014-9322 patch
Site Affected: Shared Hosting Servers ALL
Expected Down time 10 15 Minutes
Risk Classification: high
Empire Hosting Master Ticket #: 20141218
Thank You Server Operations Team
====================================
update Dec 19 02:18 EST
All shared Hosting Servers Completed
All Virtuozzo servers completed
Client Dedicated servers 10% done in progress
Client Xen VPS servers 1% done in progress
- Date - 12/19/2014 22:01 - 12/26/2014 05:24
- Last Updated - 12/18/2014 23:22
XEN Maint Patch & Security Update (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 11/09/2014
EVENT START TIME: 00:01 EDT
EVENT END DATE: 11/09/2014
EVENT END TIME: 06:00 EDT
EVENT LOCATION: Las Vegas NOC
EXPECTED DOWN TIME: 15 Minutes
EVENT DETAILS:
Maintenance Scheduled for Las Vegas Nevada Data centers
Maintenance window: Monday 9 Nov 2014 00:01 EDTto 06:00 EDT
Maintenance Window Duration: 6 hour
Maintenance Xen Server OS Upgrade & security patch
Site Affected: XENNodes 2, 4, 6, 8, 10
Expected Down time per Node 10 15 Minutes
Risk Classification: high
Empire Hosting Master Ticket #: 2014081
Thank You Server Operations Team
======================================
Updates completed XEN 4,8,10
Will reschedule xen 2 & 6 as it was not completed within the maintenance window
Those effected will Be emailed for the new time
====================================================
All Updates complete No issues
- Date - 11/09/2014 00:01 - 11/15/2014 03:10
- Last Updated - 11/11/2014 05:31
Virtuozzo Maint Patch & Security Update (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 11/09/2014
EVENT START TIME: 00:01 EDT
EVENT END DATE: 11/09/2014
EVENT END TIME: 06:00 EDT
EVENT LOCATION: SB NOC (Midwest)
EXPECTED DOWN TIME: 20 Minutes
EVENT DETAILS:
Maintenance Scheduled for Midwest Data centers
Maintenance window: Sunday 9 Nov 2014 00:01 EDTto 06:00 EDT
Maintenance Window Duration: 6 hour
Maintenance Virtuozzo Server OS Upgrade & security patch
Site Affected: Virtouzzo Nodes 4, 7
Expected Down time per Node 10 25 Minutes
Risk Classification: high
Empire Hosting Master Ticket #: 2014082
Thank You Server Operations Team
=====================================
Update completed No issues
- Date - 11/09/2014 00:01 - 11/09/2014 05:14
- Last Updated - 11/09/2014 05:13
Level3 Network Issues (Resolved)
- Priority - Low
-
EVENT START DATE: 9/9/2014
EVENT START TIME: 10:00 PDT
EVENT END DATE: UNK
EVENT END TIME: UNK
EVENT LOCATION: Midwest NOC
EXPECTED DOWN TIME: NA
EVENT DETAILS:
Risk Classification: Low
Empire Hosting Master Ticket #: 201409
Our Primary Level3 Route is has some congestion issues which is causing it to Flap with our Primary Internap Feed we have routed all traffic away to our Backup until Level3 can resolve the issues. This will be a No impact event.
Thank You Network Operations Team
================================
Update Level3 has resolved their Issue
- Date - 09/09/2014 10:06 - 09/09/2014 11:35
- Last Updated - 09/09/2014 11:35
EMERGENCY SCHEDULED MAINTENANCE (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 7/15/2014
EVENT START TIME: 07:00 PDT
EVENT END DATE: 07/18/2014
EVENT END TIME: 23:00 PDT
EVENT LOCATION: Midwest NOC
EXPECTED DOWN TIME: 10 Minutes
EVENT DETAILS:
Maintenance Scheduled for Midwest Data center
Maintenance window: Tues 15 July 2014 07:00 PDT to Friday 18 July 23:00 PDT
Maintenance Window Duration: 4 days
Maintenance Failing pair of Hard Drives
Site Affected: XEN Nodes 9
Expected Down time 1 Reboot to new Server
Risk Classification: high
Empire Hosting Master Ticket #: 201408
During a routine Drive inspection it was found that this server has 2 drives about to fail. There is currently there is no issues the raid array is continuing to function as expected, But due to the nature that 2 drives are failing simultaneously that are in the same mirror we are electing to move all data off as a precaution just in case of data corruption.
Thank You Server Operations Team
=======================================
No issues closing
- Date - 07/15/2014 07:07 - 07/19/2014 13:30
- Last Updated - 07/19/2014 13:26
Network Upgrades to 1G (Resolved)
- Priority - Critical
-
EVENT START DATE: 6/04/2014
EVENT START TIME: 09:58 PDT
EVENT END DATE: 06/18/2014
EVENT END TIME: 10:04PDT
EVENT LOCATION: Midwest NOC
EVENT DETAILS:
We will Be Expanding our network capacity at our Mid West Data Center June 4, 2014 – June 18, 2014. All Network connections will be 1G Minimum form a 10G backbone.
This is follow up to our Network Upgrades “May 2014 Announcements Network Upgrades to 1G”
We will Be Expanding our network capacity at our Mid West Data Center June 1, 2014 – June 18, 2014. All Network connections will be 1G Minimum form a 10G backbone.
Dedicated Server clients only: Beginning June 4, 2014 we will begin migrating existing clients tour All 1 G Network.
Dedicated Server Clients you now have the Option of 100Mbs Metered, 100Mbs unmetered, and 1Gbs Metered. Per Usual 100M & 1G dedicated line are still available as always. You may notice Some Slight latency during the week of Jun 8 while we cut over to the new Network.
VPS Server clients only: Beginning June 10, 2014 we will begin migrating existing clients to All 1 G Network.
For XEN VPS clients you now have the option of 10Mbs unmetered, 100Mbs metered, 100Mbs unmetered, and 1Gbs metered. Virtouzzo you will have the option of 100Mbs & 100Mbs unmetered. You may notice Some Slight latency during the week of Jun 8 while we cut over to the new Network. XEN additional: we will kill 2 birds with one stone and upgrading our few legacy XEN nodes that are still XEN 3 to XEN 4.
Shared Hosting & Shared Hosting Resellers only: Shared Hosting clients will remain on 100Mbs& burstable to 1G. You may notice Some Slight latency during the week of Jun 8 while we cut over to the new Network.
=====================================================================================
Update 90% complete
=====================================================================================
This is 100% complete leaving issue open because of some Lingering issues with Level3 Transit
- Date - 06/04/2014 08:01 - 07/19/2014 13:26
- Last Updated - 07/15/2014 07:26
EMERGENCY SCHEDULED MAINTENANCE (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 5/19/2014
EVENT START TIME: 09:58 PDT
EVENT END DATE: 05/19/2014
EVENT END TIME: 10:04PDT
EVENT LOCATION: Las Vegas NOC
EVENT DETAILS:
XEN Node 6 Locked up & required a emergency reboot
Server has been rebooted and all VPS server are on line
Risk Classification: high
We will continue to monitor the Node for any other anomalies
Thank You Server Operations Team
- Date - 05/19/2014 10:58 - 05/21/2014 05:47
- Last Updated - 05/19/2014 10:09
XEN Maint Patch & Security Update (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 5/17/2014
EVENT START TIME: 00:01 PDT
EVENT END DATE: 05/17/2014
EVENT END TIME: 23:00 PDT
EVENT LOCATION: Las Vegas NOC
EXPECTED DOWN TIME: 10 Minutes
EVENT DETAILS:
Maintenance Scheduled for Las Vegas Nevada Data centers
Maintenance window: Saturday 17 May 2014 00:01 PDT to 23:00 PDT
Maintenance Window Duration: 11 hour
Maintenance Xen Server OS Upgrade upgrade to XEN 4
Site Affected: XEN Nodes 10,8
Expected Down time 1 Reboot to new Xen 4 image
Risk Classification: high
Empire Hosting Master Ticket #: 201406
Thank You Server Operations Team
=================================
Update 17 May 21:48
Upgrade complete
- Date - 05/17/2014 00:01 - 05/18/2014 10:08
- Last Updated - 05/17/2014 21:48
EMERGENCY SCHEDULED POWER MAINTENANCE (Resolved)
- Priority - Medium
-
EVENT START DATE: 05/15/2014
EVENT START TIME: 08:00 PDT
EVENT END DATE: 05/15/2014
EVENT END TIME: 12:00 PDT
EVENT LOCATION: Las Vegas NOC
EVENT DETAILS:
Maintenance Summary: Primary ATS Replacement
Maintenance Details: Electrical contractors have discovered a critical defect in the existing ATS (Automatic Transfer Switch), which is responsible for transferring power from the utility grid to the backup generator in the event of a utility failure. The defect discovered could potentially cause the unit to malfunction in the event of a power outage, thereby causing power to be interrupted to customer equipment. Due to the potential sever of this problem, we've instructed the electrical contractor to proceed with the replacement on an emergency basis. Our electricians are not able to swap out this piece of equipment while it is still energized, so we will be transferring power load for our primary "A" power feed over to our backup generator for approximately 1-2 hours while the equipment is replaced. We expect this change to be invisible to our customers, however as with any type of power maintenance, unexpected issues can arise which could have an impact on power delivery. As such, we have elected to perform this maintenance during business hours in order to have full availability of all vendors involved, thereby minimizing the impact of any potential problems that could arise.
======================
Maintenance completed no issues
- Date - 05/15/2014 08:00 - 05/16/2014 19:15
- Last Updated - 05/16/2014 19:15
EMERGENCY SCHEDULED MAINTENANCE (Resolved)
- Priority - Low
-
EVENT START DATE: 04/30/2014
EVENT START TIME: 00:00 CDT
EVENT END DATE: 04/30/2014
EVENT END TIME: 06:00 CDT
EVENT LOCATION: Cedar Falls NOC
EVENT DETAILS:
Qwest/Century Link Engineering will be reloading a core router in the Cedar Falls datacenter on advice from vendor to resolve issue with packetloss on router. This will be a service-impacting maintenance — customers in our Cedar Falls, IA datacenter location will be unavailable for approximately 10 minutes while the router reload is completed. This reload is required to restore complete service to the datacenter.
==========================================
Update
The maintenance has been completed No issues noted.
we will be downgrading this issue to low until the end of the maintenance window.
============================================
Closed
- Date - 04/30/2014 00:00 - 04/30/2014 08:48
- Last Updated - 04/30/2014 08:48
XEN Maint Patch & Security Update (Resolved)
- Priority - High
- Affecting Other - Las Vegas XEN Nodes
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 4/2/2014
EVENT START TIME: 00:01 EDT
EVENT END DATE: 4/2/2014
EVENT END TIME: 06:00 EDT
EVENT LOCATION: Las Vegas NOC
EXPECTED DOWN TIME: 15 Minutes
EVENT DETAILS:
Maintenance Scheduled for Las Vegas Nevada Data centers
Maintenance window: Wednesday 2 April 2014 00:01 EDT to 06:00 EDT
Maintenance Window Duration: 6 hour
Maintenance Xen Server OS Upgrade & security patch
Site Affected: XEN Nodes 2,4,6 8,10
Expected Down time per Node 10 – 15 Minutes
Risk Classification: high
Empire Hosting Master Ticket #: 201405
Thank You Server Operations Team
=============================================
Update Updated time window 10 11:30 EDT
EVENT START DATE: 4/6/2014
EVENT START TIME: 00:01 EDT
EVENT END DATE: 4/6/2014
EVENT END TIME: 06:00 EDT
EVENT LOCATION: Las Vegas NOC
EVENT RESCHEDULED: From 4/2/2014
EXPECTED DOWN TIME: 15 Minutes
EVENT DETAILS:
Original Maintenance Scheduled for Las Vegas Nevada Data centers 4/2/14
Has been rescheduled: Sunday 6 April 2014 00:01 EDT to 09:00 EDT.
Maintenance Window Duration: 9 hour
Maintenance Xen Server OS Upgrade & security patch
Site Affected: XEN Nodes 6 & 8 All other Xen Nodes have been completed
Expected Down time per Node 10 – 15 Minutes Risk Classification: high
Empire Hosting Master Ticket #: 201405
Thank You Server Operations Team
==============================
Updated completed
- Date - 04/02/2014 00:01 - 04/07/2014 06:59
- Last Updated - 04/07/2014 06:59
Network Outage Cedar Falls Data Center (Resolved)
- Priority - Low
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 12/13/2013
EVENT START TIME: 04:59 PST
EVENT DETAILS:
At 04:59 PST
Our Cedar Falls Datacenter is currently experiencing a critical loss of connectivity. We are working with our upstream providers to restore this connectivity as soon as possible.
At this time we do not have any more information than this we will update further as soon as we have more information
Thank You,
Datacenter Operations Team
=============================
Update 10:53 PST
Network has Been Fully resoterd
===================================
=============================
Update 13:00 PST
Issue has been downgraded to Low
===================================
=======================================
Lowest Host/Empire Technology LLC Reason for Outage (RFO)
Description of Outage:
On Friday, December 13, 2013 at approximately 04:59 AM PST, Lowest Host Empire Technology LLC experienced a Network disruption do to a failure of our uptream’s internal core routing configuration. Which Caused our core2.CFII Router to drop all IP announcements leaving our network without any type of external routing protocols to enable routing between the core1.CFII Router and other devices within our network. This caused degraded network performance, packet loss, and the inability to reach the outside network.
Our core1.CFII Router handles our upstream BGP sessions as well as all of our peering BGP sessions on the Mid West Exchange. That caused those networks to attempt to send traffic destined for our network down our peering BGP sessions, which were ultimately unreachable due to the failed routing protocols on both the primary and secondary. This condition lasted until the routing equipment was replaced by our upstream providers.
Resolution:
The Faulty Equipment was replaced by our upstream peer. Configuration was restored, and our external routing protocols began operating normally at this time. We have not had a total network failure in our Cedar Falls facility in over 7 years. It seems we have put too much faith in the ability of our upstream to provide uninterrupted connectivity. So we be looking into making some changes to our BGP configuration over the next few weeks in order to make our external routing more fault tolerant in the case of an internal routing failure on the part of our upstream Network providers. We sincerely apologize for any inconvenience that this matter has caused and thank you for your patience as we work to resolve these issues and improve our facility to serve you better.
- Date - 12/13/2013 04:59 - 12/25/2013 20:23
- Last Updated - 12/15/2013 04:10
Hard Drive failure on VPS Nodes (Resolved)
- Priority - High
-
ear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 10/16/2013
EVENT START TIME: 09:00 CDT
EVENT END DATE: 10/16/2013
EVENT END TIME: 10:00 CDT
EVENT LOCATION: Cedar Falls NOC
EXPECTED DOWN TIME: 15 Minutes
EVENT DETAILS:
EMERGENCY MAINT: We have detected a failed hard drive on our xen 1 & virtuozzo 4 server we will be replacing them & rebooting the servers be
Thank You,
Data center Operations
========================================================
Update 10/16 2013 1000 CDT both servers hard drives have been replaced
VPS 4 raid has rebuilt action closed
XEN 1 Is being rebuilt
====================================================================
Update 10/19/ 2013
Xen 1 Has file corruption in some of the VPS's and the raid will not rebuild
We will be cleaning corruption out and rebuilding the array expect a reboot of individual VPS servers No impact
- Date - 10/16/2013 09:00 - 10/25/2013 21:27
- Last Updated - 10/19/2013 20:12
Data Center Maint: (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 10/14/2013
EVENT START TIME: 09:00 CDT
EVENT END DATE: 10/14/2013
EVENT END TIME: 13:00 CDT
EVENT LOCATION: Cedar Falls NOC
EXPECTED DOWN TIME: 15 Minutes
EVENT DETAILS:
We apologize for the blanket notification . EGERNCY MAINT: We have detected a Power Fault in one our power Distribution switches in our cedar falls Data center. We will be pulling the defective unit and replacing it with a new unit during the replacement all Servers connected to this unit will be rebooted.
Thank You,
Datacenter Operations Team
================================
No issues closing
- Date - 10/14/2013 06:02 - 10/15/2013 20:01
- Last Updated - 10/15/2013 20:01
Network Transit Provider MAINT. (Resolved)
- Priority - Medium
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
EVENT START DATE: 09/19/2013
EVENT START TIME: 00:00 CDT
EVENT END DATE: 09/19/2013
EVENT END TIME: 06:00 CDT
EVENT LOCATION: Level 3 connection
EVENT DETAILS:
Lowest Host/Empire Technology LLC has been informed by a Level 3 maintenance on a KC metro circuit. This circuit is one of a pair of redundant circuits — customer impact is expected to be limited to brief connectivity and latency between KC and Cedar Falls.
Thank You,
Datacenter Operations Team
============================
No issues closing
- Date - 09/17/2013 23:00 - 09/18/2013 09:56
- Last Updated - 09/18/2013 09:56
Network Maintenance (Resolved)
- Priority - Critical
- Affecting Other - Nework wide Routing Maintenance:
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Lowest Host/Empire Technology is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
Network Maintenance Scheduled for all network links (excluding Cedar Falls)
Maintenance window: Sunday 13 May 2013 00:01 AM to Friday May 17 06:00 AM Pacific Time
Maintenance Window Duration: 5 Days
Maintenance Network routing modifications
Expected Down time: none additional Notes: this will only affect the 198.102.29.XX net block
Network Engineers will be performing a network link swap as we will be turning down a old circuit (NC –NOC-1) and bringing up a new redundant (LV-NOC-1 & 2) to allow for network expansion. Some of you may experience some light Latency for 6-48 hours until the routes converge (this is a conservative estimate it will likely be a matter of hours or minutes depending on your IP range ).
For XEN server customers only during this time we are going take the time during cut over we are going to retire some of our older hardware nodes and upgrade you to new Hardware. (Live migration).
Thank You Network Operations Team
- Date - 05/13/2013 00:01 - 06/28/2013 13:52
- Last Updated - 05/11/2013 13:07
Transit Provider infrastructure maintenance (Resolved)
- Priority - Medium
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Empire Hosting is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
Maintenance Scheduled for North Carolina, Data centers
Maintenance window: Saturday 6 April 2013 12:00 AM to 0600 AM eastern Time
Maintenance Window Duration: 6 hours
Maintenance Transit provider Maintenance
Site Affected: North Carolina NOC
Expected Down time none
Risk Classification: Med
Empire Hosting Master Ticket #: 550451
From time to time Lowest Host Empire Technology LLC's transit provider partners must perform maintenance on their infrastructure. Charter Communications has notified us of a service impacting maintenance occurring on 4/6/2012 during the hours of 12:00 AM EDT to 6:00 AM EDT.
During the maintenance activities the following components will be impacted:
- Charter Communications 10Gbps Transport Infrastructure – in our North Carolina Facility
- During the maintenance window we will mark down Internet traffic to / from our Atlanta, GA peering points. we will still have multiple 10G peers Via our other carriers Level 3, telia. Tw telcoem & our direct peer to Las Vegas during this maintenance window. Our customers may see brief periods of increased latency and packet loss associated with the shifting of Internet traffic.
If you have any questions or concerns, please contact Support
Sincerely,
Lowest Host/Empire Technology LLC Support
- Date - 04/06/2013 12:56 - 05/11/2013 12:14
- Last Updated - 04/05/2013 09:15
Emergency hard Drive Maint (Resolved)
- Priority - High
- Affecting Server - rc1
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Empire Hosting is listing all upgrades and
changes to our data-centers nationwide. This does not necessarily mean that
your facility or server will be affected. All pertinent information can be
found in the description for each maintenance action to be performed.
Maintenance Scheduled for North Carolina, s Nevada Data centers
Maintenance window: Saturday 9 Mar 2013 10:00 AM to 10:30 AM Pacific
Time
Maintenance Window Duration: 30 Min
Maintenance Hard Drive optimization
Site Affected: RC1 Shared hosting server
Expected Down time 10 15 Minutes
Risk Classification: high
Empire Hosting Master Ticket #: 550450
March 9, 2013 Lowest Host/Empire Technology Technicians
notices that our RC1 shard hosting server was exhibiting slow drive performance
issues. We are attempted to clean up any
corruption the drives may have.
Thank You Server Operations Team
---------------------------------------
Update 13 March 2013
we have upgraded the server to New hardware and a new raid 10 Disk array
we are closing this issue at this time
- Date - 03/09/2013 08:59 - 03/13/2013 09:49
- Last Updated - 03/13/2013 09:49
XEN OS & security patch (Resolved)
- Priority - High
- Affecting Other - XEN Nodes 1-10 17,18
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Empire Hosting is listing all upgrades and
changes to our data-centers nationwide. This does not necessarily mean that
your facility or server will be affected. All pertinent information can be
found in the description for each maintenance action to be performed.
Maintenance Scheduled for Asheville North Carolina, Cedar Falls Iowa Data, Las
Vegas Nevada Data centers
Maintenance window: Saturday 23 Feb 2013 00:01 AM to 06:00 AM Pacific
Time
Maintenance Window Duration: 6 hour
Maintenance Xen Server OS Upgrade & security patch
Site Affected: XEN Nodes 1-10
17,18
Expected Down time per Node 10 15 Minutes with the exception of XEN 5 which will require additional software
installation 30-50 min (estimates)
Risk Classification: high
Empire Hosting Master Ticket #: 550449
Lowest Host/Empire Technology LLC Data center operations will be performing OS
Maint on XEN Nodes 1-10
17,18. We will be performing upgrade between 00:01 AM and 06:00 AM
Pacific on Saturday 23 Feb 2013 is will be a complete shutdown of all
affected nodes the
Thank You Server Operations Team
----------------------------------------
All Updates Complete NO issues
- Date - 02/23/2013 00:01 - 03/09/2013 08:59
- Last Updated - 02/23/2013 04:58
Routine server Maintenance (Resolved)
- Priority - High
-
EVENT START DATE: 11-11-2012
EVENT START TIME:11:00 PM PST
EVENT END DATE: 11-11-2012
EVENT END TIME: : 02:00 PM PST
EVENT IMPACT: Routine server Maintenance
EVENT DETAILS:
Server Maintenance Notice:
We will be performing server maintenance : on our (XEN 2 & XEN 6 ) ,VPS servers.
Reason: VPS NODE security & maintenance patch.
Expected Down time: 15 Min
Maintenance Window: 11:00 AM - 02:00 PM PST (07:00 PM - 10:00 PM GMT)
=========================================
Maintenance complete No issues
- Date - 11/11/2012 11:00 - 11/11/2012 14:24
- Last Updated - 11/11/2012 14:24
Routine server Maintenance (Resolved)
- Priority - High
- Affecting Server - rc1
-
EVENT START DATE: 11-11-2012
EVENT START TIME:11:00 PM PST
EVENT END DATE: 11-11-2012
EVENT END TIME: : 02:00 PM PST
EVENT IMPACT: Routine server Maintenance
EVENT DETAILS:
Server Maintenance Notice:
We will be performing server maintenance : on our (RC1) ,shared hosting server.
Reason: server security & maintenance patch.
Expected Down time: 10Min
Maintenance Window: 11:00 AM - 02:00 PM PST (07:00 PM - 10:00 PM GMT)
==================
Maintenance complete No issues
- Date - 11/11/2012 11:00 - 11/11/2012 14:24
- Last Updated - 11/11/2012 14:22
Network Outage (Resolved)
- Priority - Critical
-
EVENT START DATE: 07-20-2012
EVENT START TIME: 02:15 PM PDT
EVENT END DATE: 7-20-2012
EVENT END TIME: : 04:10 PM PDT
EVENT IMPACT: Connectivity stability in the ALO-North Carolina data center impacting multiple customers.
EVENT DETAILS: At 0215 PDT 07-20-2012 NOC Techs observed
connectivity issues impacting the North Carolina data center. Network
Services is working to identify the source of the issue and a
resolution. Updates will be provided regularly throughout this event and
all efforts are being made to minimize the impact of this issue. Our
apologies for the inconvenience.
Update 7-20-2012 04:19 connectivity has been restored
network Engineers have performed the line card swap and have Cisco support
engaged. Disruptions in Internet connectivity are continuing to occur,
however our engineers hope to restore connectivity and stability,
shortly. Thank you very much
for your patience and understanding.
Update 7-20-2012 5:50 PM
The Internet Connectivity issues our North Carolina data center was
experiencing early today have subsided. Connectivity has been up and stable and we do not anticipate any further
disruptions to service this evening. Our engineers are currently working
with Cisco support in order to determine a root cause.
- Date - 07/20/2012 02:15 - 07/22/2012 08:36
- Last Updated - 07/20/2012 17:52
Ibound DDOS Flood (Resolved)
- Priority - Critical
- Affecting Other - Ibound DDOS Flood
-
EVENT START DATE: 07-12-2012
EVENT START TIME: 11:05 PDT
EVENT END DATE: 07-12-2012
EVENT END TIME: : 11:50 PDT
EVENT DETAILS: At 11:05 PDT 07-12-2012 one of our clients was under a direct inbound DDOS Attack saturating our network equipment
The net work engineers we able to identify the attack and mitigate the attack quickly
- Date - 07/13/2012 03:05 - 07/13/2012 03:06
- Last Updated - 07/13/2012 03:06
Failing Cisco Switch replacement (Resolved)
- Priority - Low
- Affecting Other - Network infrastructure
-
Dear Lowest host Empire Technology LLC Customers,
In order to serve you better, lowest host Empire Technology LLC is listing all upgrades and changes to our datacenters nationwide. Maintenance in other lowest host Empire Technology LLC Data Centers does not necessarily mean that your facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
Emergency Maintenance Scheduled for NOC (North Carolina )
Maintenance Window: 10:00 AM to 01:00 PM PDT GMT -7 WednesdayJuly 11, 2012
Maintenance Window Duration: 3 hour
Maintenance Action: Cisco switch replacement
Site(s) Affected: North Carolina NOC
Risk Classification: High
Lowest host Empire Technology LLC Master Ticket #: 60001
Lowest host Empire Technology LLC operations will be performing maintenance on one of our Cisco switches in NOC (North Carolina ) this WednesdayJuly 11, 2012. During this window, engineers will Be performing a replacement of a failing Cisco switch customers experience 10 15 Minutes of down time while perform this Maintenance. We will perform this work between 10:00 AM to 01:00 PM PDT Time.
Update July 11, 2012 11:50 PDT
Switch replacement is complete no issues
will down grade this to low as we monitor the switch performance
- Date - 07/11/2012 10:00 - 07/13/2012 03:06
- Last Updated - 07/11/2012 11:51
Node reboot (Resolved)
- Priority - High
-
EVENT START DATE: 05-20-2012
EVENT START TIME: 07:25 PDT
EVENT END DATE: 07:30 PDT
EVENT END TIME: : 05-20-2012
EVENT DETAILS: At 06:38:25 PDT 05-20-2012 Client reported that his VPS was unresponsive and would not reboot
the only way to get his server restarted was to reboot the node very little impact 5 minute outage there will be a minor shutdown while quotas are recalculated
- Date - 05/20/2012 07:25 - 05/22/2012 06:32
- Last Updated - 05/20/2012 07:53
Short Notice Power SYS MAINT: (Resolved)
- Priority - High
- Affecting System - Anual Power & UPS Sys Maint.
-
Dear Lowest host Empire Technology LLC Customers,
In order to serve you better, lowest host Empire Technology LLC is listing all upgrades and changes to our datacenters nationwide. Maintenance in other lowest host Empire Technology LLC Data Centers does not necessarily mean that your facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
Maintenance Scheduled for NOC (North Carolina )
Maintenance Window: 00:01 AM to 01:00 PM MDT GMT -6 Saturday March 24, 2012
Maintenance Window Duration: 1 hour
Maintenance Action: Annual power/UPS Maintenance
Site(s) Affected: North Carolina NOC
Risk Classification: High
Lowest host Empire Technology LLC Master Ticket #: 5877215
Lowest host Empire Technology LLC operations will be performing maintenance on our Power infrastructure in NOC (North Carolina ) this Friday, Saturday March 24, 2012. During this window, engineers will Be performing Annul Scheduled Maintenance on our Power circuit and UPS . We Will be power Cycling all affected servers during this maintenance window, customers may experience 10 15 Minutes of down time while perform this Maintenance. We will perform this work between 00:01 AM to 01:00 PM PDT Time.
Thank You,
Update 01:26 AM 24 March 2012
No issues closing event
- Date - 03/24/2012 00:01 - 03/24/2012 00:24
- Last Updated - 03/24/2012 00:24
Moderate Packet Loss North Carolina Data cent (Resolved)
- Priority - Low
- Affecting Other - NOC/NC
-
EVENT START DATE: 02-25-2012
EVENT START TIME: 10:52 MST
EVENT END DATE: 02-25-2012
EVENT END TIME: :
EVENT IMPACT: Connectivity stability in the ALO-North Carolina data center impacting multiple customers.
EVENT DETAILS: At 10:52 MST 02-25-2012 NOC Techs observed connectivity issues impacting the North Carolina data center. Network Services is working to identify the source of the issue and a resolution. Updates will be provided regularly throughout this event and all efforts are being made to minimize the impact of this issue. Our apologies for the inconvenience.
UPATE : 02-25-2012 12:39 MST
At approximately 10:52 MST on 2/25/2012 North Carolina experienced an
unexpected event on one of its edge routers (not our equipment). At this time the Network
Engineering team is working to resolve the issue. We will give you further
updates as they become available.
The Packet Loss has been reduced to Low at this time and all services are up
UPATE : 02-25-2012 01:47 MST
This network event has been resolved we are downgrading this event to LOW as a packet distribution has returned to normal.
We will continue to monitor until 100% resolution
- Date - 02/25/2012 21:49 - 02/26/2012 10:55
- Last Updated - 02/26/2012 10:55
Server: XEN 2 OS/Hardware Upgrade (Resolved)
- Priority - Medium
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Empire Hosting is listing all upgrades and changes to our data-centers nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
Maintenance Scheduled for North Carolina Data Center 20 Dec 2011
Maintenance Time: Tuesday 20 Dec 2011 00:90 AM to Wednesday December 27 09:00 AM Pacific Time
Maintenance Window Duration: 1 week
Maintenance Xen Server OS Upgrade & Hardware Upgrade
Site Affected: XEN Node 2
Expected Down time per VPS 5 Minutes
Risk Classification: Medium
Empire Hosting Master Ticket #: 550445
Lowest Host/Empire Technology LLC Data center operations will be performing Hardware/OS Maint on XEN Node 2. We will be performing upgrade between 09:00 AM and 09:00 AM Pacific the week of 20 Dec . This will be a complete shutdown of Xen 2 while we upgrade the OS & Hardware.
During this upgrade we will be Live Migrating All VPS servers to A different Xen Node so there will be 0 Downtime with the exception of bringing up the live instance on the new node "5 Minutes"
-------------------------
Update 24 Dec 2011 12:54
All upgrades complete No issue
Thank You Server Operations Team
- Date - 12/19/2011 18:59 - 12/27/2011 00:00
- Last Updated - 12/24/2011 12:54
Notice Of Network/Power Maintenance: (Resolved)
- Priority - Critical
- Affecting System - Power Distribution
-
Dear Lowest host Empire Technology LLC Customers,
In order to serve you better, lowest host Empire Technology LLC is listing all upgrades and changes to our datacenters nationwide. Maintenance in other lowest host Empire Technology LLC Data Centers does not necessarily mean that your facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
Maintenance Scheduled for NOC (Cedar Falls )
Maintenance Window: 04:00 PM to 11:59 PM PDT GMT -7 Friday Dec 23, 2011
Maintenance Window Duration: 1 hour
Maintenance Action: Network upgrades
Site(s) Affected: Cedar Falls NOC
Risk Classification: High
Lowest host Empire Technology LLC Master Ticket #: 5877215
Lowest host Empire Technology LLC operations will be performing maintenance on network infrastructure in NOC (Cedar Falls) this Friday, Dec 23, 2011. During this window, engineers will install Replace A Failed Power Strip in We Will be power Cycling the affected servers that & Pluging them into the new switch approximately during this maintenance window, customers may experience 10 15 Minutes of down time while your server is cut over to the new switch. We will perform this work between 04:00 PM to 11:59 PM PDT Time.
Thank You,
-------------------------
Update 23 Dec 2011
Power distribution switch Replaced No issues
- Date - 12/23/2011 03:49 - 12/24/2011 00:00
- Last Updated - 12/24/2011 12:52
Network Upgrades (Resolved)
- Priority - Medium
- Affecting Other - network infrastructure
-
Dear Lowest host Empire Technology LLC Customers,
In order to serve you better, lowest host Empire Technology LLC is listing all upgrades and changes to our datacenters nationwide. Maintenance in other lowest host Empire Technology LLC Data Centers does not necessarily mean that your facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
Maintenance Scheduled for AVL01 (North Carolina )
Maintenance Time: 01:00 AM to 02:00 AM Eastern Time (05:00 to 06:00 UTC/GMT) Sunday April 24, 2011
Maintenance Window Duration: 1 hour
Maintenance Action: Network upgrades
Site(s) Affected: North Carolina NOC
Risk Classification: Medium
Lowest host Empire Technology LLC Master Ticket #: 573995
Lowest host Empire Technology LLC operations will be performing maintenance on network infrastructure in North Carolina (AVL01) this coming Sunday, April 24, 2011. During this window, engineers will install an additional 48-port gigabit module in order to increase capacity to support future network growth. Additionally, they will be power-cycling another 48-port gigabit module that is presently in a minor error state. For approximately 3 minutes during this maintenance window, customers may experience momentary periods of packet loss or latency. We will perform this work between 01:00 AM and 02:00 AM Eastern.
Thank You,
Lowest host Empire Technology LLC
NOC Support
Risk Classifications:
Risk classifications are based upon the probability of your experiencing a disruption in service.
(Low) The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Lowest Host/Empire technology LLC requires this work be performed during the maintenance window as a matter of policy.
(Medium) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.
(High) A service disruption should be anticipated. The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Lowest Host/Empire technology LLC will move rapidly to limit this interruption to the absolute minimum amount of time? This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.
------
no issure maint complete closing
- Date - 04/24/2011 00:00 - 05/07/2011 14:40
- Last Updated - 05/09/2011 18:15
Xen OS Maint Patch (Resolved)
- Priority - Medium
- Affecting Other - OS Security Patch & Software update (Xen Servers)
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Empire Hosting is listing all upgrades and changes to our datacenters nationwide. This does not necessarily mean that your facility or server will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
Maintenance Scheduled for North Carolina & Cedar Falls Data Centers
Maintenance Time: 00:00 AM Tuesday to 0300 AM Pacific TimeSunday
Maintenance Window Duration: 20 Min
Maintenance Xen Server Maint patch
Site Affected: Xen servers All (except Xen1)
Risk Classification: Medium
Empire Hosting Master Ticket #: 550598
Lowest Host/Empire Technology LLC Data center operations will be performing OS Security Patch & Software update. We will be performing upgrade between 00:00 AM and 02:00 AM Pacific. Although, this is a normal maintenance patch we have classified this as high on the Xen 2 node only in North Carolina. As it has been reading an xen error sins 10:57 Pacific time this morning and we may elect to do this patch prior to Sunday morning to clear the error. Thank You,
Sever Maint Operations Team
----------------------
Upgrades complete no issues
Closing
- Date - 05/07/2011 14:40 - 05/09/2011 18:14
- Last Updated - 05/09/2011 18:14
Cedar Falls Mild Latency (Resolved)
- Priority - Critical
-
At approximately 10:00am PDT on 03-MAR-2011 Empire Technology began experiencing latency from our upstream providers in both (Cedar Falls). The issue is still under investigation with Internap and Level 3, our primary transit providers .
Empire Technology Network Engineering is taking additional measures to re-route our traffic to resolve any continued latency from specific providers. If you are experiencing continued slow connections or an inability to reach your server hosted with Empire Technology in these locations, please check this posting for updates on our corrective measures.
We appreciate your patience while we address this situation.
update 10:29 AM PST
Traffic has been rerouted
Closing
- Date - 03/03/2011 10:14 - 03/03/2011 10:31
- Last Updated - 03/03/2011 10:31
Network upgrades North Carolina NOC (Resolved)
- Priority - High
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Empire Hosting is listing all upgrades and changes to our datacenters nationwide. This does not necessarily mean that your facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
Maintenance Scheduled for North Carolina Data Center
Maintenance Time: 12:30 PM Tuesday to 12:300 PM Eastern Time Thursday
Maintenance Window Duration: 3 Days
Maintenance Action: Network upgrades
Site Affected: AVL01
Risk Classification: High
Empire Hosting Master Ticket #: 550547
Empire Hosting s Network Technicians will be performing Maintenance On Our #1 forward public Switch Customers may experience packet loss or latency for a few minutes on each end of the window as engineers perform this maintenance. We will perform this work usual you may track all Maintenance activities at https://secure.empire-hosting.net/networkissues.php
If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact Open a support request @
https://secure.empire-hosting.net/submitticket.php
Thank You,
Datacenter Operations Team
- Date - 03/01/2011 10:32 - 03/16/2011 20:43
- Last Updated - 03/01/2011 09:26
Server Outage Server Outage Columbia shared (Resolved)
- Priority - Low
- Affecting Server - Columbia
-
Server Outage Wed Jan 5 2011 8:37 PDT
We are having a Issue with our Columbia shared hosting server it is frozen and will not boot we are currently investigating the issue
Update Wed Jan 5 2011 9:20 PDT
We have diagnosed it to a bad RAM bank we have and corrected the issue
this will remain downgraded to low while we continue to monitor it through the night
- Date - 01/05/2011 20:39 - 01/06/2011 00:00
- Last Updated - 01/08/2011 08:47
Network Outage North Carolina NOC (Resolved)
- Priority - Medium
-
Network Outage Wednesday Dec 15 14:15 MST There has been a major outage feeding the North Carolina Data center. We are investigating at this time
-----------------------------------------------------------------------------------------------------------------------------------------------
Update 15 Dec 2010 15:25 MST we have got a quick answerer from the network engineers and they have isolated the issue and the network should be coming back on line. Please note we are still in the dark as to the actual issue as this is outside of our network when we have more information available we will post it.
Update 15 Dec 2010 4:00 MST The network is back up We are we are leaving this network issue in a critical state as we still have no information on why the incident occurred in the first place.
-----------------------------------------------------------------------------------------------------------------------------------------------
Update 15 Dec 2010 18:00 MST forwarded Message from The data center Engineers
As you are aware, we experienced a major network issue this afternoon that started at around 15:45 Eastern with a few BGP flaps followed by a full-on outage starting at 15:55. While service has been restored, we're not even close to closing out this issue. I am meeting with our network infrastructure team members right now to fully understand what transpired so I can provide you with the full details. We have a case open with Cisco TAC as well to bring additional insight into the events that transpired to cause an outage of this magnitude. Please look for an update from me within the next few hours.
------------------------------------------------------------------------------------------------------------------------------------------------
Update 16 Dec 2010 11:36 MST
With No updates as promised from our upsteam will are closing this issue Network was fully restored yesterday
-------------------------------------------------------------------------------------------------------------------------------------------------
Update 18 Dec 2010 0:841 MST
Issue reopened
------------------------------------------------------------------------------------------------------------------------------------------------
Update 18 Dec 2010 08:53 MST
forwarded from our upstream:
Dear Lowest Host Empire Technology LLC ,
Here is an additional update for Friday, December 17th. In a few hours, our change management team will be sending out an emergency maintenance notification for this coming Sunday, December 19th beginning at 1:00 AM Eastern time. Working with Cisco TAC, our engineers believe we have narrowed down the cause of Wednesday’s outage to a known bug affecting the Cisco 6708 module when deployed in 7600 series routers. The 6708 module has 8 x 10 Gigabit ports that we use to cross uplink our edge to our core. All of the 10G connections on one particular module began to cycle up/down sporadically 10 minutes prior to the outage. While this up/down activity did not specifically cause any connectivity issues due to the redundancies in place, we believe that it was the ultimate trigger to the BGP failure between our edge and core which did cause the outage.
We will be using this maintenance window to update to a later IOS version on all devices, add additional 10G modules to distribute the risk over multiple modules, and incorporate use of the 10G ports on the RSPs for additional redundancy using something other than a 6708 module – the only type we have used to date for connecting the edge to the core. As an aside, we use the ES20 module for connectivity from the core to our backbone nodes in Atlanta, GA and Ashburn, VA because it supports VPLS. The ES20 module is not affected by this bug.
As always, we greatly appreciate your patience as we dissected the incident and worked through developing a concrete action plan. Many of you have emailed me personally with suggestions, and I do appreciate it. Keep them coming. I know we have a very long way to go to regain your trust, but that is my personal goal, and everything I will be focusing on between now and January 1st will revolve around that.
I will write you on Monday after the maintenance has been completed to provide you with additional information.
Sincerely,
Jonathan Hoppe
Chief Technology Officer
--------------------------------------------------------------------------------------------------------------------------------------------------
Update 18 Dec 2010 0900 MST
In order to serve you better, Loesr Host/Empire Technology LLC is listing all upgrades and changes to our datacenters nationwide. Maintenance in other Data Center does not necessarily mean that your IDC facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
Maintenance Scheduled for North Carolina Data Center #2
Maintenance Time: 01:00 AM to 05:00 AM Eastern Time (06:00 to 10:00 UTC/GMT) Sunday December 19, 2010
Maintenance Window Duration: 4 hours
Maintenance Action: Network upgrades
Site Affected: AVL01
Risk Classification: Moderate
Master Ticket #: 555892
Network operations will be performing IOS upgrades and routing changes to edge and core infrastructure this coming Sunday, December 19th in our AVL01 datacenter. During this window, engineers will reroute and fail over traffic to minimize Customer impact, however customers may experience periods of packet loss, latency or very brief service interruptions as engineers perform these upgrades. There will NOT be an extended outage during this maintenance window. We will perform this work between 01:00 AM and 05:00 AM Eastern.
Thank You,
Lowest Host /Empie Technolgy LLC Support
Risk Classifications:
Risk classifications are based upon the probability of your experiencing a disruption in service.
(Medium) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.
-------------------------------------------------------------------------------------------------------------------------------------------------
Update 19 Dec 03:42 MST
all upgrades complete no issues
closing this incident
- Date - 12/15/2010 14:01 - 12/21/2010 04:23
- Last Updated - 12/19/2010 02:43
Maintenance Scheduled for North Carolina (Resolved)
- Priority - Medium
- Affecting Other - infrastructure upgrades
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Empire Hosting is listing all upgrades and changes to our datacenters nationwide. This does not necessarily mean that your facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
Maintenance Scheduled for North Carolina Data Center
Maintenance Time: 02:00 AM to 05:00 AM Eastern Time (07:00 to 10:00 UTC/GMT) Sunday November 21, 2010
Maintenance Window Duration: 3 hours
Maintenance Action: Network upgrades
Site Affected: ATL01, AVL01
Risk Classification: Medium
Empire Hosting Master Ticket #: 550546
Empire Hosting s Upstream Transit providers will be performing various infrastructure upgrades in order to increase the number of 10G links between Atlanta (ATL01) and Asheville (AVL01). During this window, engineers may reroute traffic from Asheville to Ashburn in order to minimize Customer impact. Customers may experience periods of packet loss or latency as engineers perform these upgrades. We will perform this work between 02:00 AM and 05:00 AM Eastern. Per usual you may track all Maintenance activities at https://secure.empire-hosting.net/networkissues.php
If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact Open a support request @
https://secure.empire-hosting.net/submitticket.php
Thank You,
Datacenter Operations Team
Risk Classifications:
Risk classifications are based upon the probability of your experiencing a disruption in service.
(Medium) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.
Mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we
============================
Update Maint: concluded No impact
- Date - 11/21/2010 00:00 - 11/21/2010 06:18
- Last Updated - 11/21/2010 06:18
Maintenance Scheduled for North Carolina (Resolved)
- Priority - Low
- Affecting Other - semi-annual maintenance on our UPS
-
Dear Lowest Host/Empire Technology Customers
In order to serve you better, Empire Hosting is listing all upgrades and changes to our datacenters this does not necessarily mean that your facility will be affected. All pertinent information can be found in the description for each maintenance action to be performed.
Maintenance Scheduled for our North Carolina Data Center
Maintenance Time: 10:00 AM to 3:00 PM Eastern Time (15:00 - 18:00 UTC/GMT) Saturday November 20, 2010
Maintenance Window Duration: 5 hours
Maintenance Action: Semi-Annual UPS Preventative Maintenance for Electrical Buss 1
Site Affected: North Carolina NOC
Risk Classification: Low
Empire Hosting Master Ticket #: NA
Empire Hastingss third party service provider will be performing semi-annual maintenance on our UPS systems affecting electrical buss 1 this coming Saturday, November 20, 2010 from 10:00 AM to 3:00 PM Eastern Time. The work being performed is routine and involves testing each UPS, checking components, battery voltages etc. The datacenter will be placed on stable generator power during bypass testing to ensure no interruption to the critical load. As an additional precaution, members of our electrical and facility operations teams will be onsite during this maintenance window to supervise the process.
If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact Open a support request @
https://secure.empire-hosting.net/submitticket.php
Thank You,
Datacenter Operations Team
Risk Classifications:
Risk classifications are based upon the probability of your experiencing a disruption in service.
(Low) The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Empire Hosting requires this work be performed during the maintenance window as a matter of policy.
=================================
Update Maint copleate No issues
- Date - 11/20/2010 00:00 - 11/20/2010 00:00
- Last Updated - 11/20/2010 20:56
Xen Servers Maint Patch (Resolved)
- Priority - High
- Affecting Other - Maintenance Scheduled for Both Cedar Falls IA & Asheville NC data centers
-
Maintenance Scheduled for Both Cedar Falls IA & Asheville NC data centers
Maintenance Time: 11:00 PM to 01:00 AM (PDT, GMT - 7:00) Sunday Oct 31, 2010
Maintenance Window Duration: 20 Minutes
Maintenance Action: OS Security Patch & Software upgrade
Site Affected: Xen Servers ALL
Risk Classification: High
Lowest Host/Empire Technology LLC Master Ticket #: 9428
Lowest Host/Empire Technology LLC Data center operations will be performing OS Security Patch & Software update. We will be performing upgrade between 01:00 AM and 02:00 AM Pacific. Although, this is a normal maintenance patch we have classified this as high all XEN Nodes have been on line for 900 + days with 0 down time (only brief outages to apply similar maintenance Patches) and the Hard Drives may go into a forced maintenance cycle. Should this happen the network status page will be upgraded to reflect the extended maintenance. (aprox 45 min 3 hours)
If you have any questions, or would like to determine the extent of impact any maintenance may have on your service, please contact us at https://secure.empire-hosting.net/submitticket.php and reference the specific case number listed above.
Thank You,
Datacenter Operations Team
Risk classifications
(High) Risk classifications are based upon the probability of your experiencing a disruption in service longer than the posted time.
-----------------------------------------------------------
Update: Oct 31 2010 5:19 PM PDT
updated all Centos 5 Images minimal, webmin, cPanel, cPanel DNS, DirectAdmin,
-------------------------------------------------------------
Update: Nov 1 2010 1:58 AM PDT
All updates complete No issues
closing
- Date - 10/31/2010 09:51 - 11/01/2010 01:00
- Last Updated - 11/01/2010 01:00
Maintenance Scheduled for Cedar Falls NOC (Resolved)
- Priority - Low
- Affecting System - Virtuozzo Server VPS 4 OS Security Patch & Software upgrade
-
Dear Lowest Host/Empire Technology LLC Customers
Maintenance Scheduled for Cedar Falls NOC
Maintenance Time: 01:00 AM to 02:00 AM (PDT, GMT - 7:00) Tues Oct 5, 2010
Maintenance Window Duration: 20 Minutes
Maintenance Action: OS Security Patch & Software upgrade
Site Affected: Virtuozzo Server VPS 4
Risk Classification: High
Lowest Host/Empire Technology LLC Master Ticket #: 9428
Lowest Host/Empire Technology LLC Data center operations will be performing OS Security Patch & Software update. We will be performing upgrade between 01:00 AM and 02:00 AM Pacific. Although, this is a normal maintenance patch we have classified this as high the VPS 4 NODE has been on line for virtually 937 days with 0 down time (only brief outages to apply similar maintenance Patches) and the Hard Drives may go into a forced maintenance cycle. Should this happen the network status page will be upgraded to reflect the extended maintenance. (aprox 45 min 3 hours)
If you have any questions, or would like to determine the extent of impact any maintenance may have on your service, please contact us at https://secure.empire-hosting.net/submitticket.php and reference the specific case number listed above.
Thank You,
Datacenter Operations Team
Risk classifications
(High) Risk classifications are based upon the probability of your experiencing a disruption in service longer than the posted time.
============================
Update: 5 Oct 2010
After installing the Patch one of the Hard drives has faild it has since been replaced
and the server is performing Normal Drive Miant
=================================
Update 5 Oct 2010 03:54 PDT
The raid array is currently being rebuilt & Quotas are being repaird on client VPS's. We wil be downgrding this Maint action to Low as we do not foresee having any further issues we will leave this maint incident open until the raid array is rebuilt 100%
- Date - 10/05/2010 00:00 - 10/27/2010 06:56
- Last Updated - 10/05/2010 16:00
North Carlina NOC (Resolved)
- Priority - Low
- Affecting Other - Netwark Outage
-
Network Outage Sunday October 3 21:30 PST
There has been a major outage feeding the North Carolina Data center. The Main Trunk Feed from the main NAP we do not have many details at this time
Washington DC or Atlanta Portions of the Atlanta NAP are down as well. The issue is being investigated.
I would only be speculating at the time if it was a fiber cut or main switch failure the Network engineers are currently rerouting traffic away from the main feed into the data center we will update your further when I have more information.
=====================================
Oct 2 2010 23:05
The traffic has been rerouted the network has been restored
Downgrading this issue to low
======================================
From the Network engineers
At 12:09AM EDT
our Atlanta secondary path went down for scheduled and anticipated maintenance that was slated to be non-customer impacting, however at 12:12AM our primary
Fiber to Atlanta and Ashburn went down as well, both served by a different provider. At 01:42AM, Ashburn connectivity was restored, and
At 03:08AM Atlanta secondary/protect connectivity was restored. We are still working with our providers to fully understand the reason for this massive outage, and why our
Atlanta primary fiber remains offline. As soon as we have further information, we will update you
===============================================================
As of 9:46AM
our Atlanta primary fiber is also back online completing full service restoration
===============================================================
All Traffic routes have converged there should be no further issues we are closing this issue at this time
- Date - 10/03/2010 00:00 - 10/04/2010 10:48
- Last Updated - 10/04/2010 10:48
Maintenance Notification North Carilina NOC (Resolved)
- Priority - Low
- Affecting Other - Data Center Maint
-
Dear Lowest Host/Empire Technology LLC Customers
Maintenance Scheduled for North Carolina NOC
Maintenance Time: 01:00 AM to 02:00 AM (Eastern, GMT-4) Sunday August 29, 2010
Maintenance Window Duration: 1 hour
Maintenance Action: Network Grooming
Site Affected: North Carolina NOC
Risk Classification: Low
Lowest Host/Empire Technology LLC Master Ticket #: 535923
Lowest Host/Empire Technology LLC network operations will be performing low impact network grooming on our core infrastructure in Noth Carolina to remove configurations no longer in use as a result of previous upgrades. Customers should not experience any issues, however as a precautionary measure we are sending out this notice. We will perform this work between 01:00 AM and 02:00 AM Eastern.
If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us at https://secure.empire-hosting.net/submitticket.php and reference the specific case number listed above.
Thank You,
Datacenter Operations Team
Risk Classifications:
Risk classifications are based upon the probability of your experiencing a disruption in service.
(Low) The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Lowest Host/Empire Technology LLC requires this work be performed during the maintenance window as a matter of policy.
- Date - 08/29/2010 00:00 - 10/03/2010 22:09
- Last Updated - 08/26/2010 07:41
Emergency server Maint (Resolved)
- Priority - Low
- Affecting Server - Columbia
-
Emergency Server maint
Maintenance Scheduled for Columbia shared hosting server
Maintenance Time: 15:10 - 15:50 PST 7-23-2010
Maintenance Window Duration: 1 hour
Maintenance Action: Hard drive failure & rebuild of raid array
Site Affected: Columbia shared hosting server
Risk Classification: Low
Master Ticket #: NA
we will be performing a a Hard drive maintenance on our Columbia shared hosting server to facilitate post maint action to facilitate hard drive failure
Thank You,
-----------------------------------------------------------------------------------------
Update 16:07 PDT 7-23-2010
the Maint is completed on schedule and server was back on line at 3:50 PDT
we will continue to monitor the server and the the rebuild of the raid array
-----------------------------------------------------------------------------------
Update 16:25 PDT 7-23-2010
we are down grading this incident to low as we expect no further issues
-------------------------------------------------------------------------------------
Risk Classifications:
Risk classifications are based upon the probability of your experiencing a disruption in service.
(Low) The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, this requires this work be performed during the maintenance window as a matter of policy.
(Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.
(High) A service disruption should be anticipated. The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Lowest Host Empire Technology will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.
- Date - 07/23/2010 15:33 - 08/26/2010 07:39
- Last Updated - 07/23/2010 16:29
core routing infrastructure upgrades (Resolved)
- Priority - High
- Affecting System - Network Upgrades
-
Dear Lowest Host/Empire Technology Customers,
Maintenance Scheduled for North Craolinia NOC
Maintenance Time: 00:01 AM (Eastern, GMT-4) Monday July 5, 2010
Maintenance Window Duration: 5 hours
Maintenance Action: Network upgrades/maintenance
Site Affected: NC NOC
Risk Classification: High
Master Ticket #: 527718
Lowest Host Empire Technology network operations will be performing various core routing infrastructure upgrades in order to integrate North Carolina NOC into our new regional backbone. Customers may experience periods of packet loss or latency as engineers perform these upgrades. We will perform this work between 00:01 AM and 05:00 AM Eastern.
Thank You,
Datacenter Operations Team
Risk Classifications:
(High) A service disruption should be anticipated. The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Lowest Host Empire Technology will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.
-----------------------
update closing issue no outages reported all maint complete
- Date - 07/04/2010 05:26 - 07/05/2010 07:55
- Last Updated - 07/05/2010 07:54
Network Incident Report: ALO (Cedar Falls) 27 (Resolved)
- Priority - Low
- Affecting Other - Modorate Packet Loss
-
EVENT START DATE: 27-06-2010
EVENT START TIME: 22:00 CDT
EVENT END DATE:
EVENT END TIME: :
EVENT IMPACT: Connectivity stability in the ALO-Cedar Falls data center impacting multiple customers.
EVENT DETAILS: At 22:00 CDT 27-06-2010 NOC Techs observed connectivity issues impacting the Cedar Falls data center. Network Services is working to identify the source of the issue and a resolution. Updates will be provided regularly throughout this event and all efforts are being made to minimize the impact of this issue. Our apologies for the inconvenience.
UPDATE: As of 22:40 Network Services is continuing to work to isolate the cause of the connectivity stability issues in Cedar Falls. Steps are being taken to isolate and mitigate the impact to the various segments of the network and to return stability as soon as possible. At this time there continues to be some instability that is manifesting itself for customers as latency/packet loss and loss of connectivity to various hosts. We will continue to update this posting as progress is made, and appreciate your patience while we work to resolve this issue.
Update: As of 23:30 This has been Downgraded to Low
This has been identified a DDOS attack to another tenant that resides within our data center which is impacting portions of the Cedar Falls network. Efforts to isolate the source and targets of this attack are underway. Customers should begin seeing an improvement as these efforts continue. We appreciate your patience in this matter as we work to fully resolve the issue. We are downgrading this issue to low as traffic has returned for the most part to normal and will continue to improve as the attack is being mitigated
UPDATE: As of approximately 01:00 CDT, our uptream transit provider was able to identify the various sources and targets of a DDOS which was of significant size and scope to have impacted multiple portions of the Cedar Falls network. External sources of the attack and internal targets have been null routed to eliminate the impact to the remainder of our customer base in this facility. Overall network performance and stability has returned to normal as a result of these efforts.
Again, we appreciate everyone's patience while we worked to identify and resolve this issue. If further updates are necessary they will be posted here.
- Date - 06/27/2010 20:58 - 06/28/2010 05:21
- Last Updated - 06/28/2010 00:00
Maintenance Action: Network upgrades/maintena (Resolved)
- Priority - Low
- Affecting System - Upstram Transit Maint:
-
Maintenance Scheduled for North Carolina NOC
Maintenance Time: 00:01 AM (Eastern, GMT-4) Sunday June 27, 2010
Maintenance Window Duration: 4 hours
Site Affected: North Carolina NOC
Risk Classification: Low
Master Ticket #: 526847
Our Upstream transit providers operations will be performing various low-impact network upgrades in order to optimize routes and increase capacity for future expansion activities. Customers may experience short periods of packet loss or latency as engineers perform these upgrades. We will perform this work between 00:01 AM and 04:00 AM Eastern.
Thank You,
Datacenter Operations Team
)
Risk Classifications:
Risk classifications are based upon the probability of your experiencing a disruption in service.
(Low) The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, requires this work be performed during the maintenance window as a matter of policy.
Update SUN JUN 27 2010
All upgrades Complete No issues reported Closing
- Date - 06/27/2010 00:00 - 06/27/2010 21:44
- Last Updated - 06/27/2010 21:38
upstream Transit Provider circut maint: (Resolved)
- Priority - Low
- Affecting Other - Scheduled Maintenance
-
Server Maintenance Notice: To all in the Cedar Falls facility: Scheduled Maintenance: ALO01 (Cedar Falls) 05-30-2010 01:00 06:00 EST.
EVENT START DATE: 05-30-2010
EVENT START TIME: 01:00 EST (05:00 GMT)
EVENT END DATE: 5-30-2010
EVENT END TIME: 07:00 EST (11:00 GMT)
EVENT IMPACT: ALO01 (Cedar Falls)
On Sunday 5/30/2010 at 01:00 CST, our upstream Transit providers will be performing scheduled maintenance in the Cedar Falls data center to provide new customer aggregate connectivity. Customers in the Cedar Falls data center will experience brief periods of outage as customer connectivity is aggregated in a phased and controlled migration. Recovery for customers affected during this migration should be rapid and complete with no residual affect following resumption of normal connectivity.
Update : 05-30-2010 02:25 EST (06:25 GMT)
Maintenance has been completed on our network segment Network ticket will remain open but downgraded to low. As there may be some residual latency as routes converge.
- Date - 05/30/2010 00:00 - 06/06/2010 15:14
- Last Updated - 05/30/2010 00:27
Utility Transformer Replacement (Resolved)
- Priority - Medium
- Affecting Other - Maintenance Scheduled for Ashville NOC
-
Maintenance Scheduled for AVL01 (North Carolina NOC)
Maintenance Time: 07:00 AM (Eastern, GMT-4) May 2, 2010
Maintenance Window Duration: 4 to 6 hours hours
Maintenance Action: Progress Energy Transformer Replacement
Site Affected: AVL01 (Morth Carolina NIC)
Risk Classification: Moderate
Master Ticket #: 514649
Progress Energy has informed us that they will need to replace a utility transformer behind AVL01 feeding our facility. No downtime is anticipated and all precautions have been implemented. We will switch to stable generator power prior to and for the duration of their maintenance activities. They will perform this activity tomorrow, Sunday May 2, 2010 starting at 7:00 AM and should complete at or around 12:00 Noon.
Monday May 3 2010 (Update)
All Maint completed without error No disruption
Closing incident
- Date - 05/02/2010 13:29 - 05/03/2010 08:40
- Last Updated - 05/03/2010 08:40
Multiple IP DDOS flood (Resolved)
- Priority - Critical
- Affecting Other - Ashville NOC
-
Maintenance Unscheduled AVL01
Maintenance Time: 20:50 PDT
Maintenance Window Duration: Until Resolved
Maintenance Action:Latency Multiple IP DDOS flood
Site Affected: AVL01 CAB14
Risk Classification: High
Master Ticket #: 510304
Currently our Asheville NC Data Center is under a heavy sustained multiple IP DDOS flood
Our Network Engineers' on site at the facility are working diligently to to mitigate the attack ant route the traffic either though the firewall or null routing the attacking IP's
we will update this network ticket as we have moor information
Datacenter Operations Team
-------------------------------------------------------------------------------------------------------------------------------------------
Identified and the affected server taken off Line. The affected IP block has been Null routed. The Flood traffic is still overwhelming our switch & Firewall when the attacking Networks' Reroute & converge on the Null routed IP addresses the latency will improve
Maintenance Unscheduled AVL01 (Update)
Maintenance Time: 19:42 PDT
Maintenance Window Duration: Until Resolved
Maintenance Action:Latency Multiple IP DDOS flood
Site Affected: AVL01
Risk Classification: High
Master Ticket #: 479240
The Netwowek engineers have Mitagated the attack though a DDOS Layer switch
everything is back to nomal and has been since 04:00 PDT, we will leave this network issue open until we reroute our netwok routes routes
-------------------------------------------------------------------------------------------------------------------------------------------
Maintenance Unscheduled AVL01 (update)
Maintenance Time: 06:00 PDT (12:00 GMT) 12-14-10
Maintenance Window Duration: Until Resolved
Maintenance Action:Latency Multiple IP DDOS flood
Site Affected: AVL01 CAB14
Risk Classification: Low
Master Ticket #: 510304
Update The Distributed Denial of Service (DDOS) attack that started 20:50 PDT (02:00 GMT) 04-12-10 at Our North Carolina Facility Was under our control AT 04:00 (10:00 GMT ). 04-13-10We remained Under a DDOS Layer Switch (anti DDOS prevention) Until 14:00 PDT 20:00 (GMT) after that we allowed normal traffic to flow.
During that Time a few clients Had access issues this was the exception not the rule as most did not see any anomalies. And a even fewer reported slowness while pulling data from other networks. This was unavoidable because while under the DDOS layer the inbound traffic must be filtered.We apologize for any inconvenience we had no control over the malicious acts of criminals who seem to have nothing better to do. The alternative would have been total network failure.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Maintenance Unscheduled AVL01 (Update)
Maintenance Time: 06:30 PDT (12:30 GMT) 12-14-10
Maintenance Window Duration: Until Resolved
Maintenance Action:Latency Multiple IP DDOS flood
Site Affected: AVL01 CAB14
Risk Classification: Low
Master Ticket #: 510304
Update 40 minutes after restoring traffic to normal the effected IP's of the attack were again under attack these IP's have been Null routed until 100% of the malicious traffic can be stopped. The respective Client(s) have been notified separately from this Ticket
This ticket will remain open but reduced to a low priority as there should be no further impact on any other users in our North Carolina Facility.
- Date - 04/12/2010 23:47 - 04/25/2010 08:21
- Last Updated - 04/14/2010 06:46
(CedarFalls) core network upgrades (Resolved)
- Priority - Medium
- Affecting System - (CedarFalls) core network upgrades
-
Upstreem Transit provider Maint Rescsheduled
EVENT IMPACT: ALO01 (CedarFalls) core network upgrades, expected periods of increased latency during route propagation
EVENT DETAILS:
This is a rescheduled event from 21-02-2010**On Sunday, 28-02-2010, one of our transit providers will be performing emergency maintenance to address stability and redundancy issues for a core network link which services the ALO01 (Cedar Falls) datacenter.
This maintenance is only significant to customers with services hosted out of the Cedar Falls facility and no service disruptions are expected. However, because the maintenance involves core network components this maintenance window is being announced. Customers may notice periods of increased latency during this maintenance window as route updates are applied.
- Date - 02/18/2010 22:24 - 03/01/2010 05:22
- Last Updated - 02/26/2010 14:06
Cisco Core Switch/Router Replacement (Resolved)
- Priority - High
- Affecting Other - Data Center I Equipment upgrade
-
Maintenance Time: 1 Oct 2009 – 31 December 2010
Maintenance Window Duration: 1 year 3 months
Actual Impact Window Duration: No impact
Maintenance Upgrade of aging Cisco Switches
Data center Site Affected: Cedar Fall NOC all network segments
Risk Classification: High
Master Ticket #: 473911
Beginning October 1 2009 We will be doing a major overhaul Of our Network Infrastructure, Replacing all of our core switches with new Cisco Switches. This will affect all Customers with the exception of clients on the 208.91.131.XXX & 66.219.30.XX Network.
Rather than have any Service disruption we have opted do a Live Migration to a a upgraded segment of our network, but unfortunately we can not port the IP's and move the servers to the new Rack/Switch without service disruption and our upstream will not allow us to move the IP's to a different location. So we will be moving affected clients & servers in Phases.
July 1 2009 Phase I: Build Rack Space to accommodate All clients & servers.
September 20 2009 Phase II: move all empty & non essential servers to new network segment (100% completed)
October 1 2009 Phase III Move XEN VPS servers & cut to new network segment (100% complete)
TBA in 2010 Phase IV: Move Virtuozzo servers
TBA 2009/2010 Phase V: Move all shared & reseller services We will be retiring our current DNS servers
TBA 2010 Phase VI: move all affected Dedicated servers ( 25% complete)
- Date - 10/02/2009 15:53 - 02/18/2010 22:19
- Last Updated - 12/09/2009 21:38
Emergencey Fiber Splice (Resolved)
- Priority - Critical
- Affecting Other - Asheville NOC
-
Maintenance Time: Between 01:00 AM and 06:00 AM (U.S. Eastern Daylight, GMT-4) September 15, 2009
Maintenance Window Duration: 5 Hours
Actual Impact Window Duration: 30 minutes
Maintenance Action: Emergency Fiber Splice
Datacenter Site Affected: Ashville NOC 208.91.XXX.XXX network segment
Risk Classification: High
Master Ticket #: 473911
On Tuesday, September 15, 2009 at 01:00 AM through to 06:00 AM, Lowest Host/Empire Technology LLC fiber provider for Ashville NOC will be performing an emergency fiber splice to replace a severely damaged section of fiber between Asheville and Atlanta. Customers will experience packet loss, latency, or other connectivity issues when traffic shifts to and from our other circuits.
While we have taken every precaution to minimize complications during this event, there exists a certain amount of risk and the possibility of extended service disruption.
We are committed to providing the highest possible level of service; therefore, we strive to increase client satisfaction and productivity through our proactive approach to customer notification. If you have questions or concerns in advance of this event,
Thank You,
- Date - 09/15/2009 12:44 - 10/02/2009 15:48
- Last Updated - 09/14/2009 12:49
Network Distribution Upgrade Asheville NC NOC (Resolved)
- Priority - Critical
- Affecting System - Network Upgrades
-
On Sunday, August 2, 2009 at 00:01 AM through to 06:00 AM,
Empire Technology will be migrating customer VLANs from our current distribution routers to our new routers in order to take advantage of our new multi-10 Gigabit distribution infrastructure. This upgrade has been implemented in response to rapid growth of our network and in order to natively support 10 Gigabit and Cisco VSS 1440.
Although this maint is Marked as critical it will have little impact as the Asheville facility is new and is posted just for informational purposes only.
- Date - 08/02/2009 00:00 - 09/13/2009 18:51
- Last Updated - 08/01/2009 11:09
New Circuit deployment at Cedar Falls NOC (Resolved)
- Priority - Low
- Affecting System - NetWork Maint
-
DATE: 8/06/2009
START TIME: 0000 CDT
ESTIMATED END TIME: 0400 CDT
SERVICES/EQUIPMENT: New circuits on production network
TYPE OF WORK: Circuit deployment
IMPACT OF WORK: None expected
On August 6, 2009 starting at 10:00 AM PDT until 2:00 AM PDT, we will be doing general maintenance and bringing new circuits into production to improve service for our customers on several of our core systems. Though none of the work is expected to affect or impair the flow of traffic or cause an outage, we are posting this announcement as a courtesy in the event a problem or an outage would occur. Systems affected with the scheduled maintenance are all internal and external BGP traffic in our Cedar Falls Data Center.
- Date - 08/06/2009 00:00 - 09/13/2009 18:51
- Last Updated - 08/01/2009 10:56