Network Status News & Information

There will be outages this evening 17th April as we try again to bring the new drive array online to resolve recent performance issues.

We apologise for inconvenience caused.

We are scheduling a necessary web hosting service outage this evening 12th April at about 6pm - 7pm. This is required to bring a new faster drive array online. This cannot be done without taking the server off-line temporarily. We apologise for inconvenience caused.

Update 13 April 7am
This has obviously taken vastly longer than anticipated. A two year old bug in clonezilla when encountering LVM with overlapping UUID’s sent the disk copy into an endless loop. This meant removing the drives and starting again with a manual copy. Technicians believe we are now 30 minutes to an hour away from completion. We apologise for inconvenience caused.

Update 13 April 1pm
The platform is now back up but unfortunately despite working through the night technicians were not able to bring the new drive array online so we are not yet operating at high performance. 

Our technician encountered numerous unforeseen issues with what should have been a simple routine drive transfer. We ran into bugs with the CloneZilla software used to clone the drives and then into ongoing issues with the way the Dell RAID controllers function. Unfortunately since resolving these issues was taking too long we have had to revert back to the original drive array to get websites and email back up and give technicians breathing space to work through the solution calmly. These issues are not insurmountable and we will get the faster drive array online without another round of significant disruption. I want to assure you that all of your data is backed in multiple locations and is safe. I know that this has been incredibly frustrating for you and it's been agony for me too!

We will be processing account credits for the downtime as we have failed to meet our uptime reliability promises. More information on this will follow next week including a more detailed incident report.

I apologise for the disruption to business and I thank you for being a BoltHost customer.

Kind regards
Chris

Web hosting is off-line this morning. We are investigating. Updates will be posted here

Update 26th March:
Web hosting and email is back up but we continue to experience seriously degraded performance on da2.securewebhosting.co.nz due to a combination of DDOS attacks and extremely high traffic usage.

We recommend moving sites behind a service such as CloudFlare or Red Shield to provide an additional layer of protection for your website from DDOS and other attacks as well as providing caching to speed your website up. CloudFlare has a free tier that is suitable for most sites. Contact your website developer if you need assistance with this.

We have been working on the firewall and resource limits to mitigate the DDOS attack but have not managed to consistently block it, as a result website performance and access has been up and down over the weekend.

Our mitigation of the DDOS attack may be impacting connectivity for international customers. This will be because your IP falls within a block. If you cannot connect at all from overseas please open a ticket and let us know your IP address. To find your IP address see https://whatismyipaddress.com

We are working to bring online hardware upgrades to handle the additional load we are currently seeing now and into the future

Update 27th March:
We believe the DDOS attack has now been thwarted and performance this morning is back close to normal. There may be intermittent periods of slow access. International connectivity has been restored.

Update 28th March:
We had to do several server restarts last night to update firmware causing outages around 5 - 6pm. We continue to experience intermittent periods of slow access today

Update 29th March:
New server hardware is on it's way to handle the spikes in traffic we are still experiencing, unfortunately due to availability issues this will not arrive until next week.

Recommendations:

• Move your website behind CloudFlare, we have seen significant performance improvements from this. 
• If your site runs PHP then enable the opcache extension. From the control panel go to Select PHP Version and tick opcache. This will make a big difference to CMS or framework based websites that are made up from a large number of PHP files such as WordPress, Drupal, SilverStripe or Laravel based sites.
• If you are still running PHP 5 please upgrade your website to run on PHP 7 or newer, PHP 7 and 8 are significantly faster than 5 and more secure. To change your PHP version from the control panel go to Select PHP Version

We are currently investigating an expired SSL certificate issue that is affecting the controlpanel and email

There is currently a brute force attack underway on our customer's email accounts, this affects all web hosting customers who host their email accounts with us.

If you host your email elsewhere, such as with Google or Microsoft, you should still follow the advice given here because brute force attacks can happen on any platform at any time.

A brute force attack is an automated attack that attempts to crack people's passwords by using a password dictionary or randomly generated passwords to attempt to log into people's accounts, the attack continues until they find a match.

Their goal is to access email accounts to send spam, to scan your emails for passwords to other services and to try your password on other services since most people use the same password everywhere.

This attack is distributed across a network of hundreds of compromised servers and computers making prevention a game of whac-a-mole.

We are continually blocking these attacks as they appear and are confident in the security that we have in place.

But you should do your part too, if you use a weak password they may be able to crack it before we can block them.

Is your password strong enough? You could use this free online tool to find out: https://random-ize.com/how-long-to-hack-pass/

If you are using weak passwords please consider changing them. You can change your email passwords from our web hosting control panel at https://controlpanel.securewebhosting.co.nz:2222

If you need assistance please let me know!

Update 12 Jul

The attack appears to be over, we appear to have now blocked their entire botnet as we have not see another attack pop up for over 48 hours now. As far as we are aware no accounts were compromised and there was no impact on our services. Thank you to everyone who did take the time to update their email password for a stronger password.

I'd like to remind anyone still using a weak password for their email accounts (or for any other service) to please make sure they are using a strong password, brute force attacks happen all the time on all services and can crack weak passwords very quickly. To see how long it might take for an attack to crack your password see https://random-ize.com/how-long-to-hack-pass/

We are experiencing connectivity issues as a result of an outage at our data centre

Web hosting traffic is currently down, however BoltMail is currently unaffected

The data centre is managed by Voyager internet and they are working through the issue

Please follow the link below for updates:

https://status.voyager.nz/incidents/gdkv1cffgzy3

We will be performing updates and general maintenance on BoltMail this Sunday 19th June at approximately 11pm. During this time BoltMail will be unavailable. Total estimated downtime could be up to an hour.

Campaigns scheduled to send during this time will be delayed and will instead send after the work is completed. We recommend not scheduling any campaigns to send during this time.

If you have any questions or concerns please let us know.

For BoltMail 2 users this update will bring the ability to create additional logins for your account and numerous minor enhancements.

Some users are reporting seeing a deceptive site warning when using BoltMail.

What appears to have happened is that a link in an email campaign was redirecting to our login screen, we believe someone clicked that link and thought that the login screen was there to try and steal their password so reported it. We have explained the situation to Google and are now at their mercy to get that warning removed.

Further details:
There was a bug in the report abuse link that would redirect a subscriber to the login page if they had already un-subscribed before reporting the campaign as abuse. This was because only subscribed subscribers had permissions to report an email as abuse and our app redirects to the login screen on permission failure. We have resolved that bug. We are still waiting on Google to review the situation.

We know that it was only one subscriber who reported this to Google because the report abuse URLs are unique to each subscriber and only one URL is listed as deceptive in Google's Search Console. This seems very heavy handed on their part.

Our datacenter and upstream connectivity providers are experiencing significant service disruptions due to external influences. Technicians are working to resolve these issues which are expected to continue for several days. While this is going on you may experience slower than usual web page loading on our BoltHost web hosting and BoltMail email marketing services. If your website connects to an externally hosted service, such as to perform DNS queries or connect to a payment provider, these services may experience intermittent timeouts. If you are using BoltHost for your email you may experience slow performance or timeouts when sending or receiving email. Technicians are working around the clock to resolve the situation as quickly as possible.

We will be performing hardware upgrades on BoltMail this Sunday 19th July in the early hours of the morning between midnight and 6am. During this time BoltMail will be unavailable. Total estimated downtime is about 4 hours.

Campaigns scheduled to send during these times will be delayed and will instead send after the work is completed. We recommend not scheduling any campaigns to send during this time.

If you have any questions or concerns please let us know.

There is currently an issue with the web hosting server, technician is investigating now. Please check back here for updates.

Update: server is now back up and running. Downtime was around 20 minutes

An issue on the web server today affected all sites using PHP / MySQL resulting in server error messages.

Issue started approx 1.30pm and was resolved at approx 2.30pm.

Issues was caused by errant Apache process filling up the temporary storage volume.

Web hosting customers including reseller customers may experience a few minutes of disruption to web and email hosting services this afternoon while we apply some urgent software patches.

Resolved

During the night mod security rapidly filled the /var directory with log files, once the directory was full this crashed MySQL.
Mod security logging has been disabled and the MySQL issue has been resolved.

BoltHost web hosting accounts on IP addresses 113.21.225.100 and 113.21.225.102 will be migrated across to a new server this weekend (24th-26th June) in the early hours of the morning. 

This may occur on Saturday, Sunday or Monday morning sometime between the hours of 12am and 6am depending how we go for time and where your account is in the transfer queue.

Downtime should be no more than two hours and should occur sometime between 12am and 6am in the morning.

No action is required on your part unless you have recently received an email from us regarding required IP Address changes (in which case please action ASAP).

If you have any questions or concerns please let us know. 

Things to be aware of:

The default version of PHP will change from 5.5 to 5.6. If you run a PHP site please ensure it is compatible with PHP 5.6 to ensure that there is no downtime to your site. If your site specifically requires PHP 5.5 you can configure this after the transfer has completed. The new server has the ability to run any PHP version that you require. This can be set from the "Select PHP version" option in the Control Panel on the new server.

If your site was built by myself, The Fount or Intuit Creative then no action is required on your part.

Why we are doing this:

We are moving customers to a new server simply because the existing one is now due for replacement. Replacing hardware before it wears out is an important part of maintaining reliability.

New features coming with the new server:

Ability to run any version of PHP from 4.4 up. You can choose the PHP version that you require from the control panel.

All older PHP versions will be hardened with patches to fix security vulnerabilities, even on versions no-longer supported by the PHP development team.

Ability to install/remove most PHP modules with a simple point and click interface from within the control panel.

Ability to set PHP configuration values from the control panel using a simple point and click interface without the need to edit configuration files.

Significant security, reliability and performance improvements. No restarts required for Kernel security updates will reduce downtime.

If you have any questions or concerns please get in touch!

Websites on the IP Address 113.21.225.101 will be moved accross to a new server this Sunday 18th June in the early hours of the morning. Downtime should be between one to two hours and should occur sometime between 12am and 6am.

Affected customers will receive an email from us during the week to notify them of the upcoming disruption.

No action is required on the part of customers unless you have recently received an email from us regarding required IP Address changes.

If you have any questions or concerns please let us know.

Why we are doing this:
We are moving customers to a new server simply because the existing one is now due for replacement. Replacing hardware before it wears out is an important part of maintaining reliability and high uptime.

New features coming with the new server:
Ability to run any version of PHP from 4.4 up. You can choose the PHP version that you require from the control panel.
All older PHP versions will be hardened with patches to fix security vulnerabilities, even on versions no-longer supported by the PHP development team.
Ability to install/remove most PHP modules with a simple point and click interface from within the control panel.
Ability to set PHP configuration values from the control panel using a simple point and click interface without the need to edit configuration files.
Significant security, reliability and performance improvements. No restarts required for Kernel security updates will reduce downtime.

There will be periodic outages to BoltMail this Sunday 11th June and Monday 12th June in the early morning between the hours of 12am and 6am while we perform some hardware and software upgrades. Total estimated downtime is about 4 hours.
Campaigns scheduled to send during these times will be delayed and will instead send after 6am. We recommend not scheduling any campaigns to send during these times.
If you have any questions or concerns please let us know.

Update - Sunday morning's upgrades are taking much longer than anticipated. BoltMail will be unavailable until early Sunday afternoon. We apologise for the delay.

The webserver is down at the moment. We are investigating.

Update 15:00
Server had locked up. Powered down and performed hardware tests. All tests pass. Powered back on.
All running OK at present. Will be monitoring it over night.

There has been a power outage and a UPS failure at the datacenter in Albany. Please bear with us while they restore their systems.
There is also a significant performance degradation on the web server which may be related to the outage, technicians are currently working on it, please bear with us while we work to resolve these issues.

Update Friday 23:00

Information on the Data Center power incident
The Data Center's UPS-B power feed was impacted by a Vector Power surge fault within the Albany area at 13:10 today.
Batteries in the UPS-B bank have been damaged.
The Data Center is currently on generator power. Once Vector has confirmed stability estimated between 02:00 and 20:00 Sat 4th March HD will transition back from generator power to the Vector grid. As the UPS-B battery bank has been severely damaged it is expected that there will be a brief power loss at this time.
A shipment replacement of all batteries within the UPS-B battery bank has been ordered and is being expedited.

Further information on Vector Incident: A HV cable & substation fire has resulted in significant damage to their infrastructure.

Information on the web server outage
The web server is experiencing a suspected fault in the RAID controller that begain after the power surges, this has caused a serious degredation in performance and web services have been temporarily disabled. Technicians continue to work on the issue through the night.

Update Saturday 06:00

The web server is back online and all services are currently operational.

The server appears to have 2 failing DIMM slots that had resulted in an IO slow down, the RAID issue looks to have been a false flag. The server is running on reduced RAM (8GB) while we explore repair options. Performance may be marginly slower than usual due to the reduced RAM. All services are operational.

Update Saturday 12:00

The Data Centre have smoothly transitioned off generator power back to the grid without further incident. Vector have bypassed the burnt out sub feeder and HV road switches and plan further work next week on a long term fix.

Update Monday
Everything is operating OK. Web hosting accounts are not currently showing acurate disk usage, we will need to re-boot the server during the night to resolve this. Downtime will be only a few minutes.

More information will be posted here in the future.

Scheduled PHP server software upgrade 2nd - 3rd April

On the weekend of 2nd to 3rd April will will be performing an upgrade of PHP from version 5.3 to 5.5.

PHP is the software that runs most CMS products and web applications including Joomla!, Wordpress, Drupal, JoJo, Concrete and SilverStripe.

So if your website uses one of these systems, or if you are running scripts written in PHP, then please make sure that your website and PHP scripts are compatible with PHP 5.5.

If you are not sure then we recommend updating to the latest release of your CMS to be sure that your site will continue to operate without error.

We are giving a months notice to allow time for updating your site if required.

If your website was developed by me, The Fount, or Intuit Creative, then you do not need to do anything.

If you need assistance with updating your site please let me know or contact your preferred web developer.

This upgrade affects all web hosting and reseller hosting customers and will result in a short downtime over the scheduled weekend. BoltMail will be unaffected.

Kind regards
Chris

--
Chris Bolt
web developer & designer
www.christopherbolt.com
BoltHost | BoltMail
p: +64 9 950 3348
m: +64 21 267 3758

PO Box 8014, Symonds Street, Auckland 1150, New Zealand

There are issues connecting to email this morning, a technician is working on this now, check back here for updates.

This issue has now been resolved.

The web hosting server is running very slow this morning technicians are looking into it.
This is unrelated to the updates over the weekend which went through without issue on Saturday night.
I will update here once resolved.

Over the coming weekend of 27th - 28th Feb 2016 we will be performing a number of long over-due system and security updates including improvements to email security, spam detection, email anti-virus, support for FTP over TLS (FTPS) and also it will become possisble to install SSL certificates without the need for a dedicated IP address.

This will affect all web hosting accounts and re-seller web hosting accounts. BoltMail is unaffected.

You may experience some minor disruption to web and email services over the weekend, however any disruption should be minimal.

New Secure Mail Settings for Monday
From Monday when you access your email in an email program such as Outlook or Apple Mail you may receive a pop up or warning message regarding an SSL Certificate.
Or you may already see such messages.
If you do not see such messages then you do not need to do anything. However if you do please follow the instructions below:

As of 6PM Friday 13th 2012 all legacy IPs on the 60.234.*.* range allocated to us by Orcon have been NULL routed by Orcon. 
We did not get a choice in the matter, neither were we notified. This is despite assurances that we would have use of these IPs until the 23rd. 

Because of this we are moving forward our plans to migrate to new IP addresses that was scheduled for Sunday the 16th and taking this action now. All domains using our nameservers or custom nameservers running on our servers will be back online soon once this migration is completed.

Any domains using external nameservers should update their DNS now! Do not wait till Monday morning as originally requested.

The new IP address for all websites hosted on da2.christopherbolt.com that do not have their own fixed IP is: 103.6.28.18

The following websites are exceptions and have been allocated a different IP:

stumpbusters.co.nz - 113.21.225.98

cadencecarpets.com - 113.21.225.98

cadencecarpets.co.nz - 113.21.225.98

gatagencies.co.nz - 113.21.225.98

care4.co.nz - 113.21.225.98

care4graffitiremover.co.nz - 113.21.225.98

care4nz.co.nz - 113.21.225.98

care4nz.com - 113.21.225.98

care4products.co.nz - 113.21.225.98

Websites on da1.christopherbolt.com and da3.christionerbolt.com are not affected.

We sincerely apologizes for this inconvenience and are working around the clock to resolve this all weekend.

Due to network restructuring at the datacenter the IP addresses of websites hosted on da2.christopherbolt.com will be changing.

This will occur on Sunday evening July 15th and will result in some disruption to services during the night.

Your action may be required: If you have any domains that are NOT using our nameservers (ns1.christopherbolt.com, ns2.christopherbolt.com) these will need their DNS updated to the new IP address first thing on Monday morning 16th July (or better still Sunday evening) otherwise they will eventually stop working.

Domains that ARE using our nameservers will not require any action on your part.

The new IP address for your hosted websites on da2.christopherbolt.com will be 103.6.28.18

Please check all your domains, you must update the DNS records for any domains NOT using our nameservers to this new IP on Monday 16th July.

We apologise for any inconvenience caused.

We are working to set up a proxy so that domains not changed on the day will continue to function for a few days after the change. This should allow adequate time for DNS propogation and any delays by 3rd parties.

Due to network restructuring BoltMail users may experience intermittent service outages on Sunday 8th July 2012.

We recommend not scheduling any email sending on this date.

Yesterday during an update to our server software the support ticket system freaked out and re-opened a number of old tickets.

This problem also alerted us to an issue where the ticketing system had failed to open some tickets in the past.

We apologies for any frustration this has caused.

The issue is now resolved and the system should now be functioning correctly.

Today at about 2:40pm BoltMail crashed and was unavailable for about 20mins while we restored normal service. While it was down we took the opportunity to re-launch it with several new and exciting features that we've been working on, and of course we assessed the source of the problem and fixed it for the future. Any email campaigns scheduled to send during the crash, or in the process of sending, have resumed sending once the system came back online. 

Please see:

http://www.christopherbolt.com/support/announcements.php?id=1