iClassPro experienced extended, intermittent performance problems last week that were resolved late in the evening on October 12th. iClassPro would like to be as transparent as possible about what happened, how it was resolved and what to expect next.

We originally planned our scheduled maintenance for August 6th. The plan of action was to migrate our web servers to new hardware, running a newer technology stack and protected by additional security measures. As that date approached, we felt that more time was needed to adequately test, prepare and execute the hardware upgrade. We rescheduled the maintenance window for October 6th.

(Unfortunately, in the meantime, we experienced a hardware failure on one of the old web servers that caused an unexpected service interruption in mid-August.)

On October 6th, we executed the DNS change that began routing traffic through our new web servers. There were a few technical issues that we worked through right away and the system was placed back into “production mode”.

Once we started handling the full load on the new web servers, some intermittent connection errors between the web servers and the database servers began getting logged. iClassPro opened tickets with our system administrators who worked diligently on the issue. Tickets were opened with the operating system’s enterprise support team. Technicians worked through day and night trying to find the cause of the intermittent connection errors.

One consequence of these errors was that a payment may have successfully filed in the payment gateway, but then failed to file successfully in the iClassPro database. That is why we strongly urge everyone to check their gateway transaction reports and compare them to the Bank Deposit Report in iClassPro for the affected time period.

After a week of determined effort to find and fix the problem, we made the decision to switch back to our old web servers. We still have the system administrators and operating system support team working on the problem. We are attempting to recreate it without the heavy load of being “in production”. We are also evaluating alternative ways of achieving the same goals.

It is mandatory to continually update hardware and software to keep your data secure. Updating systems is never easy and iClassPro does its best to diligently plan and prepare such updates with minimal impact on our valued clients. We are very sorry for the difficulties that last week’s system issues caused you and your customers. We hope that you understand the need for such updates and the complexities that can be involved.