Issue
Incident Report for iClassPro

iClassPro experienced extended, intermittent performance problems last week that were resolved late in the evening on October 12th. iClassPro would like to be as transparent as possible about what happened, how it was resolved and what to expect next.

We originally planned our scheduled maintenance for August 6th. The plan of action was to migrate our web servers to new hardware, running a newer technology stack and protected by additional security measures. As that date approached, we felt that more time was needed to adequately test, prepare and execute the hardware upgrade. We rescheduled the maintenance window for October 6th.

(Unfortunately, in the meantime, we experienced a hardware failure on one of the old web servers that caused an unexpected service interruption in mid-August.)

On October 6th, we executed the DNS change that began routing traffic through our new web servers. There were a few technical issues that we worked through right away and the system was placed back into “production mode”.

Once we started handling the full load on the new web servers, some intermittent connection errors between the web servers and the database servers began getting logged. iClassPro opened tickets with our system administrators who worked diligently on the issue. Tickets were opened with the operating system’s enterprise support team. Technicians worked through day and night trying to find the cause of the intermittent connection errors.

One consequence of these errors was that a payment may have successfully filed in the payment gateway, but then failed to file successfully in the iClassPro database. That is why we strongly urge everyone to check their gateway transaction reports and compare them to the Bank Deposit Report in iClassPro for the affected time period.

After a week of determined effort to find and fix the problem, we made the decision to switch back to our old web servers. We still have the system administrators and operating system support team working on the problem. We are attempting to recreate it without the heavy load of being “in production”. We are also evaluating alternative ways of achieving the same goals.

It is mandatory to continually update hardware and software to keep your data secure. Updating systems is never easy and iClassPro does its best to diligently plan and prepare such updates with minimal impact on our valued clients. We are very sorry for the difficulties that last week’s system issues caused you and your customers. We hope that you understand the need for such updates and the complexities that can be involved.

Posted 6 days ago. Oct 17, 2017 - 11:19 CDT

Resolved
We have identified and resolved the issue causing system instability.

There were some incidents of duplicate payments being processed between Friday, 10/6/17 and Friday, 10/13/17. As a precaution, we recommend checking the payment gateway to verify all payments are correctly recorded in iClassPro. If you need assistance, please contact us via phone (903-353-9329) or web (https://support.iclasspro.com/hc/en-us/requests/new).

Customers using PayWay may need to make adjustments their settings to include the following IP addresses:
206.188.31.112, 206.188.31.113, 206.188.31.115, 64.27.116.240, 64.27.116.241, 64.27.116.242
Posted 10 days ago. Oct 13, 2017 - 10:42 CDT
Update
In order to prevent duplicate transactions as we work to restore the full functionality of the application, we are temporarily disabling the customer/parent portal. Work on returning to full functionality is ongoing and we will update the status page as information becomes available.
Posted 11 days ago. Oct 12, 2017 - 10:02 CDT
Monitoring
We are continuing to monitor the application status and will provide updates as they come in.
Posted 12 days ago. Oct 11, 2017 - 16:40 CDT
Investigating
Some users are reporting an issue that is impacting access to the iClassPro application. Please accept our apologies for the interruption, we are working to restore functionality as quickly as possible.

If you are seeing errors when attempting to process payments, please double check your gateway to verify if the payment was actually processed or not. These payments may need to be manually recorded in iClassPro.
Posted 12 days ago. Oct 11, 2017 - 15:21 CDT