Root Cause Analysis (RCA)
Time: 2:36pm - 4:15 pm Pacific Standard Time
Type: Inability to Login
Affected Clients: 179
- On February 2nd 2017 at 2:36 pm Pacific Standard Time, Velocify detected a software error affecting one of our redundant systems, which prevented login for some Lead Manager clients. Velocify views system uptime as one of our highest priorities and we sincerely apologize for the impact this problem had on your business. We value all our clients and take our role in your success seriously.
- During login, a session state is stored in a high performance open source memory cache, supported by a server farm. We are capable of handling any errors in this software due to redundancy. However this time, a very rare corner case caused the server farm to be affected, creating a failure of our usual redundancy processes. This required us to trigger a secondary redundancy process. Velocify maintains two full production environments and in response to this error, began migrating clients to the unaffected environment. The migration process was completed at approximately 4:15 pm Pacific Standard Time. Affected clients were seeing service restoration as the migration progressed.
- In order to prevent this problem from occurring again, a software hot fix was rolled out the following day. We believe this error was an anomaly and do not expect this problem to occur again.
During any system wide problem, updates and notifications will be posted on our Now Trending page: Velocify Now Trending
Please contact our Support Team with any additional questions.
Monday - Friday 6am to 6pm Pacific Standard Time