All times Pacific Standard Time
Incident Date: 05/09/2017
Type: Lost Call Recordings
On May 11th, 2017 Velocify leadership received communication from our telephony partner regarding the loss of call recordings on May 9th, 2017.
Here is summary statement provided by our Partner.
On May 9, 2017 from 10:48am to 11:31am Pacific Daylight Time, the Programmable Voice Recording service was unable to upload a significant number of Recordings to the Recording Storage system. This was due to a software defect introduced in a previous deployment.
Once this defect was addressed and new recordings were processed successfully, telephony engineers resumed the upload process of failed recordings from the Recording service hosts. During the recovery process, an unexpected failure mode occurred in deployment tooling software that shut down hosts processing failed recordings. As a result, a percentage of recordings for calls ending during the incident time window were permanently lost.
Unfortunately, in this case, the percentage of call recordings that were dropped are unrecoverable. Please note, some of these records may not contain actual calls, only the activity of an attempted call or a pre-recorded voicemail where an actual lead was not contacted. If you attempt to access a recording from May 9th, 2017 and it was deleted, you will hear a: “(404) Not Found” error.
The corrective action that will be put in place to address root cause moving forward includes:
- Modifying Standard Operating Procedure to upload failed recordings safely
- Add a safety flag to indicate that inactive hosts should not be shut down
- Overload and Stress Testing for recording services
- Automated testing of re-drive processing prior to every deployment
Velocify clients are our highest priority and we apologize for this inconvenience.
If you have any further questions, please contact your Velocify Account Manager.