Delays in master data synchro loads

We've now resolved the incident. Thanks for your patience.

After successively deploying a more performant user synchronisation process across the customer pool between February 3rd and February 8th we have been closely monitoring the updates of user data and the performance of the process. No issues were encountered in this period. The delay of loading files has consistently been within 24 hours(the average was between minutes and a few hours).

We will continue to monitor and, using the experience gathered in this improvement, we will also investigate other synchro processes like task and task assignment to see if performance improvements can also be gained there.

Positive news ! The first deployments did not cause any issue and confirmed the expected improvements.

As such, we keep going with our progressive and monitored deployments.

We expect all customer pools to be covered in the course of next week, resolving all delays in the process.

We successfully completed the testing of an performance-improved user synchro process.

These changes will be deployed to production in course of coming 2 weeks. The deployments will be progressive (per customer pool) to mitigate impact and facilitate monitoring.

Stay tuned for more progress updates !

At this time, a few customers only should still experience delays of more than a day.

Our team is meanwhile finalizing a batch of improvements that is expected to resolve all remaining impacts. A set of improvements is already being tested thoroughly on our STAGING environment, and we expect to move ahead the coming few weeks.

The general delay has improved with the vast majority of customers having data fully loaded up to yesterday. Also, good progress has been made on the development side - stay tuned !

A few customers are facing more delays, but these case may not be related to the present issue and are undergoing a separate investigation.

The operational situation has not evolved, we continue to closely monitor the situation and take mitigation actions when required.

Our team is investigating on software improvements to completely resolve the situation. Given the criticality of this process, this development will require sometime for proper regression testing.

Meanwhile, we are still experiencing delays, and the situation worsened a bit for a number of customers with delays of more than 2 days. We are looking into mitigation operations to decrease this backlog as soon as possible.

Several backend improvements and mitigations were implemented in the past 2 days. These are already easing up the situation putting most delays under control.

The systems are now recovering under our close monitoring. We'll analyse how much progress is made in the coming hours and days and determine our next steps from that basis.

Current status :

• 50% of our customer pools are fully up-to-speed

• 30% have data from today itself queued for processing

• 20% have older files in queue

The issue is confirmed, and several causes were already identified, among which a sensible increase of user data volumes and a snowball effect in the queueing. The delays are ranging from 1 to 7 days, whereas in normal conditions most files are loaded within 24 hours maximum. We do not expect any loss, and aim to process all the data received ASAP on top of resuming normal conditions.

The delays between uploading your synchro data and their introduction into our application database have increased since mid-November. We are currently investigating this issue.