ProofMe — Amazingly Simple Review and Approval Platform

ProofMe Outage Status

Jun 12, 2018

ProofMe engineers are continuing to migrate customers to the new version of ProofMe.com. We are currently working through a migration queue, you will receive an email once your account has been migrated.


Jun 11, 2018

ProofMe engineers have restored service for select customers on the new version of ProofMe.com. We are currently monitoring and validating application performance and functionality for these customers. If the new version of ProofMe remains stable, we will continue efforts migrating existing ProofMe customers over to the new instance this week.


Jun 7, 2018

ProofMe engineers have resolved issues with the team and subscription migration process. We have validated functionality of the new version of ProofMe.com. We are currently enhancing our migration script to optimize account migration efforts. Once these efforts are complete, we will begin migrating customers onto the system. While migrating customers, we will limit new account sign-ups in order to prioritize support for existing customers.


Jun 6, 2018

ProofMe engineers are continuing efforts to finalize the user migration process. We have resolved an issue with user account migration and are currently working on team and subscription migration. If we can validate successful account migrations, we will resume bringing accounts over to the new ProofMe version as planned.


Jun 5, 2018

ProofMe engineers are continuing efforts to resolve account migration issues. We also continue our evaluation efforts related to viability of ProofMe's platform going forward and additional contingency plans.


Jun 4, 2018

ProofMe engineers conducted testing and migration efforts over the weekend as planned. We have identified several issues impacting some migrated accounts. These issues are impacting our ability to complete additional account migrations into the new instance of ProofMe. We are carefully evaluating options around the continued viability of ProofMe's platform going forward.


Jun 1, 2018

ProofMe engineers have successfully migrated an initial batch of user accounts and data onto the new ProofMe instance. We will be conducting testing of migrated accounts in this new environment over the next several days. If functionality with migrated accounts is validated, we will begin migration of additional user accounts starting next week.


May 31, 2018

ProofMe engineers have successfully brought up a new version of ProofMe.com internally. We are currently preparing the data archive to allow us to import active users into the new ProofMe.com. Because of the volume of data in this archive, processing is taking some time to complete. Our current projection is that we will be able to test importing of user data sometime within the next 24 hours.


May 30, 2018

ProofMe engineers have completed several refactoring efforts on the ProofMe application codebase in an attempt to stabilize the service. Due to limitations in the underlying data cluster's capacity to handle load, we have been unable to restore ProofMe service for all users. We are planning to bring up a new version of ProofMe and migrate active users over in an effort to reduce data processing requirements. We expect to know the results of this effort within 24 hours.


May 29, 2018

Last week, ProofMe engineers brought ProofMe.com online for a brief period of time. Unfortunately, reconfiguration efforts leading up to that moment did not result in a stable application. ProofMe's data cluster is still suffering from performance issues under load. In an attempt to resolve these issues, engineers worked on refactoring several portions of the application over the weekend. We will reassess our options going forward within the next 48 hours.


May 24, 2018

ProofMe engineers are investigating and addressing issues related to a non-performing data cluster. This is causing an outage for all ProofMe users presently. We will post more information regarding status and timelines as it becomes available.


May 23, 2018

ProofMe engineers have successfully completed regression testing. We are aware of an issue involving some users receiving past due notifications for proofs that were active prior to the outage. This issue is being addressed by our support staff. We have opened up access to ProofMe.com for all users and are closely monitoring performance and functionality.


May 22, 2018

ProofMe engineers have diagnosed and applied a fix for the issue we encountered yesterday during regression testing. This fix involves a repair operation on the data cluster that is in progress. We anticipate this operation will conclude sometime this evening. We will resume regression testing tomorrow. If no additional issues are detected, we expect to open ProofMe.com to customers by the end of this week.


May 21, 2018

Over the weekend, ProofMe engineers applied the final scaling optimizations to our data cluster. We are putting the finishing touches on our production cluster and completing regression testing today. We have identified a few issues through this effort that we are currently addressing. We will begin bringing customers onto ProofMe.com once we have resolved all issues. We thank our customers for your continued patience as we work to get ProofMe.com back up and functioning in a normal capacity for you.


May 18, 2018

ProofMe engineers successfully completed initial testing on ProofMe.com last night. ProofMe behaved as expected during our testing efforts. We are applying one additional scaling optimization on our data cluster that will be processing over the weekend. Anticipating this will be a successful scaling optimization, we expect to bring customers onto a live ProofMe.com sometime Monday. Additional updates will be provided Monday morning. Once the system is back online, information regarding subscription credits or refunds for the disruption in service will be provided.


May 17, 2018

Our data cluster build process has resulted in a functional cluster. We are now taking steps to bring ProofMe.com online internally and will be conducting tests against the enhanced data cluster. We anticipate bringing customers back onto ProofMe between tomorrow and Monday. More specific timing will be shared via tomorrow's update.


May 16, 2018

Data cluster build operations are still in progress and nearing completion. We have successfully applied many of the scaling optimizations as planned. There are several steps remaining in the process to bring ProofMe back online. At present, we are still targeting a return to service this week. We thank our customers for your continued patience.


May 15, 2018

We continue working diligently towards the the restoration of ProofMe.com service. Data cluster build operations are currently in progress. ProofMe engineers are monitoring this multi-step process and fine-tuning performance efforts. Scaling optimizations will be applied once the initial cluster build is complete. We remain on-track to restore service this week. Progress and timeline updates will continue to be published until this issue has been resolved.


May 14, 2018

Over the weekend, ProofMe.com engineers completed the initial phase of data cluster optimizations utilizing new parameters. This is a multi-step process that will take several days to complete. We will continue to post updates informing of our progress and timelines. The productivity of our customers is of utmost importance to us, the restoration of ProofMe remains our top priority.


May 11, 2018

ProofMe engineers continue to make progress to bring ProofMe.com back online. By working to create an upgraded clean instance of the data cluster, we have made a number of improvements in the underlying data cluster configuration that will improve stability of the service after it is back online. Once the clean instance of the data cluster is online, it will be replicated to a second scaled-up cluster. This process could take up to several days due to the amount of data we are dealing with. We remain on-target to restore service for our customers next week. Information regarding subscription credits or refunds for the disruption in service will be provided once ProofMe.com is back online.


May 10, 2018

Our engineers are currently building a clean instance of the data cluster on an upgraded version of the database software. This effort involves configuration optimization and scaling of the underlying data cluster. We're targeting a return to service within the next week. We will continue to advise our customers on any impacts to this timeline. We appreciate your continued patience during this process as we work hard to get ProofMe back up and running for you.


May 8, 2018

On April 30, some ProofMe.com users began reporting intermittent service disruptions and inconsistent search results. These issues were caused by out-of-sync data indexes in ProofMe's distributed database cluster.

After an attempted repair of the out-of-sync indexes, the application entered a failure state and became unavailable for all users. No customer data has been lost. No sensitive data has been leaked or compromised in any way.

Service disruption is never completely avoidable, but this level of disruption is unacceptable. To ensure that this does not happen again, we're working on multiple improvements to application architecture and operational procedures. This page is designed to provide customers with up-to-date information on the current status of this outage and any future issues that may affect ProofMe users.

To all customers affected by the outage: we are sorry. We apologize for the hardship and disruption this has caused you and we take your feedback very seriously. We are working around the clock to rectify this issue as quickly as possible and will put a long-term solution in place to ensure reliability in the future.

Timeline

  • April 28 - April 30: Initial reports of intermittent service disruptions and inconsistent data retrieval results. Investigated and resolved issues.
  • May 1: Attempted repair of search indexes resulting in ProofMe outage
  • May 2 - May 3: Continued failed attempts to repair search indexes
  • May 4 - May 7: Architect contingency plans, additional engineers added to issue
  • May 8: Multiple contingency plans initiated

Recovering ProofMe.com

We anticipate a full restoration of ProofMe service. We have implemented an enhanced cluster architecture and are in the process of configuring and migrating data to the new cluster. Once operational, we will conduct testing and migrate live traffic onto the new cluster.

Future Architecture Improvements

The ProofMe team had previously put in place a migration plan that would involve moving ProofMe onto a modern and supported data store. Unfortunately, this level of migration takes time to complete and was not yet ready to use at the time of this incident. This plan is still in place and we are accelerating the implementation timeline for the plan in response to this incident.

Root Cause Analysis

We will be conducting a root cause analysis and posting our findings on status.proofme.com.