When Technology Fails: Woes With SharePoint Online Restore this Library
In April, Microsoft launched the Restore this library feature for SharePoint Online document libraries. I like the functionality very much because users can restore files without administrator intervention or, even worse, the need to ask Microsoft to restore a complete site collection from backup.
The SharePoint Online Restore this Library Feature
Microsoft built the Restore this library functionality using the experience gained with a similar feature called Restore your OneDrive that’s available to OneDrive users with an Office 365 subscription. In both cases, the technology depends on knowing when changes occurred and the existence of previous versions of files that can be used to roll back. The versions generated automatically for Office documents stored in SharePoint Online and OneDrive for Business ensure that a high degree of granularity is available for a restore, but this isn’t always true for other formats. It all depends on how files are saved in SharePoint (for instance, PDFs edited in the Adobe cloud are versioned properly).
Using Restore this Library
Restore this library is available in the settings (cogwheel) menu of document libraries. When invoked, SharePoint queries the Microsoft Graph to retrieve a list of changes made to the library over the last four weeks. The user can then browse the list and decide what changes to undo (Figure 1).
A set of precanned queries for yesterday, one week ago, and three weeks ago are also available to select sets of changes, or you can use a slider to go back to any point up to 30 days ago.
After selecting the set of changes to undo, click Restore and SharePoint reverses the changes. Depending on how many changes there are to process, this could take some time.
When Things Go Wrong
On July 30, an Office 365 tenant in Asia experienced a problem when a workstation was infected with the Format ransomware. The infection spread to SharePoint and encrypted thousands of documents in a Projects subfolder in a document library. No other folder was affected.
The first action taken by the administrators was to rescue files deleted by the infected user account from the site recycle bin. This recovered about 10,000 files, but over 32,000 files in the folder remained encrypted.
No problem. When a file is encrypted, a new version is generated, so it should be possible to use Restore this library to discard all the changes made by the ransomware. Before doing anything, the administrators synchronized all the files in the document library (both good and bad) to another PC, broke the synchronization link, and deleted all the infected files in the synchronized copy to create a set of “good” files as a backup.
Then they started the Restore this library function to roll back the changes wreaked by the infection. At this point, the infection had happened eight days prior, so they needed to use the slider to find the point in the set of changes just before the infection started. The first issue appeared when the slider became slower and slower as it navigated the list of changes. After many hours of struggling to move the slider (and several restarts), the administrators got it to 9:35AM on August 1.
Even though this point was two days after the infection started, they decided to proceed with the restore, which worked and rolled back all changes before 9:35AM on August 1. Unhappily, as you’d expect, many encrypted files remained in the folder.
Further attempts to roll back changes prior to August 1 also failed and the administrators eventually had to resort to choosing the three-week restore option in the knowledge that this would remove many good files. However, they felt that they had no choice but to get the document library back to full working order in the hope that the lost files could be recovered from other repositories or recreated.
Lessons Learned
This tale of woe revealed several issues surrounding the Restore this library feature.
- It struggles with very large document libraries. Large document libraries tend to generate lots of change activities. SharePoint Online looks for those change activities to give administrators the chance to select the restore point. It seems like the slider used in the SharePoint GUI struggles with large number of activities by slowing to a sluggish crawl before stopping.
- More precanned restore points (for example, three days, five days, and two weeks) would help by avoiding the need to use the slider. In fact, it would be best to allow the administrator to enter a date and time for SharePoint Online to use the restore point.
- Restore this library processes everything for a document library. You can’t set the scope to a specific folder.
- Restore operations to roll back tens of thousands of changes take time. Better feedback is needed for administrators so that they know exactly what point a restore has reached. The screen used by SharePoint Online is pretty, but that percentage progress doesn’t increase often large quantities of files are being processed (Figure 2). For audit purposes, an emailed list of operations processed by the restore would also be appreciated.
New Technology Takes Time to Bed In
Restore this library is new technology and it will take time to experience the full range of experiences that customers can throw at it. In this instance, the restore worked after hours of effort to get to an August 1 restore point, but the slider couldn’t go back far enough to do the full job. Imperfect feedback to administrators is something Microsoft can quickly fix. I hope the same is true to improve the scalability of the Restore this library feature. After all, SharePoint Online supports up to 30 million files and folders in a document library. Enough said.