Testing… it is one of those things we all universally agree is a great idea and should be law. We all hop on our high horse and say things like “We will test and re-test until this migration project is iron tight. Leave no stone unturned”. HA HA HA! What we really mean is testing is awesome and someone else should do it because we do not want to do it.
It is okay. We are in a safe place. Remember in the previous parts of this series we agreed to not getting defensive. We talked about the SharePoint sins of our past and how we should have organized our content better and like one of those workout places we don’t go to; this is a judgment free zone. So, it is okay to admit that while you talk big about testing, you aren’t going to do it really.
To help us better understand the reality of testing I am going to tell you about the migration project I just finished up. Names will be changed to protect the innocent, but in reality, they spent so much time with me on this project there is no way they would read anything I wrote. So, they will never know I am writing this.
Their project was to move a SharePoint 2010 on-prem environment to SharePoint 2016 on-prem. I tried to get them to move to SharePoint Online, but licensing issues stopped that conversation pretty quickly. The best compromise we came up with is we tried to design 2016 in ways that would make the transition to Online later easier. We weren’t perfect in this design, but it was at least a topic as we built.
For the migration process, we decided to use the database attach method for the bulk of the upgrade and a 3rd party place for those tough spots. Cool. The long story short of how the database attach process goes is you take a backup of your SharePoint 2010 content database and attach it to 2013. That changes the database schema to 2013. Then you run a PowerShell cmdlet that upgrades the site collection schema from 2010 to 2013. With that done you attach the database to SharePoint 2016 and boom, you have a 2016 site collection. We also ran a script on 2016 that migrated the users from classic to claims.
The great thing about this process is it is really easy to test. So about six months ago we tested the process and brought up all of the 2010 content in a 2016 test farm. From a pure infrastructure point of view, it was a smashing success. <que your favorite victory music> But folks, that is only 10% of the battle. <sad trombone>
That is okay. We knew that was only part of the battle so what did we do? We made that new 2016 environment accessible to the users and said: “go forth and test.” This particular customer is interesting in that IT, the people I am working with, aren’t responsible for the content or the SharePoint functionality within the site collections. The departments build out their SharePoint solutions for their problems. My IT friends help them along the way and support them very well but the owner of the sites is the business.
What this meant was IT doesn’t have a list of the different InfoPath forms (puke), workflows, or 3rd party web parts that are in use or how they are being used. So, we couldn’t test. All we could do was the migration. But that is okay. The business said they had it covered. They would test over the next few months so we could work out all of the kinks. And they did… kind of…
This is where testing went sideways. I know they got in and tested. They surfaced some issues over those months. And we squashed every one of them. It was great. Made us feel like man, they are really getting in there and testing. Look at all of these issues they are finding, and we are fixing. Things were looking great!
In reality, at no fault of their own, testing was very haphazard. They would mess with this or mess with that. There was not an organized testing plan. The users would be thinking about this crazy web part they were using so they would go look on the test farm, see it not rendering correctly, and open a ticket. We would fix it, and there was much rejoicing. We had spot checks defined but not a list of these are the seven most common things they do in SharePoint.
After a long weekend of us (IT) working to migrate about a Terabyte of content from SharePoint 2010 to SharePoint 2016 and some lite weekend testing by a couple of users Monday morning came. Good news, it went pretty well. But, it was not the ironclad we had expected from our “testing.”
It turns out there were quite a few key things missed in testing. Most were quick little “jiggle the buttons” fixes, but a couple were kind of awkward. Two worth mentioning here. The worst one was a 3rd party webpart that just flat out didn’t work and was a key business driver. The other was the InfoPath forms were mad about our URL change. One fix required lots of calls with a vendor and the other required learning to manually edit XML files and repackage them.
This lead to m changing my testing philosophy for future projects. Usually when you ask people what should be in the test plan you either grab a generic list off the internet or you ask users “what should we test”? That can work but here is my more specific new way of positioning it. Tell me everything you do with SharePoint on Mondays.
That seems odd, but I think it would work better. Why? For one Monday mornings are a high utilization time for most SharePoint servers. People come in blurry eyed from the weekend and need something concrete to do while they drink their caffeine. SharePoint is usually a structured environment where they go fill out TPS reports or consume information, giving them some semblance of productivity through the fog. The other reason? You always do the migration on the weekend. So, if you can validate with testing all of their Monday activities you should have a better day one.
Lessons to share with you
Now that I have made you read through all of the fun we had let’s talk about things I would differently next time. First, it would be better to understand the vendors involved. If you have 3rd party software, have the license and support information at your fingertips. Which you probably do but how about support hours? Wish we had known that support was 9-5 on weekdays only, would have set our expectations of solving that problem differently going in.
Secondly, I would have had a list of those Monday morning activities. They hit most of the problems within 30 minutes of coming in on Monday. People knew what they did on Monday’s, why didn’t we test that already? We had six months of testing.
Here was how those calls went on Monday. Caller: “X is broke” Support: “Does it work in test?” Caller: “I think so” Support: <checks Test and sees it doesn’t work> Nothing that was tested and fixed in dev was broken post-migration.
The migration went really well. We did find some oddities post migration I don’t think we would have ever found in testing. For example, blog comments on one of the corporate blogs didn’t work. SharePoint is a terrible blogging platform so no shock there. We had some bumps and bruises, there was some yelling, but at the end of the day the business is still up and running, and we are on to the next migration. Have a plan, have the right tools, have the right people, and for goodness sakes test! If you can handle all of that, I think you will be happy at the end of your migration once you get some sleep.