No-nos, Gotchas, Warnings, Best Practices, and Things to Remember for SharePoint 2013 Distributed Cache Service

Someone told me at SPC12 that Distributed Cache in SharePoint 2013 is User Profile Sync Service in SharePoint 2010. If you are seasoned SharePoint professional, you know that doesn’t bring smile on your face.

After reading Microsoft’s TechNet articles, one would assume that statement made at SPC12 was 100% on money. Reading TechNet articles on Distributed Cache, one should feel like they are walking on terrain of landmines and one wrong step could bring down the SharePoint farm and un-stabilize the SharePoint 2013 environment. If you still haven’t got the point, read this – Microsoft has raised first ever warning in SharePoint Products history stating “The Distributed Cache service can end up in a non-functioning or unrecoverable state if you do not follow the procedures listed in TechNet article. In extreme scenarios, you might have to rebuild the server farm“.

I don’t recall Microsoft ever issued warnings in this matter in previous four releases of the product which could bring down the farm, not even for User Profile Service in SharePoint 2010. Having worked on complex SharePoint 2013 farm over the last year, I have noticed that these warnings will surface it’s ugly head one time or another. Unfortunately many of core SharePoint 2013 features are dependent on Distributed Cache and disabling this service is not an option at this moment. If you ever notice that users can’t post anything in their my site news feed or people can’t login properly, either your Distributed Cache is lacking memory cache or Distributed Cache is dead and you may force to make necessary memory allocation changes for Distributed cache or revive dead Distributed Cache service.

Here are the major No-nos, Gotchas, or Warnings SharePoint admins should keep in mind regarding Distributed Cache service. Please note that most of these information directly derived from TechNet, Steve Peschka, and Josh Gavant‘s articles.

By default, 10% of Total RAM and 5% of Total RAM allocated to Cache Size to Distributed Cache Service

When SharePoint Server 2013 is installed, it assigns the Distributed Cache service 10 percent of the total physical memory on the server. The Distributed Cache service uses half of that memory allocation for data storage (also known as cache size, 5% cache size), and the other half of that memory allocation is used for memory management overhead. When the cached data grows, the Distributed Cache service uses the entire 10 percent of the allocated memory. http://technet.microsoft.com/en-us/library/jj219613.aspx

Never Administrator the Distributed Cache Service from the Services Snap-in in Administrative Tools

The Distributed Cache depends on Windows Server AppFabric as a prerequisite. Do not administer the AppFabric Caching Service from the Services window in Administrative Tools in Control Panel. Do not use the applications in the folder named AppFabric for Windows Server on the Start menu. Always administrator Distributed Cache using PowerShell or using Central Administration Services on Server Page. http://technet.microsoft.com/en-us/library/jj219613.aspx

Total memory allocation for Distributed Cache host shouldn’t be more than 16 GB

On a server that has more than 16 GB of total physical memory, allocate a maximum of 16 GB of memory to the Distributed Cache service. If you allocate more than 16 GB of memory to the Distributed Cache service, the server might unexpectedly stop responding for more than 10 seconds. If the cache host’s size is larger than 16GB/8GB, garbage collection could take long enough to cause a noticeable interruption for clients. If you require more memory, you can configure the Distributed Cache service to run on several application servers or dedicated Distributed Cache servers. As far as server specs, for dedicated Distributed Cache servers, your Virtual or Physical Server RAM shouldn’t be more than 24 GB (16 GB allocated to Distributed Cache and other for OS). – http://technet.microsoft.com/en-us/library/jj219572.aspx and http://blogs.msdn.com/b/besidethepoint/archive/2013/03/27/appfabric-caching-and-sharepoint-2.aspx

Dynamic memory is not supported in SharePoint 2013 environments nor App fabric servers.

The Distributed Cache service can run on either a physical or virtual server. When using virtualization, do not use Dynamic Memory to manage shared memory resources among other virtual machines and the Distributed Cache servers. The memory allocation for virtualized Distributed Cache servers must be fixed. http://technet.microsoft.com/en-us/library/jj219572.aspx

Plan to run Distributed Cache in Dedicated Mode

The Distributed Cache service can run in dedicated or collocated mode. When running in dedicated mode, the Distributed Cache service is started and all other services are stopped on the server. In collocated mode, the Distributed Cache service is running along with other services on the server. Dedicated mode is the recommended mode in which to deploy the Distributed Cache service. Review Steve Peschka’s article – http://blogs.technet.com/b/speschka/archive/2013/08/29/dedicating-servers-to-distributed-cache-in-sharepoint-2013.aspx

Developing code against the Distributed Cache of SharePoint 2013 is NOT supported

If you are using custom applications in SharePoint Server 2013 which use the AppFabric client APIs, or are creating custom caches, you should create a separate AppFabric cache cluster to support your custom applications. Do not use the AppFabric cache cluster supporting your SharePoint Server 2013 farm. Run your separate AppFabric cache cluster for your custom applications on separate servers from the servers dedicated to your SharePoint Server 2013 farm.  Microsoft has stated that additional named caches should not be deployed to the SharePoint AppFabric cluster (i.e. by using the New-AFCache cmdlet). If you need a cache for a custom solution, you’ll need to deploy a separate AppFabric cluster (or server) and create the cache there. Then point your solution at the external AppFabric cluster. There’s also no supported way to add your own cached items to SharePoint’s named caches. http://technet.microsoft.com/en-us/library/jj219572.aspx & http://blogs.msdn.com/b/besidethepoint/archive/2013/03/27/appfabric-caching-and-sharepoint-2.aspx

Install Windows AppFabric using SharePoint Server 2013 prerequisite installer

It is important to note that SharePoint 2013 RTM uses Windows AppFabric 1.1 for Distributed Cache Service. When the SharePoint Server 2013 prerequisite installer runs, it installs Windows Server AppFabric 1.1. This is the recommended approach for installing Windows Server AppFabric on a server that is running SharePoint Server 2013. If you already have Windows Server AppFabric 1.0 installed on the server (SharePoint 2013 Preview was based on Windows Server AppFabric 1.0) before you run the prerequisite installer, you must uninstall Windows Server AppFabric before you run the prerequisite installer. If an administrator decides to install Windows Server AppFabric manually, the administrator must install the CacheAdmin, CachingService, and CacheClient features, and use the /gac switch. For more information see Automated Installation (AppFabric 1.1 Caching) in the MSDN Library. http://technet.microsoft.com/en-us/library/jj219572.aspx

Plan to have Memory Allocation to all Distributed Cache Host are Same

You must ensure that the memory allocation assigned to the Distributed Cache service is the same on all servers that are running the Distributed Cache service. An administrator can change the memory allocation for the Distributed Cache service by using the Update-SPDistributedCacheSize or Set-AFCacheHostConfiguration cmdlet. http://technet.microsoft.com/en-us/library/jj219572.aspx and http://nikpatel.net/2013/09/24/adjust-cache-size-of-the-sharepoint-2013-distributed-cache-service-part-i-overview

To have Distributed Cache working on more than one servers, the first server with Distributed Cache needs to have its firewall set to allow for Inbound ICMP (ICMPv4)

If you are using more than one cache host in your server farm and they are not in same subnet, you must configure the first cache host running the Distributed Cache service to allow Inbound ICMP (ICMPv4) traffic through the firewall. If an administrator removes the first cache host from the cluster which was configured to allow Inbound ICMP (ICMPv4) traffic through the firewall, you must configure the first server of the new cluster to allow Inbound ICMP (ICMPv4) traffic through the firewall. http://technet.microsoft.com/en-us/library/jj219572.aspx  and http://blogs.technet.com/b/speschka/archive/2013/01/11/configuring-multiple-distributed-cache-servers-in-sharepoint-2013.aspx and http://blogs.technet.com/b/uktechnet/archive/2013/05/07/guest-post-distributed-cache-service-in-sharepoint-2013.aspx

Plan to update Distributed Cache memory allocation while updating physical or virtual RAM on Cache host

When you add physical memory to the server. The Distributed Cache service does not automatically recalculate the 10% memory allocation, so when you increase the total physical memory on the server, you have to manually increase the Distributed Cache service’s memory allocation. http://technet.microsoft.com/en-us/library/jj219613.aspx

Use skipRegisterAsDistributedCachehost parameter to configure Dedicated Distributed Cache Host in SharePoint Farm

When you run PSCONFIG or SharePoint Products Configuration Wizard to configure SharePoint on server in the farm, it will start Distributed Cache service by default and server will be added as a Cache Host. As an alternative to the default configuration, an administrator can install SharePoint Server 2013 without registering the Distributed Cache service on servers not intended to be part of the cache cluster. This can be achieved by using the skipRegisterAsDistributedCachehost parameter with the New-SPConfigurationDatabase or the Connect-SPConfigurationDatabaseWindows PowerShell cmdlets, or when running psconfig.exe at the command line. This parameter is optional. – http://technet.microsoft.com/en-us/library/jj219572.aspx

Use Add-SPDistributedCacheServiceInstance to add Cache Host in the Distributed Cache Cluster

If a Distributed Cache Service is missing for a particular server in Central Administration, you need to run “Add-SPDistributedCacheServiceInstance” from a SharePoint PowerShell window on the server it is missing on. It is important to note that adding server instance using Add-SPDistributedCacheServiceInstance resets cache size to 5%. Whether you use AppFabric or SharePoint cmdlets to modify cache host size, note that if you uninstall and reinstall the Distributed Cache Service Instance on a server (i.e. by running Remove-SPDistributedCacheServiceInstance and then Add-SPDistributedCacheServiceInstance) the cache host size will be reset to the default (5% of physical memory at time of installation). If removing and adding the cache service instance is part of your maintenance cycles, make sure to also modify the cache size afterwards if needed. – http://technet.microsoft.com/en-us/library/jj219613.aspx and http://blogs.msdn.com/b/besidethepoint/archive/2013/03/27/appfabric-caching-and-sharepoint-2.aspx

Plan to avoid Farm Account to run Distributed Cache Service

When AppFabric is installed as part of the SharePoint pre-requisites, the server farm account (e.g. Niks\sp_farm) is set as the service account of the AppFabric Caching service. The Distributed Cache service depends on the AppFabric Caching service. This will eventually trigger a violation of a Health Analyzer Rule. You must plan to change the service account of the AppFabric Caching service to a managed account. Primarily it can be same service account (e.g. domain user account – Niks\sp_svcapp) used to run all other Service Applications in the farm. On the side note, if you use AutoSPInstaller, this step is automatically configured by the AutoSPInstaller and Distributed Cache will run under domain service account other than Farm Account. http://technet.microsoft.com/en-us/library/jj219613.aspx – and http://blogs.technet.com/b/uktechnet/archive/2013/05/07/guest-post-distributed-cache-service-in-sharepoint-2013.aspx

Plan to stop gracefully and remove server from Distributed Cache cluster during patching and maintenance of the SharePoint server in the farmRun Stop-SPDistributedCacheServiceInstance –Graceful and Remove-SPDistributedCacheServiceInstance before shutting down the server for patching. Run Add-SPDistributedCacheServiceInstance after patching is done. It is also important to remember that adding server instance using Add-SPDistributedCacheServiceInstance resets cache size to 5% and you must resize the cache size using Update-SPDistributedCacheSize or Set-AFCacheHostConfiguration. Do not use this approach to stop the server if you need to update cache memory size. Instead manually start/stop Distributed Cache service from Service on Server page or start/stop service instance using PowerShell command. To adjust the memory cache for host, plan to use approached previously mentioned in my articles – http://nikpatel.net/2013/09/24/adjust-cache-size-of-the-sharepoint-2013-distributed-cache-service-part-i-overview/

SharePoint doesn’t provide high availability to Distributed Cache component of the farm

SharePoint (as of August 2013 CU) does not provide any high availability for its caches. It is important to note that Cached data stored in one server may not available on another Distributed Cache server in farm. If the server where that item has been stored in memory is lost or shut down ungracefully, that cached item will be lost. This is generally not a problem for cached items because they are authoritatively stored in the database except items in feed cache – Tags and document follow activities.

The Feed Cache depends on the Distributed Cache service. Tags and document activities are saved only to the Feed Cache. Tags and document activities are not persisted to content databases. When the Distributed Cache service is stopped, tags and document activities are lost. If these cached items are lost they won’t be able to be regenerated and will no longer appear in users’ feeds. To avoid losing items from the cache and/or having to retrieve them again, you can use the Stop-SPDistributedCacheServiceInstance cmdlet with the -Graceful switch. When the graceful shutdown of the Distributed Cache service method is used, all cache data is moved from one server to another server before the Distributed Cache service is stopped. For this to be effective, there must be space on the other servers to accommodate these items. Also note that if shutting down the entire cluster, such as to change the cache host size, there’s no way to avoid losing all of the feed caches items.

When the Distributed Cache service is started, repopulation occurs when the feed cache repopulation timer job runs. It is important to note that retrieving cached items all over again involves a performance hit. There could be interruptions and delays while the caches are being refilled. http://technet.microsoft.com/en-us/library/jj219613.aspx & http://blogs.msdn.com/b/besidethepoint/archive/2013/03/27/appfabric-caching-and-sharepoint-2.aspx

Additional References

• Overview of micro-blog features, feeds, and the Distributed Cache service in SharePoint Server 2013 – http://technet.microsoft.com/en-us/library/jj219700.aspx
• Plan for feeds and the Distributed Cache service in SharePoint Server 2013 – http://technet.microsoft.com/en-us/library/jj219572.aspx
• Manage the Distributed Cache service in SharePoint Server 2013 – http://technet.microsoft.com/en-us/library/jj219613.aspx
• Josh Gavant’s App Fabric Caching and SharePoint Part I and II – http://blogs.technet.com/b/mspfe/archive/2013/04/02/appfabric-caching-and-sharepoint.aspx
• Distributed Cache Fun in SharePoint 2013 – http://blogs.technet.com/b/sharepoint_republic/archive/2013/04/11/distributed-cache-fun-in-sharepoint-2013.aspx
• Sahil Malik’s Distributed Cache in SharePoint 2013 – http://www.code-magazine.com/Article.aspx?quickid=1309021
• Distributed Cache Service in SharePoint 2013 -http://blogs.technet.com/b/uktechnet/archive/2013/05/07/guest-post-distributed-cache-service-in-sharepoint-2013.aspx
• What every SharePoint admin should know about troubleshooting and managing the newsfeed & distributed cache -http://blogs.technet.com/b/sharepoint_ramblings/archive/2013/05/11/what-every-sharepoint-admin-should-know-about-troubleshooting-and-managing-the-newsfeed-amp-distributed-cache.aspx
• Points to consider with Distributed Cache on SharePoint 2013 – http://blogs.msdn.com/b/calvarro/archive/2013/08/29/points-to-consider-with-distributed-cache-on-sharepoint-2013.aspx
• Supportability of custom developments in SPS 2013 and APPFabric Cache – http://blogs.msdn.com/b/calvarro/archive/2013/06/05/supportability-of-custom-developments-in-sps-2013-and-appfabric-cache.aspx
• How to make SharePoint 2013 Distributed Cache service Semi High Available – http://mahmoudhamed.wordpress.com/2013/09/12/how-to-make-sharepoint-2013-distributed-cache-service-semi-high-available/
• Understanding SharePoint 2013 Distributed Cache Service, Microblogs and feeds – http://dayansameera.wordpress.com/2013/08/30/understanding-sharepoint-2013-distributed-cache-service-microblogs-and-feeds/
• AppFabric Distributed Caching and SharePoint 2013  – http://mysharedpoints.blogspot.com/2012/08/appfabric-distributed-caching-and.html
• SharePoint 2013: Distributed Cache Services – http://kerls.blogspot.com/2013/03/sharepoint-2013-distributed-cache.html
• Distributed Cache Service in SharePoint 2013 – http://www.c-sharpcorner.com/UploadFile/d2ee01/distributed-cache-service-in-sharepoint-2013/
• SharePoint distributed cache issues – http://www.andrevanhaaften.com/?p=94
• How to check memory allocation for distributed cache on SharePoint 2013?  – http://sp-vinod.blogspot.com/2013/03/how-to-check-memory-allocation-for.html

Good luck managing this beast!!!!!!