Saturday, November 14, 2020

The Curious Case of Sitecore's Enforce Version Presence Permission Denied - The Fix

Standard

In a previous post, I shared a baffling enforce version presence & language fallback issue, that lead my team down a rabbit hole until we discovered that it was indeed a critical Sitecore.Kernel bug that impacted all versions, including the recent 10.0 version.

Since then, the Sitecore product team has provided a fix for this issue, which will be part of the upcoming 10.1 release.  

Until then, you can open a support ticket and reference bug #416301 and request help with your specific Sitecore version if you run into this problem.


How was it fixed?

As previously mentioned, the piece of code responsible for the permission denied error was within the GetAncestorAccess within the Sitecore.Security.AccessControl.ItemAuthorizationHelper class which is part of the Sitecore.Kernel. 

Within this method, regardless of the value that was being returned for the security checks, the key/value combo was stored in AccessResultCache and resulted in the permission denied error being thrown the next time the item was requested for a different language.

To correct this problem, a EnforceVersionPresenceDisabler "using statement wrapper" was added within the GetAccess method that is responsible for returning the "access allowed for an operation on an item".  See line 26 below. 

The switcher disables the Enforce Version Presence functionality, more specifically, it bypasses the functionality that enforces the relevant translated language version of the item to be be available for it to be returned from the API.

This was the key in corrected the access issue related to the extranet\anonymous user, and enforce version presence logic.

Saturday, November 7, 2020

Sitecore Publishing Service - Publishing Sub Items of Related Items

Standard

Background

We ran into an issue with our Sitecore 9.1 and Publishing Service 4.0 environment where when a page item with a rendering was being published, the corresponding data source of the rendering was not published fully.

To be more specific, if the rendering referred to a data source item that had multiple levels of items, then only the root of the data source was being published but not the child items.

A good example would be a navigation rendering that had a data source item with a lot of children. Content authors were making updates to all the link items within Experience Editor, but they were not being published.

This was happening for both manual publishing and publishing through workflow.


Configuration Updates

In our research, we discovered that publishing service allows you to specify the templates of the items you wish to publish as descendants of related items. 

Adding the following node to sc.publishing.relateditems.xml did trick (after a restart):

It is very important to note that the the template nodes need to have unique names in order for this to work. 

In other words: DatasourceTemplate1, DatasourceTemplate2, DatasourceTemplate3 etc. 

So as you can imagine, if you want to include a lot of data source item templates, the list in your configuration can get extremely large!

Final Words

I hope that this information helps developers who face a similar issue, as I could not find anything online about this related publishing configuration.

As always, feel free to comment or reach me on Slack or Twitter if you have any questions.                                                    

Saturday, October 31, 2020

Control Tracking JavaScript Using Sitecore Rule-based Configuration

Standard

Background

Being able to control tracking JavaScript via Environment and Server role is a common problem that Sitecore developers are faced with. For example, your client doesn't want their Production Google Tag Manager or agency delivered tracking pixel scripts firing on any server / app instance other than their Production Content Delivery as it will spoil analytics.

Most of the time, developers will add some type of "if statement" code in Layouts or Renderings to help facilitate this, but this could be difficult to control and maintain based on the number of scripts you end up adding to you site(s). 

In addition, if you are using SXA and HTML snippets in your metadata partial design to house the scripts, this becomes even harder.

Post-Processing HTML

I wanted to focus on finding the sweet spot in Sitecore where I could inspect the entire HTML output after it had been glued together by the various pipelines, and then remove the target script from the HTML before it was transmitted to the browser.

Sitecore's httpRequestProcessed pipeline gives is the entry point, where we can leverage the MVC framework's result filter to manipulate the HTML.

I told my content authors to add a new attribute called "data-xp-off" to their scripts that I would use as the flag to determine if the script would be removed from the page.

For example:
<script data-xp-off>some tracking stuff</script>



Writing the Code

The first step was to create a new HttpRequest processer and associated configuration to inject into the httpRequestProcessed pipeline. Within this, I was able to access the HttpContext response filter object where I could perform the targeted script removal.

As you can see by the config, you can use whatever rule-based role config to apply the processor.

Next, I created a class based on the System.IO Stream class, where I overrode the Flush method. Within this new Flush method, I removed the script using a regular expression (based on the existence of the data-xp-off attribute within the html), and then wrote it to the response.

You will notice that I also included the "noscript" and "style" tags as an option for the filtering which was a bonus.

So you may ask me; "Martin, why did you not use the powers of the Html Agility Pack to perform your HTML manipulation?".  To be honest, that was my first approach. I wrote this code:

I discovered that the InnerHtml returned by the Agility Pack was making unintentional changes to my HTML markup, and that caused problems with client-side heavy components.  Digging into Sitecore's code, I discovered that they used the regular expression approach when injecting their Experience Editor "chrome" elements, and so I went down that path too.


Saturday, October 17, 2020

The Curious Case of Sitecore's Enforce Version Presence Permission Denied

Standard

Background

I ran into a perplexing enforce version presence & language fallback issue that I wanted to share so that others won't have to go down the same rabbit hole as I did to uncover the underlying issue.

Language, Fallback and Version Presence Config

Our site is configured for almost 3 dozen languages, and as you can image, we rely on language fallback a lot!  

Language fallback works by displaying a fallback version of a language item when there is no version available in the current language context. 

So if you got to https://www.mysite.com/de-de/mypage and there isn't a German version available for that page, it will instead render the English version (if one exists) at the url.

Now, let's say that you don't have an English version for that particular page either. So, there is only a Dutch version of the page available at https://www.mysite.com/nl-nl/mypage. Then you go to the German url https://www.mysite.com/de-de/mypage. Since there isn't any fallback English version available, you would expect to see a 404 response right?

To set this up, the item would need to have both "Enable Item Fallback", and "Enforce Version Presence" enabled at its item level. The recommended way to do this is by setting these values on the template Standard Values.


Finally, you would need to set fallback and version presence on your site configuration. If you are using SXA, you would enable so this under the /sitecore/content/TenantFolder/TenantName/SiteName/Settings/Site Grouping/SiteName item.

In "Other Properties", you set "enforceVersionPresence" set to true:

Problematic Results

Initially, this seemed to work as expected. 

Referencing the example again: We only had a single language version of the item, and nothing else (not even the English fallback). The Dutch version of the page was available at https://www.mysite.com/nl-nl/mypage and then accessing any other language url https://www.mysite.com/de-de/mypage or https://www.mysite.com/en/mypage we would see the expected 404.

Over time we discovered "Permission to the requested document was denied" issues started bubbling up all over the place for our items that were supposed to fallback. Initially, this seemed to be random: different users would access the same url, and some would experience the permission denied message, while the fallback and content would be resolved without issue for others.

Investigation

This was very tricky reproduce in a scaled environment.  In our investigation, we discovered the following scenario:

1. Create any page in a non-english version only.
2. Save and publish the page.
3. Access the page url in any other language version that does not exist for that item. 
    3.1 404 for that item is returned.
4. On the same server, try and access the page url in a different language version that does not exist for that item. 
    4.1. "Permission to the requested document was denied" is returned.
5. Clearing the AccessResult cache and first accessing the item in number 4 above that didn't previously work, would make it work. Trying to access the item in number 3 above that worked previously would then throw the permission denied error.

Perplexing right?

Let me provide a more specific example:

  • A Dutch version of the page was available at https://www.mysite.com/nl-nl/mypage 
  • Accessed the page via the German url https://www.mysite.com/de-de/mypage
    • The expected 404 for the item is returned.
  • On the same server, access the English page url via  https://www.mysite.com/en/mypage
    • "Permission to the requested document was denied" is returned.

Root Cause

The support team confirmed this critical bug in the Sitecore.Kernel that is present in all recent Sitecore versions.

The piece of code responsible for the permission denied error as:


Regardless of the value that is returned for the checks, the key/value combo is stored in AccessResultCache.

Next time the item is requested it is served from this cache, the permission denied error is thrown.

Fix Forthcoming

Due to the critically of this bug, the product team is working on the fix. Once ready, and it has gone through our own QA cycles, I intend to post an update with more information about it.

Wednesday, September 30, 2020

Understanding Sitecore's Self-Adjusting Thread Pool Size Monitor

Standard

Background

In a previous post, I focused on the inner workings of the .NET CLR and Thread Pool and how they can impact the stability of the Sitecore application.

I must admit, I have become mildly obsessed with the threading over the last couple years, mostly because a great deal of my work has involved stabilization and optimization practices on high-traffic Sitecore sites. 

In this post, I want to focus in the Thread Pool Size Monitor that comes baked into Sitecore from 9 onwards, because it is not widely known that it exists, the job it does, and how it can be tuned to optimize performance.

Thread Pool Size Monitor

To recap, the most important thread configuration settings are the minWorkerThreads and minIOThreads where you can specify the minimum number of threads that are available to your application's Thread Pool instead of relying on the default formula's based on processor count which is always too few.

Threads that are controlled by these values can be created at a much faster rate (because they are spawned from the Thread Pool), than worker threads that are created from the CLR's default "thread-tuning" capabilities. 

To summarize: 

  • Thread pool threads get thrown in faster to handle work. 
  • The CLR thread spin up algorithm is too slow and we can't rely on it to support high performance applications.

As previously mentioned, in Sitecore 9 and above, there is a pipeline processor that allows the application to adjust thread limits dynamically based on real-time thread availability (using the Thread Pool API).

This can be found in the following namespace: Sitecore.Analytics.ThreadPoolSizeMonitor.

By default, every 500 milliseconds, the processor will keep adding a value of 50 to the minWorkerThreads setting via the Thread Pool API until it determines that the minimum number of threads is adequate based on available threads.

How It Works

Since a picture is worth a thousand words, I put together a diagram of how the logic of the Thread Pool Size Monitor logic works, and provided an example with the default settings that are set on an Azure P3v2 App Service that has 4 cores.  




Custom Thread Pool Size Monitor Configuration

An enhancement that I have made on my past 9.1 PaaS implementation was to tune Sitecore’s dynamic thread processor using a more “aggressive” configuration. This helped me with those “bursty” web traffic situations where I needed to be sure that I had enough threads available to serve the current demands. 

Here is the configuration that I used:

Monday, September 7, 2020

Sitecore Azure PaaS - Fixing "No owin.Environment item was found in the context" When Application Initialization Configured

Standard

Background

On Azure PaaS, it is well known that Application Initialization can assist in “warming up” your Sitecore application when scaling up or swapping slots, so that your App Services instances aren't thrown into rotation to receive traffic when they aren't ready. 

Unfortunately, since Sitecore 9.1, there is a nasty Owin error that prevents this from working correctly and this in turn impacts the healthy signaling of your App Services. 

Strangely, the error would disappear after about 2 minutes.


Sitecore Support Response

After talking with Sitecore Support engineers, they ended up telling us that using <applicationInitialization> had not been tested for Sitecore, and that they don't have any official guidelines on configuring Azure App Initialization.

Finally, they said that they were not able to provide us with any assistance.

Cause

I started deciphering the problem on my own. 

Looking at this error, the key to understand its root was on line 6: the BeginDiagnostics processor.

1:  [InvalidOperationException: No owin.Environment item was found in the context.]  
2:    System.Web.HttpContextExtensions.GetOwinContext(HttpContext context) +100  
3:    Sitecore.Owin.Authentication.Publishing.PreviewManager.get_User() +15  
4:    Sitecore.Owin.Authentication.Publishing.PreviewManager.GetShellUser() +23  
5:    Sitecore.Pipelines.HttpRequest.BeginDiagnostics.ResolveUser() +41  
6:    Sitecore.Pipelines.HttpRequest.BeginDiagnostics.Process(HttpRequestArgs args) +43  
7:    (Object , Object ) +15  

The BeginDiagnostics processor in the httpRequestBegin pipeline is used for Sitecore's diagnostic tools. This determines what happens if you try to run in Debug Mode in the page editor.

In this case, the PreviewManager User property was trying to return the HttpContext.Current.User which was not available yet, and was returning null and the error “No owin. Environment item was found in the context.” was thrown.

The Fix

After determining what was causing this error, the fix was simple. The processor isn't needed on Content Delivery instances as Debug Mode is only used on the Content Management instances.

Removing the processor via a simple patch fixed the error:

Monday, August 31, 2020

Sitecore xDB - Extending xConnect To Reduce xDB Index Sizes And Rebuild Times

Standard

Background

In my previous post, I write about how you can extend the xConnect Data Adapter Provider to prevent unwanted data from being written to your shard databases.

But, what if you already have a lot of analytics and contact data in your shards, have an xDB index issue, and are struggling to rebuild your index?  

Or, what if you want to keep the data in your databases just in case you may need it some time in the future but want to lighten the load on your xDB index to improve performance?

In this post, I will show you how to filter data from your xDB index to reduce the index size and growth rate. This will also improve your rebuild times tremendously.

You can also skip ahead and check out the full source code for my example on my GitHub repo: https://github.com/martinrayenglish/Test.SCExtensions


Business Scenario

In a 9.1 project, I was faced with a very slow rebuild times, where it was literally taking weeks to rebuild our xDB index. I was very concerned with the amount of data we were ingesting into the xDB index from the shard databases, specifically the rate at which interactions where being added. I have an older post this talks about this topic, so give it a read if you have not already.

Prior to 9.2, there is no true supported way to purge old interaction data, and so I was faced with the question of how I could reduce the amount of interaction data that our xDB index contained, and preventing our business from being paralyzed for weeks waiting for a rebuild to complete in the future.

Solving The Problem

As sheer quantity of interactions was the problem, I wanted to be able to filter both new and existing interactions from being brought in from the shard databases during a rebuild or after the session end data aggregation process.

Round 1

In my initial exploration, I focused on customizing the index writer code as that seemed to be the sweet spot.  The code below shows an example of where I customized it to remove ALL interactions from making their way to the xDB index (see line 28 where I simply new up a new blank list of InteractionDataRecord objects):

To apply the updated writer, I put the new assembly into
{xconnect}\bin
and
{xconnect}\App_data\jobs\continuous\IndexWorker

 and then updated the XML files to use new writer in {xconnect}\App_data\jobs\continuous\IndexWorker\App_data\Config\Sitecore\SearchIndexer\sc.Xdb.Collection.IndexWriter.Solr.xml
and
{xconnect}\App_data\config\sitecore\SearchIndexer\sc.Xdb.Collection.IndexWriter.Solr.xml

 <IIndexWriter>  
         <Type>Test.SCExtensions.Xdb.Collection.Search.Solr, Test.SCExtensions</Type>  
         <As>Sitecore.Xdb.Collection.Indexing.IIndexWriter, Sitecore.Xdb.Collection</As>  
         <LifeTime>Singleton</LifeTime>  
  </IIndexWriter>  

This worked well for new interactions coming in from the shard databases after session end, but did not work for rebuild operations so I needed to explore further.

Round 2

The problem that I discovered after my initial try, was that the SolrWriter type specified above is not used by the rebuilder because of the dependency injection implementation details. so writing the custom implementation of the rebuilder class was not a valid option.

I started looking into other options to override the related logic.

After investigating further, I discovered that a custom implementation of both the IndexRebuilderFilter and IndexWriterFilter would take care of both new interactions from sessions and existing interactions coming in via a rebuild operation.  

I created 2 new classes to perform these tasks that I used to decorate the related interface. 

Again, in these examples, I am removing ALL interactions from my xDB index. In a real world scenario, it would be best to filter based on analytics data that is valuable to your business so that it will be available via the xDB search API. 

I show you how to do this further down in my post: More Bang - Selective xDB Index Filtering.

Next, I created decorator XML configs to allow xConnect to use my new writer extension code.

Rebuilder writer config location: {xconnect}\App_Data\jobs\continuous\IndexWorker\App_data\config\sitecore\SearchIndexer\sc.Xdb.Collection.Indexing.IndexRebuilderFilterDecorator.xml

 <?xml version="1.0" encoding="utf-8"?>  
 <Settings>  
   <Sitecore>  
     <XConnect>  
       <SearchIndexer>  
         <Services>  
           <IIndexRebuilderFilterDecorator>  
             <Type>Sitecore.XConnect.Configuration.ServiceDecorator`2[[Sitecore.Xdb.Collection.Indexing.IIndexRebuilder, Sitecore.Xdb.Collection], [Test.SCExtensions.Xdb.Collection.Search.Solr.IndexRebuilderFilterDecorator, Test.SCExtensions]], Sitecore.XConnect.Configuration</Type>  
             <LifeTime>Singleton</LifeTime>  
           </IIndexRebuilderFilterDecorator>  
         </Services>  
       </SearchIndexer>  
     </XConnect>  
   </Sitecore>  
 </Settings>  
Writer config location: {xconnect}\App_Data\jobs\continuous\IndexWorker\App_data\config\sitecore\SearchIndexer\sc.Xdb.Collection.Indexing.IndexWriterFilterDecorator.xml
 <?xml version="1.0" encoding="utf-8"?>  
 <Settings>  
   <Sitecore>  
     <XConnect>  
       <SearchIndexer>  
         <Services>  
           <IIndexWriterFilterDecorator>  
             <Type>Sitecore.XConnect.Configuration.ServiceDecorator`2[[Sitecore.Xdb.Collection.Indexing.IIndexWriter, Sitecore.Xdb.Collection], [Test.SCExtensions.Xdb.Collection.Search.Solr.IndexWriterFilterDecorator, Test.SCExtensions]], Sitecore.XConnect.Configuration</Type>  
             <LifeTime>Singleton</LifeTime>  
           </IIndexWriterFilterDecorator>  
         </Services>  
       </SearchIndexer>  
     </XConnect>  
   </Sitecore>  
 </Settings>  

After applying these updates, I was able to successful filter interactions coming from the shard databases to my xDB index during both rebuilds and session ends!

More Bang - Selective xDB Index Filtering

As you don't necessarily want to remove all interactions from the xDB index unless you are in dire straights, I added a new class for filtering interactions based on specific events. In other words, if an interaction has an event in it that we care about, we allow it to go into the index.


By using this example, you can lighten your index, and still include interactions that are important to you for personalization etc. 

This full source code for this example can be found on my GitHub repo: https://github.com/martinrayenglish/Test.SCExtensions