Wednesday, March 20, 2024

Sitecore Publishing Service - Publishing Job Queue Report Using Sitecore PowerShell Extensions

Standard

Background

Sitecore's Publishing Service only allows you to see a maximum of 10 items at a time within the Queued or Recent jobs reports within the dashboard.

This is not ideal if you need to see how many total items are in the queue, need to get an estimate of how long it will take to get your publish live or quite simply need to do any type of analysis or troubleshooting.

Usually, you will have to talk to a DevOps person who has access to your Sitecore Master database, and get them to write a somewhat complex SQL query against your Publishing_JobQueue table to get you the information that you need.

It is a bit complex due to the fact that most of the key information is stored in an XML field called "Options" within this particular table.


PowerShell For The Win

After spending a bit of time formulating a decent SQL query that would get the key information that we were after, I decided to take it one step further by incorporating it into a PowerShell script that could be generated on demand from within the Sitecore console, and also output a searchable and downloadable report.

A clear win for our Authoring Admins and DevOps teams!

I hope you find this script a useful add to your PowerShell toolbox.



Wednesday, March 13, 2024

Sitecore Content Hub - Set up SAML-based SSO in Azure AD using an App Registration

Standard

Background

In this post, I will show you how to create and configure an Azure Application Registration in your tenant to allow Sitecore Content Hub users to successfully authenticate against your Azure Active Directory.

Options

The Content Hub team's preferred set up option is to create an Enterprise application within your Azure AD, but unfortunately for us, our DevOps would not allow this due to very strict security constraints that we had to abide by. This is the main reason that we had to go the App Registration route.

We initially tried to get the App Registration working using Microsoft Provider SSO, but could not get the proper Group claims working correctly.

As a result, we focused on configuring SAML Auth within our App Registration, and were able to get all the claims needed to successfully get SSO authentication working with this approach.

Set up within Azure

Within your Azure Portal, find App registrations and click on the New Registration button. Give it a name, and leave the default options selected, and click Register.


Within the newly created registration, go to the Authentication menu option within the Manage section.

Click "Add a platform", and then select "Web".


Set your Redirect URIs to be the Content Hub portal url. You will be able to add additional URIs after the initial set up. For now, I will use a default one.

Make sure you check the Access tokens and ID tokens boxes within the Implicit grant and hybrid flows section.




After this, click the Configure button.

Next, go to the Token configuration menu option within the Manage section.

Click Add group claim, and check the Security group box. Confirm that the Group ID radio option is selected within the ID, Access and SAML options.

Click the Add button.


Next, click Add optional claim.

Within Token type, select SAML, and check the email Claim box. Click Add.


When prompted, check the "Turn on the Microsoft Graph email permission" box to allow the claims to appear in the token. Click Add.


Next, go to the Expose an API menu option within the Manage section. Click Add a scope, and it will generate an Application ID URI for you. 

Make note of this, as you will need it for the Content Hub side.

Click Save and continue.


After is has been created, you can click the Cancel button.




Go to the Overview menu option, and click Endpoints. Go to the Federation metadata document XML url and make note of it

Then, copy and past it into a new browser tab.





Make note of the entityID.

Your set of notes should look similar to this:


Set up within Content Hub

Log into your Content Hub portal. Click on Manage, and then go to Settings.



Within Settings, go to PortalConfiguration, and select the Authentication menu option. Change the view to Text as it's easier to work with.

Within the ExternalAuthenticationProviders, saml XML config, set the key values to what you saved in your notes. Make sure you set the provider_name and add some basic messages.



Example:
 ExternalAuthenticationProviders": {  
   "global_username_claim_type": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name",  
   "global_email_claim_type": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress",  
   "google": [],  
   "Microsoft": [],  
   "saml": [  
    {  
     "metadata_location": "https://login.microsoftonline.com/8ac76c91-e7f1-41ff-a89c-3553b2da2c17/federationmetadata/2007-06/federationmetadata.xml",  
     "sp_entity_id": "api://c8696890-1d5f-479b-9df1-154e8f315165",  
     "idp_entity_id": "https://sts.windows.net/8ac76c91-e7f1-41ff-a89c-3553b2da2c17/",  
     "password": null,  
     "certificate": null,  
     "binding": "HttpRedirect",  
     "authn_request_protocol_binding": null,  
     "is_enabled": true,  
     "provider_name": "martinSamlNewLocal",  
     "messages": {  
      "signIn": "Martin SAML SSO Test"  
     },  
     "authentication_mode": "Passive"  
    }  
   ],  
   "sitecore": [],  
   "ws_federation": [],  
   "yandex": []  
  }  

Click Save, and you are done!

You are now ready to test out your authentication using your shiny, new authentication button.

Users with more than 200 groups

We found a limitation with SSO authentication group claims in Azure AD https://docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-fed-group-claims wherein if there are more than 200 groups associated to a user, then the SSO authentication will provide a graph link instead of passing in the group claims. 

There is currently no solution for this problem. We are handling these handful of users via manual security set up.

Tuesday, November 14, 2023

The Curious Case of Sitecore's Enforce Version Presence Permission Denied - The Fix

Standard

In a previous post, I shared a baffling enforce version presence & language fallback issue, that lead my team down a rabbit hole until we discovered that it was indeed a critical Sitecore.Kernel bug that impacted all versions, including the recent 10.0 version.

Since then, the Sitecore product team has provided a fix for this issue, which will be part of the upcoming 10.1 release.  

Until then, you can open a support ticket and reference bug #416301 and request help with your specific Sitecore version if you run into this problem.


How was it fixed?

As previously mentioned, the piece of code responsible for the permission denied error was within the GetAncestorAccess within the Sitecore.Security.AccessControl.ItemAuthorizationHelper class which is part of the Sitecore.Kernel. 

Within this method, regardless of the value that was being returned for the security checks, the key/value combo was stored in AccessResultCache and resulted in the permission denied error being thrown the next time the item was requested for a different language.

To correct this problem, a EnforceVersionPresenceDisabler "using statement wrapper" was added within the GetAccess method that is responsible for returning the "access allowed for an operation on an item".  See line 26 below. 

The switcher disables the Enforce Version Presence functionality, more specifically, it bypasses the functionality that enforces the relevant translated language version of the item to be be available for it to be returned from the API.

This was the key in corrected the access issue related to the extranet\anonymous user, and enforce version presence logic.

Monday, October 30, 2023

Control Tracking JavaScript Using Sitecore Rule-based Configuration

Standard

Background

Being able to control tracking JavaScript via Environment and Server role is a common problem that Sitecore developers are faced with. For example, your client doesn't want their Production Google Tag Manager or agency delivered tracking pixel scripts firing on any server / app instance other than their Production Content Delivery as it will spoil analytics.

Most of the time, developers will add some type of "if statement" code in Layouts or Renderings to help facilitate this, but this could be difficult to control and maintain based on the number of scripts you end up adding to you site(s). 

In addition, if you are using SXA and HTML snippets in your metadata partial design to house the scripts, this becomes even harder.

Post-Processing HTML

I wanted to focus on finding the sweet spot in Sitecore where I could inspect the entire HTML output after it had been glued together by the various pipelines, and then remove the target script from the HTML before it was transmitted to the browser.

Sitecore's httpRequestProcessed pipeline gives is the entry point, where we can leverage the MVC framework's result filter to manipulate the HTML.

I told my content authors to add a new attribute called "data-xp-off" to their scripts that I would use as the flag to determine if the script would be removed from the page.

For example:
<script data-xp-off>some tracking stuff</script>



Writing the Code

The first step was to create a new HttpRequest processer and associated configuration to inject into the httpRequestProcessed pipeline. Within this, I was able to access the HttpContext response filter object where I could perform the targeted script removal.

As you can see by the config, you can use whatever rule-based role config to apply the processor.

Next, I created a class based on the System.IO Stream class, where I overrode the Flush method. Within this new Flush method, I removed the script using a regular expression (based on the existence of the data-xp-off attribute within the html), and then wrote it to the response.

You will notice that I also included the "noscript" and "style" tags as an option for the filtering which was a bonus.

So you may ask me; "Martin, why did you not use the powers of the Html Agility Pack to perform your HTML manipulation?".  To be honest, that was my first approach. I wrote this code:

I discovered that the InnerHtml returned by the Agility Pack was making unintentional changes to my HTML markup, and that caused problems with client-side heavy components.  Digging into Sitecore's code, I discovered that they used the regular expression approach when injecting their Experience Editor "chrome" elements, and so I went down that path too.


Thursday, September 21, 2023

Sitecore xDB - Troubleshooting xDB Index Rebuilds on Azure

Standard
In my previous post, I shared some important tips to help ensure that if you are faced with an xDB index rebuild, you can get it done successfully and as quickly as possible.

I mentioned a lot of things in the post, but now, I want to mention common reasons where and why things can go wrong, and highlight the most critical items that impact the rebuild speed and stability.


Causes of Need To Rebuild xDB Index

Your xDB relies on your shard database's SQL Server change tracking feature in order to ensure that it stays in sync. This basically determines how long changes are stored in SQL. As mentioned in Sitecore's docs, the Retention Period setting is set to 5 days for each collection shard. 

So, why would 5-day old data not be indexed in time?
  • The Search Indexer is shut down for too long
  • Live indexing is stuck for too long
  • Live indexing falls too far behind

Causes of Indexing Being Stuck or Falling Behind, and Rebuild Failures

High Resource Utilization: Collection Shards 
99% of the time, this is due to high resource utilization on your shard databases. Basically, if you see your shard databases hitting above 80% DTUs, you will run into this problem.

High Resource Utilization: Azure Search or Solr
If you have a lot of data, you need to scale your Azure Search Service or Solr instance.  Sharding is the answer, and I will touch in this further down.

What to check?

If you are on Azure, make sure your xConnect Search Indexer WebJob is running.
Most importantly, check your xConnect Search Indexer logs for SQL timeouts. 

On Azure, the Webjob logs are found in this location: D:\local\Temp\jobs\continuous\IndexWorker\{randomjobname}\App_data\Logs"

Key Ingredients For Rebuild Indexing Speed and Stability

SQL Collection Shards

Database Health 

Maintaining the database indexes and statistics is critically important. As I mentioned in my previous post:  "Optimize, optimize, optimize your shard databases!!!" 

If you are preparing for a rebuild, make sure that you run the AzureSQLMaintenance Stored Procedure on all of your shard databases.

Database Size

The amount of data and the number of collection shards is directly related to resource utilization and rebuild speed and stability. 

Unfortunately, there is no supported way to "reshard" your databases after the fact. We are hoping this will be a feature that is added to a future Sitecore release.

xDB Search Index

Similarly to the collection shards, the amount of data and the number of shards is directly related to resource utilization on both Azure Search and Solr. 

Specifically on Solr, you will see high JVM heap utilization.

If your rebuilds are slowing down or failing, or even if search performance on your xDB index is deteriorating, it's most likely due to the amount of data in your index, the number of shards and distribution amongst nodes that you have set up.  

Search index sharding strategies can be pretty complex, and I might touch on in these in a later post.

Reduce Your Indexer Batch Size

Another item that I mentioned in my previous post. If you drop this down from 1000 to 500 and you are still having trouble, reduce it even further. 

I have dropped the batch size to 250 on large databases to reduce the chance of timeouts (default is 30 seconds) when the indexer is reading contacts and interactions from the collection shards.


Monday, January 3, 2022

Fix Email Campaign Pausing: Sitecore Email Experience Manager 3.x Retry Data Provider

Standard

Background

My company uses Email Experience Manager (EXM) to send several million emails a day, and we have been facing issues where our large campaigns would pause mid-send.

We have a scaled EXM environment with 2 dedicated dispatch servers, and a separate SQL Server, all with appropriate resources so the hardware was not an issue. We also ensured that databases were kept in tiptop condition (proper maintenance plans with stats being updated), and configurations where optimal for our environment.


The causing of the pausing

After digging in, I discovered that the pausing was caused by SQL deadlocks due to the massive amount of records and CRUD activity on the EXM SQL databases.

Sample Exception:

 ERROR Transaction (Process ID 116) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction.  
 Exception: System.Data.SqlClient.SqlException  
 Message: Transaction (Process ID 116) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction.  
 Source: .Net SqlClient Data Provider  
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)  
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)  
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)  
   at System.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean& moreRows)  
   at System.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more)  
   at System.Data.SqlClient.SqlDataReader.Read()  
   at System.Data.SqlClient.SqlCommand.CompleteExecuteScalar(SqlDataReader ds, Boolean returnSqlValue)  
   at System.Data.SqlClient.SqlCommand.ExecuteScalar()  
   at Sitecore.Modules.EmailCampaign.Core.Data.SqlDbEcmDataProvider.CountRecipientsInDispatchQueue(Guid messageId, RecipientQueue[] queueStates)  
   at Sitecore.Modules.EmailCampaign.Core.Gateways.DefaultEcmDataGateway.CountRecipientsInDispatchQueue(Guid messageId, RecipientQueue[] queueStates)  
   at Sitecore.Modules.EmailCampaign.Core.Analytics.MessageStatistics.get_Unprocessed()  
   at Sitecore.Modules.EmailCampaign.Core.Analytics.MessageStatistics.get_Processed()  
   at Sitecore.Modules.EmailCampaign.Core.MessageStateInfo.InitializeSendingState()  
   at Sitecore.Modules.EmailCampaign.Core.MessageStateInfo.InitializeMessageStateInfo()  
   at Sitecore.Modules.EmailCampaign.Factory.GetMessageStateInfo(String messageItemId, String contextLanguage)  
   at Sitecore.EmailCampaign.Server.Services.MessageInfoService.Get(String messageId, String contextLanguage)  
   at Sitecore.EmailCampaign.Server.Controllers.MessageInfo.MessageInfoController.MessageInfo(MessageInfoContext data)  
   

How does this new data provider fix the problem?

The new data provider introduces efficient SQL deadlock handling. When a deadlock is detected, it will wait 5 seconds and then retry the transaction. The code will try to execute a deadlocked transaction 3 times.

Configuration

Defaults are set to wait 5 seconds for the retry, and the max retry attempts is 3. The DelaySeconds and RetryCount settings can be modified to suit your needs.

 <configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">  
  <sitecore>  
   <ecmDataProvider defaultProvider="sqlretry">  
    <providers>  
     <clear/>  
     <add name="sqlretry" type="Sitecore.EmailCampaign.RetryDataProvider.RetrySqlDbEcmDataProvider, Sitecore.EmailCampaign.RetryDataProvider" connectionStringName="exm.master">  
      <Logger type="Sitecore.ExM.Framework.Diagnostics.Logger, Sitecore.ExM.Framework" factoryMethod="get_Instance"/>  
      <DelaySeconds>5</DelaySeconds>  
      <RetryCount>3</RetryCount>  
     </add>  
     <add name="sqlbase" type="Sitecore.Modules.EmailCampaign.Core.Data.SqlDbEcmDataProvider, Sitecore.EmailCampaign" connectionStringName="exm.master">  
      <Logger type="Sitecore.ExM.Framework.Diagnostics.Logger, Sitecore.ExM.Framework" factoryMethod="get_Instance"/>  
     </add>  
    </providers>  
   </ecmDataProvider>  
  </sitecore>  
 </configuration>  

Source Code and Documentation

Full source code, documentation and package download is available from my GitHub repository:

https://github.com/martinrayenglish/Sitecore-EXM-3.x-Retry-Data-Provider

Sunday, November 7, 2021

Sitecore Publishing Service - Publishing Sub Items of Related Items

Standard

Background

We ran into an issue with our Sitecore 9.1 and Publishing Service 4.0 environment where when a page item with a rendering was being published, the corresponding data source of the rendering was not published fully.

To be more specific, if the rendering referred to a data source item that had multiple levels of items, then only the root of the data source was being published but not the child items.

A good example would be a navigation rendering that had a data source item with a lot of children. Content authors were making updates to all the link items within Experience Editor, but they were not being published.

This was happening for both manual publishing and publishing through workflow.


Configuration Updates

In our research, we discovered that publishing service allows you to specify the templates of the items you wish to publish as descendants of related items. 

Adding the following node to sc.publishing.relateditems.xml did trick (after a restart):

It is very important to note that the the template nodes need to have unique names in order for this to work. 

In other words: DatasourceTemplate1, DatasourceTemplate2, DatasourceTemplate3 etc. 

So as you can imagine, if you want to include a lot of data source item templates, the list in your configuration can get extremely large!

Final Words

I hope that this information helps developers who face a similar issue, as I could not find anything online about this related publishing configuration.

As always, feel free to comment or reach me on Slack or Twitter if you have any questions.