Sitecore xDB - Troubleshooting Contacts Moving Slowly Through Marketing Automation Plans

Background

We ran into an issue on our Sitecore 9.1 Azure PaaS deployment, where contacts were moving extremely slowly through the elements of several of our Marketing Automation plans. The contacts were being enrolled correctly, but when they got to a certain step, the could be stuck for several days.

Our instance includes several high traffic sites, but our plans were not complicated at all.  For example, one was set up like this:

• Visitor updates their profile, and after clicking submit, a goal is triggered that enrolls the contact into the plan.
• There is a 1 minute delay.
• An email is sent.

In this post, I will describe the approach I took to getting to the bottom of this problem.

Getting More Information From Marketing Automation Logs

This first step in any troubleshooting process is to get as much detail from your logs as possible so that you can better understand what could be going wrong. 

One of the key components of Sitecore Marketing Automation, is the task that is responsible for the processing of contacts in the Automation Plans. In Azure, this is the Marketing Automation Operations App Service (ma-ops) WebJob, or the Sitecore Marketing Automation Engine service on a Windows Server. What you need to do is set the logging level to “Information” in the sc.Serilog.xml file. 

The full path of the file location is the following (Azure):

wwwroot/App_Data/jobs/continuous/AutomationEngine/App_Data/Config/sitecore/CoreServices/sc.Serilog.xml

For more information, check the following article: https://kb.sitecore.net/articles/017744

After getting this in place, I was able to rule out the possibility of exceptions being thrown that could have been causing the problem.

Finding the clogged Automation Pool

After digging in, I discovered that the number of contacts in the UI corresponded to the data in the Marketing Automation (ma-db) database’s AutomationPool table.  

These are the other things I discovered:

• Running a “row count” query against the AutomationPool table determined that there were 100s of millions of records.
• Checking Azure resource analytics showed large database utilization spikes (100% DTU) that corresponded to high CPU on my ma-ops service (averaging above 70%).
• Running a profile on the maengine.exe job revealed that getting and processing the data from the SQL server seemed to be the bottleneck.
  
  

Fixing the clogged Automation Pool

Database Health

One key assumption here was that my ma-db was healthy, and that its indexes were not seriously fragmented.

In previous posts, I have spoken about how important it is to make sure that regular maintenance is performed on your databases. Same thing applies here. Make sure that you run the AzureSQLMaintenance Stored Procedure on your ma-db regularly.

Increase Pricing Tier If Possible

The analysis determined that both my ma-ops app service and ma-db database were struggling to keep up with the processing load, and so I increased the pricing tier on both to give them more horsepower.

Increase Low Priority Worker Batch Size

The AutomationPool table showed me that almost all of the items had a priority of 80, and would be processed by the LowPriorityWorker. I opted to increase the batch size of this worker, so that it would process more items in each batch.  My worker was initially set to 200, and so I tippled it to 600.

The config file location can be found here: App_Data/jobs/continuous/AutomationEngine/App_Data/Config/sitecore/MarketingAutomation/sc.MarketingAutomation.Workers.xml

<LowPriorityWorkerOptions>
<!-- How long the worker sleeps when there is no work available -->
<Schedule>00:00:20</Schedule>
<!-- The minimum priority of work item to process. -->
<MinimumPriority>0</MinimumPriority>
<!-- The timeout period to use for work items. -->
<WorkItemTimeout>00:00:45</WorkItemTimeout>
<!-- The period after which the work item timeout is set again. -->
<WorkItemTimeoutSchedule>00:00:30</WorkItemTimeoutSchedule>
<!-- The batch size multiplier to use when checking out work items from the pool. -->
<BatchHead>4</BatchHead>
<!-- The batch size to use when checking items out from the pool. -->
<BatchSize>150</BatchSize>
</LowPriorityWorkerOptions>

Clog Removed

After making these adjustments, I kept a close eye on the number of rows in the AutomationPool table.

I am happy to report that they were decreasing at a pleasing rate, and that the data started to appear in the reports in the UI.

Read More

Sitecore GeoIP - A Developer's Guide To What Has Changed In Sitecore 9

In my previous post, I took a dive into the 8.x version of the Sitecore GeoIP service from a developer's point of view. Sitecore 9 introduced great improvements to xDB, GeoIP being one of those features.

In this post, I intend to help developers understand what has changed in Sitecore 9 GeoIP.  Like my previous post, the purpose is to arm developers with the necessary details to understand what is happening under the hood, so that they can successfully troubleshoot a problem if one arises.

Reference Data

One of the first things that I discovered when diving into version 9 is the use of a series of "ReferenceDataClientDictionaries" that are exposed to us as "KnownDataDictionaries".

As inferred by the name, these are known collections of things that are used to store common data, one being IP Geolocation data. The data is ultimately stored in a SQL database, so that it can be referenced throughout the Experience Platform.

There is a new pipeline in Sitecore 9 that initializes these dictionaries, as shown here:

Config: 

<initializeKnownDataDictionaries patch:source="Sitecore.Analytics.Tracking.config">
<processor type="Sitecore.Analytics.DataAccess.Pipelines.InitializeKnownDataDictionaries.InitializeKnownDataDictionariesProcessor, Sitecore.Analytics.DataAccess"/>
<processor type="Sitecore.Analytics.XConnect.DataAccess.Pipelines.InitializeKnownDataDictionaries.InitializeDeviceDataDictionaryProcessor, Sitecore.Analytics.XConnect" patch:source="Sitecore.Analytics.Tracking.Database.config"/>
</initializeKnownDataDictionaries>

Processor Code:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

namespace Sitecore.Analytics.DataAccess.Pipelines.InitializeKnownDataDictionaries { public class InitializeKnownDataDictionariesProcessor : InitializeKnownDataDictionariesProcessorBase { public override void Process(InitializeKnownDataDictionariesArgs args) { Condition.Requires<InitializeKnownDataDictionariesArgs>(args, nameof (args)).IsNotNull<InitializeKnownDataDictionariesArgs>(); GetDictionaryDataPipelineArgs args1 = new GetDictionaryDataPipelineArgs(); GetDictionaryDataPipeline.Run(args1); Condition.Ensures<DictionaryBase>(args1.Result).IsNotNull<DictionaryBase>("Check configuration, 'getDictionaryDataStorage' pipeline must set args.Result property with instance of DictionaryBase type."); args.LocationsDictionary = new LocationsDictionary(args1.Result); args.ReferringSitesDictionary = new ReferringSitesDictionary(args1.Result); args.GeoIpDataDictionary = new GeoIpDataDictionary(args1.Result); args.UserAgentsDictionary = new UserAgentsDictionary(args1.Result); args.DeviceDictionary = new DeviceDictionary(args1.Result); } } }

If you look at line 13 above, the GeoIpDataDictionary object being created is inherited from Sitecore's new ReferenceDataDictionary.

This is the glue between GeoIP and the new Reference Data "shared storage" mechanism.

Here is what the code looks like:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

namespace Sitecore.Analytics.DataAccess.Dictionaries {   public class GeoIpDataDictionary : ReferenceDataDictionary<Guid, GeoIpData>   {     public GeoIpDataDictionary(DictionaryBase dictionary, int cacheSize)       : base(dictionary, "GeoIpDataDictionaryCache", XdbSettings.GeoIps.CacheSize * cacheSize)     {       this.ReadCounter = AnalyticsDataAccessCount.DataDictionariesGeoIpsReads;       this.WriteCounter = AnalyticsDataAccessCount.DataDictionariesGeoIpsWrites;       this.CacheHitCounter = AnalyticsDataAccessCount.DataDictionariesGeoIpsCacheHits;       this.DataStoreReadCounter = AnalyticsDataAccessCount.DataDictionariesGeoIpsDataStoreReads;       this.DataStoreReadTimeCounter = AnalyticsDataAccessCount.DataDictionariesGeoIpsDataStoreReadTime;       this.DataStoreWriteTimeCounter = AnalyticsDataAccessCount.DataDictionariesGeoIpsDataStoreWriteTime;     }     public GeoIpDataDictionary(DictionaryBase dictionary)       : this(dictionary, XdbSettings.GeoIps.CacheSize)     {     }     public override TimeSpan CacheExpirationTimeout     {       get       {         return TimeSpan.FromSeconds(600.0);       }     }     public override Guid GetKey(GeoIpData value)     {       return value.Id;     }     public string GetKey(Guid id)     {       return id.ToString();     }   } }

Notice on line 25 that this object is cached for 10 minutes. More on this below.

Reference Data Storage and the GeoIP Lookup Flow

You may be wondering how this Reference Data feature changes what you know about the GeoIP flow from previous versions of the platform.

Let's review the steps:

• Sitecore runs the CreateVisits pipeline. Within this pipeline, there is a processor called UpdateGeoIpData that fires a method called GeoIpManager.GetGeoIpData within Sitecore.Analytics.Tracking.CurrentVisitContext that initiates the GeoIP lookup for the visitor's interaction.

• Sitecore performs a GeoIP data lookup in the GeoIP memory cache.
• NOTE: Cache expiration is set to 10 seconds => TimeSpan.FromSeconds(10.0)

Sitecore.Analytics.Lookups.GeoIpCache:

1 2 3 4 5 6 7 8

public void Add(GeoIpHandle handle) { Assert.ArgumentNotNull((object) handle, nameof (handle)); if (this.cache.Count >= this.maxCount) this.Scavenge(); this.cache.Add(handle.Id, (object) handle, TimeSpan.FromSeconds(10.0)); AnalyticsTrackingCount.GeoIPCacheSize.Value = (long) this.cache.Count; }

• If the GeoIP data IS in the GeoIP memory cache, then it will attach it to the visitor's interaction.

• If the GeoIP data IS NOT in the GeoIP memory cache, it performs a lookup in the Reference Data's GeoIpDataDictionary (KnownDictionaries) memory cache.
• NOTE: Cache expiration is set to 10 minutes => TimeSpan.FromSeconds(600.0). See above for the 10 minute CacheExpirationTimout property on the Sitecore.Analytics.DataAccess.Dictionaries.GeoIpDataDictionary class.

• If the GeoIP data IS in the Reference Data's GeoIpDataDictionary memory cache, it attaches it to the visitor's interaction and adds it to the GeoIP memory cache.

• If the GeoIP data IS NOT in the Reference Data's GeoIpDataDictionary memory cache, it performs a lookup in the SQL ReferenceData database and if found, stores the result in the Reference Data's GeoIpDataDictionary cache and GeoIP memory cache, and then attaches it to the visitor's interaction.

• If the GeoIP data IS NOT in the SQL ReferenceData database, it performs a lookup using the Sitecore Geolocation service and stores the result in the SQL ReferenceData database, the Reference Data's GeoIpDataDictionary cache and GeoIP memory cache, and then attaches it to the visitor's interaction.

Reference Data Storage in SQL

By using SQL Server Management Studio, and opening up the ReferenceData database's DefinitionTypes table, you can see the different types of reference data that is being stored. The GeoIp data type name as you can see below, is called "Tracking Dictionary - GeoIpData".

By looking at the Definitions table, you can see that the data is stored as a Binary data type:

The following SQL Query will return the top 100 GeoIP reference data results:

1 2 3 4

SELECT TOP 100 [xdb_refdata].[DefinitionTypes].Name, [xdb_refdata].[Definitions].Data, [xdb_refdata].[Definitions].IsActive, [xdb_refdata].[Definitions].LastModified, [xdb_refdata].[Definitions].Version FROM [xdb_refdata].[Definitions] INNER JOIN [xdb_refdata].[DefinitionTypes] ON [xdb_refdata].[DefinitionTypes].ID = [xdb_refdata].[Definitions].TypeID WHERE [xdb_refdata].[DefinitionTypes].Name = 'Tracking Dictionary - GeoIpData'

Changes to the GeoIpManager class

Finally, I wanted to provide a glimpse of the changes in the GeoIpManager class that I referenced in my previous post.

By comparing the 8.x version of the GeoIpManager code to 9, you can see the usage of the KnownDataDictionaries.GeoIPs dictionary instead of the Tracker.Dictionaries.GeoIpData (ContactLocation class) from 8.x:

Final Words

I hope that this information helps developers understand more about Reference Data and the updated GeoIP Lookup Flow in Sitecore 9.

As always, feel free to comment or reach me on Slack or Twitter if you have any questions.

Read More

Sitecore xDB: A Mechanic's Guide to Personalization Testing Troubleshooting

Background

On my last few projects, I have experienced first-hand how marketers have leveraged the true power of Content Testing in the Experience Platform to truly gain some fantastic insights so that they can successfully optimize content in order to deliver an improved contextual customer experience.

Most of the tests I have experienced have been personalization-based, and I have helped various teams troubleshoot some glitches that have popped up along the way. I guess you can call me the "content testing mechanic".

In this post, I will provide some insight from my experiences to help other developers who face similar issues, get the issues resolved quickly.

Nuts and Bolts

The main entry point into content testing for users is workflow, and so the assumption is that you have some type of workflow in place to successfully launch tests.

For more information on this, please review Sitecore's documentation on Adding content testing to a workflow  as well as Jonathan Robbins' Sitecore 8 Content Testing post.

With a workflow in place, the following things happen under the covers when you launch a new test:

• A new test item is created that contains all the information about the test. This can be found at this location: /sitecore/system/Marketing Control Panel/Test Lab.
• The Final Renderings XML field will be updated with specific testing attributes that contain values with test reference information*.
• The item that you are testing and the new test item will be published to the web database (based on workflow action).

* To view this, you will need to enable raw values and standard fields in the “View” section of the “View” tab

An example of the Final Renderings XML looks something like the below:

Looking at this XML, you will see that it contains a p:t attribute. This denotes a personalization test reference. More on this further down.

Springs that Pop Out

If after starting a personalization test, personalization on the component(s) that you are testing stops working and you don't see any data appearing in your Test Result dialogue, the most common error in your Sitecore logs will be the following "Evaluation of condition failed. Rule item ID: Unknown, condition item ID exception":

 ERROR Evaluation of condition failed. Rule item ID: Unknown, condition item ID: {4888ABBB-F17D-4485-B14B-842413F88732}  
 Exception: System.NullReferenceException  
 Message: Object reference not set to an instance of an object.  
 Source: Sitecore.ContentTesting  
   at Sitecore.ContentTesting.Pipelines.RenderingRuleEvaluated.TestingRule.Process(RenderingRuleEvaluatedArgs args)  
   at (Object , Object[] )  
   at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)  
   at Sitecore.Rules.RuleList`1.Run(T ruleContext, Boolean stopOnFirstMatching, Int32& executedRulesCount)  
   

At first glance, you might think that this error is simply because of the condition item with ID {4888ABBB-F17D-4485-B14B-842413F88732} that is not published to the web database.

 Unfortunately, this isn't the case.

Under the Hood

After working through Sitecore's testing code, the exception occurs while Sitecore evaluates one of the conditions associated with the personalization rule.

Sitecore.ContentTesting.Pipelines.RenderingRuleEvaluated.TestingRule                 Sitecore.Rules.Evaluate

The invocation that fails and causes the NullReferenceException is the 'rule.Condition.Evaluate(ruleContext)' where:

rule - is the Rule object instantiated from the rule XML definition ( from the presentation details )
Condition - is the root condition definition from the rule XML definitions
ruleContext - is the object containing additional data for the rule evaluation such as:

• Item reference: this should be the page definition item. 
• Test reference: this should be the test associated with the rendering.
• The current MVC rendering object reference

My analysis determined that the most common cause of the error is due to old tests that are still part of the content item's configuration, that are either not stopped correctly, inactive or have been removed.

Fixing the Issue

The fix is to remove the bad/old test references from the item in question's Final Renderings XML field. 

My process to do this is the following:

• Determine what item is throwing the testing exception.
• Enable raw values and standard fields in the “View” section of the “View” tab.
• Copy the Final Renderings XML value of the item and format it so that it is easy to read. This site does a nice job: https://www.freeformatter.com/xml-formatter.html
• Paste you’re the XML into Visual Studio or another editor.
• Locate the attributes in the XML that have a s:pt and remove the attributes.
• Copy and paste the updated XML back into the item's Final Renderings field.
• Save and publish.

After this, the errors will stop appearing in your logs. You will however need to launch your test again.

Final Gotcha

The exact same "Evaluation of condition failed. Rule item ID: Unknown, condition item ID" exception error mentioned above could also occur if content testing has been disabled.

In XP 8.1 and later, it is disabled when the ContentTesting.AutomaticContentTesting.Enabled setting is set to false in the App_Config\Include\ContentTesting\Sitecore.ContentTesting.config file.

This is a bit obscure, as one would think that there would be some other messaging in the logs indicating that this setting has been disabled.

Testing Diagnostics Page for your Toolbox

I have seen some cases where the ribbon of the Optimization tab shows a different number of active tests if compared to the active test list. An example of this is shown below:

One of the first things that you can try is to rebuild your sitecore_testing_index index. If this doesn't help, you can use the diagnostics page below to help troubleshoot the issue.

The page output will look similar to this:

~~~~~~~~~~~~~~

Active filtered tests:
Test search result object checked for Item: sitecore://{703EED9B-C574-4310-AC47-EBCCB651F67E}?lang=en&ver=2, Test Item: sitecore://master/{57807f6f-a836-4132-8b2b-48124b0c4031}?lang=en&ver=1, Is Running: True, Is Cancelled: False

Test search result object checked for Item: sitecore://{0CA61CF3-D35A-4FE7-8CD6-90CF8F61179A}?lang=en&ver=9, Test Item: sitecore://master/{cb771932-d06d-4f42-9392-483fab3cbc1c}?lang=en&ver=1, Is Running: True, Is Cancelled: False
Configuration is null

Test search result object checked for Item: sitecore://{8C65AEB8-90A2-4348-BCDB-D8AB6CBA5974}?lang=en&ver=1, Test Item: sitecore://master/{0b298164-fcc5-4803-acda-5a321e8c2797}?lang=en&ver=1, Is Running: True, Is Cancelled: False

1 {57807F6F-A836-4132-8B2B-48124B0C4031}
2 {0B298164-FCC5-4803-ACDA-5A321E8C2797}
Active tests:
1 sitecore://master/{57807f6f-a836-4132-8b2b-48124b0c4031}?lang=en&ver=1
2 sitecore://master/{cb771932-d06d-4f42-9392-483fab3cbc1c}?lang=en&ver=1
3 sitecore://master/{0b298164-fcc5-4803-acda-5a321e8c2797}?lang=en&ver=1

~~~~~~~~~~~~~~~

The diagnostic output example above shows us that the issue lies with the test item with ID {cb771932-d06d-4f42-9392-483fab3cbc1c}.

To fix the issue, you will need to locate the test item with that ID and either set its "Is Running" field value to "No" or simply delete that item.

Make sure that these updates get published to your web database.

Read More

Sitecore xDB - Adding Custom Data to Outcomes and Using it for Personalization

In a recent Digital Strategist MVP Webinar hosted by Chris Nash, Chris made a quick mention of a custom values property on outcomes that was available to developers to store custom data associated with the outcome.

Having not known about this Easter egg prior, I looked into it straight away.

My research revealed that there wasn't any documentation or an example on the web, so I thought that I would take the opportunity to demonstrate the usage in a real-world implementation.

Use Case

One of the objectives in our Xccelerate roadmap was the ability to personalize based on a contact's previous purchase. So for example; "If a visitor has purchased a product in the last 10 days, let's show them a CTA of a related product". The obvious objective was to drive the contact directly into the purchase funnel, increasing the possibility of another conversion.

In our configuration, we had already created a Purchase outcome item and were capturing monetary value, so it was just a matter of attaching the purchased line items to the outcome and building a new condition.

It is important to note that the "Monetary Value Applicable" checkbox must be checked on the Outcome item in Sitecore so that the values will show up in the various reporting dashboards.

Adding Custom Data to the Outcome

The code to achieve this is pretty straightforward. In our case, I added the logic to a location where I had a list of line items available that a visitor had just purchased.

The method accepts a monetary value and a list of key value pairs containing data that I would attach to my registered outcome. In my usage, I stored all the product SKUs, along with the quantity purchased for each line.

1:            public void RecordPurchaseOutcome(decimal monetaryValue, List<KeyValuePair<string, string>> customValues = null)  
2:            {       
3:                 var id = ID.NewID;  
4:                 var interactionId = ID.Parse(Tracker.Current.Interaction.InteractionId);  
5:                 var contactId = ID.Parse(Tracker.Current.Contact.ContactId);  
6:                 var definitionId = new ID(ItemConstants.ProductPurchaseOutcomeItem);  
7:    
8:                 var outcome = new ContactOutcome(id, definitionId, contactId)  
9:                 {  
10:                      DateTime = DateTime.UtcNow.Date,  
11:                      MonetaryValue = monetaryValue,  
12:                      InteractionId = interactionId,  
13:                 };  
14:    
15:                 if (customValues != null && customValues.Any())  
16:                 {  
17:                      foreach (var customValue in customValues)  
18:                      {  
19:                           outcome.CustomValues[customValue.Key.ToUpperInvariant()] = customValue.Value;  
20:                      }  
21:                 }  
22:    
23:                 Tracker.Current.RegisterContactOutcome(outcome);  
24:            }  

The id of the outcome item on line 6 displayed as ItemConstants.ProductPurchaseOutcomeItem is unique to my implementation.

After a contact's session ended, the custom data was stored in the MongoDB Outcome collection as follows:

Personalization using the Outcome's Custom Data

The final piece of the puzzle was to build the personalization condition that could pull out the custom data from the contact's recorded outcome, based on a time frame.

I created the condition item in the Outcomes element folder at this location: /sitecore/system/Settings/Rules/Definitions/Elements/Outcomes.

The rule text was set to the following:

 where the current contact has registered the [OutcomeId,Tree,root=/sitecore/system/Marketing Control Panel/Outcomes,specific] outcome with a custom data key that is case-insensitively equal to [CustomData,,,value] within the last [days,Integer,,number] day(s)  

This is the code that powered the condition, accessing the recorded custom data in the contact's outcome, and determined if it fell within a day range:

1:    public class CustomDataOutcomeRegisteredWithinLastDaysCondition<T> : WhenCondition<T> where T : RuleContext  
2:    {  
3:      private OutcomeManager _outcomeManager;  
4:    
5:      public string OutcomeId { get; set; }  
6:      public string Days { get; set; }  
7:      public string CustomData { get; set; }  
8:    
9:      protected override bool Execute(T ruleContext)  
10:      {  
11:        Assert.ArgumentNotNull(ruleContext, "ruleContext");  
12:        Assert.IsNotNull(Tracker.Current, "Tracker.Current is not initialized");  
13:        Assert.IsNotNull(Tracker.Current.Session, "Tracker.Current.Session is not initialized");  
14:    
15:        Guid result;  
16:        if (!Guid.TryParse(OutcomeId, out result))  
17:        {  
18:          Log.Debug(string.Format("Specified outcome [{0}] was not a valid Guid", OutcomeId));  
19:          return false;  
20:        }  
21:    
22:        _outcomeManager = Factory.CreateObject("outcome/outcomeManager", true) as OutcomeManager;  
23:    
24:        if (_outcomeManager != null)  
25:        {  
26:          int pastDays;  
27:          var validDays = int.TryParse(Days, out pastDays);  
28:    
29:          if (!validDays)  
30:          {  
31:            return false;  
32:          }  
33:    
34:          var targetDate = DateTime.Today.AddDays(-pastDays);  
35:          var pastOutcomes = _outcomeManager.GetForEntity<IOutcome>(Tracker.Current.Contact.ContactId.ToID(), result.ToID());  
36:    
37:          foreach (var outcome in pastOutcomes)  
38:          {  
39:            if (outcome.DateTime >= targetDate)  
40:            {  
41:              return outcome.CustomValues[CustomData.ToUpperInvariant()] != null;  
42:            }  
43:          }  
44:    
45:          return false;  
46:        }  
47:    
48:        return false;  
49:      }  
50:    }  

After my rule and code were added to Sitecore, I was able to apply the freshly minted condition to my component:

The end result was the ability to personalize based on a previous purchase that a contact had made within the last x number of days.

In my example, I personalized the content of my component if the contact had purchased a Large, Hot Penne Pasta and Meatballs Tray within the last 10 days.

Read More

How to trigger xDB Pattern Cards using jQuery AJAX with Sitecore MVC

Background 

Sitecore makes it easy for content marketers to assign Profile Cards to pages when implementing a behavioral profile strategy, but more often than not, there are areas of today's modern Sitecore sites where this is a bit challenging.

For the same reason that you may want to trigger Goals or Outcomes using jQuery AJAX, an event driven way to trigger Pattern Cards would be equally useful.

My interest to achieve this was sparked while working on a eCommerce site where we implemented many, many dynamic lightboxes / modals during the ordering flow where we wanted to implement a part of our behavioral profiling strategy.

NOTE: The P’s of Sitecore Personalization can by somewhat tricky to understand, so if you need to brush up on the lingo before reading further, I suggest that you take a look at Mike Shaw's post on Profile Cards and Other P’s of Sitecore Personalization.

Architecture 

I am a fan of using Sitecore.Services.Client, and as I explained in this post, making your controllers xDB / session aware is pretty easy. If you prefer, you can most certainly use a regular MVC controller.

When thinking through the architecture, I wanted to be able to trigger multiple pattern cards using percentages as it would provide the most flexibility.

I needed to be sure that I had the right score calculation checks in place, as I wanted the same results as if the cards were triggered from a page where they had been assigned using the Content or Experience Editor.

You will see in the code that follows where I needed to ensure that the percentages sum had to be equal to 100 if there were multiple cards, or if there were more than one card without an assigned percentage to any of them, the percentages needed to be distributed evenly.

Pattern Card Model

Controller Action

This controller action may be a bit long-winded, but I wanted to demonstrate the flow of logic, from top to bottom.

Trigger using jQuery

Finally, here is an example button click event where we post an array of Pattern Card objects to our controller action.

Note, if you are using Sitecore.Services.Client or WebAPI, there is a known issue that will force you to change your data to be a single anonymous object instead of a raw array.

Read More

Sitecore's IP Geolocation Service - Working with the Missing Area Code, Missing Time Zone and Testing GeoIP Lookups

Background

My current projects make heavy use of GeoIP personalization, and as a result, I have had the opportunity to dig deep into Sitecore's IP Geolocation Service features, uncovering the gaps and figuring out ways to get around them.

If you need help setting up the Geolocation Service, make sure you check out my post: http://sitecoreart.martinrayenglish.com/2015/08/setting-up-sitecores-geolocation-lookup.html

The Missing Area Code

Sitecore gives you a really nice set of rules to work with once you have the service enabled:

Looking above, you will see that the first rule is based on a visitor's area code.

Unfortunately, this rule doesn't work. After decompiling the GeoIP assembly, I was able to determine that the AreaCode property of the WhoIs object is never set in the MapGeoIpResponse processor:

And the rule that is supposed to use the area code:

So, make sure that you use the "postal code" condition if you plan on doing this type of personalization, and not the "area code" one.

The Missing Time Zone

One of my projects has a requirement around the ability to personalize based on the time of day.

When talking about this over lunch with some fellow Sitecore geeks, I got the "..doesn't Sitecore do that out of the box?" response.

At first thought, this seemed like a valid response, as Sitecore advertises that their IP Geolocation Service has the ability to identify a visitor's time zone.

Well, we know that there aren't any time-based rules shown by the Geolocation ruleset above, but if we have the person's time zone, we could at least use this information to conjure up some custom condition that will allow us to do this type of personalization.

Focusing on the GeoIpResponseProcessor again, you will notice that there is no time zone property on the WhoIs object. So, it's clear that we actually don't have a time zone to use.

After digging in a bit further, and confirming a few things with Sitecore support, I was able to determine that the JSON response from Sitecore's Geolocation service does actually contain the time zone information:

So, why wouldn't they make this available via the WhoIs object and API?

This seems a bit odd.

I registered a wish with my support ticket, so hopefully we will see this in a future release. Until then, we have to write a bit of code to peel off the value so that we can use it in our customization.

Time Zone Fun

Getting the Time Zone

In order to get that time zone value from the service, I had to create a custom processor to grab the raw value from service response.

Processor

Patch Configuration

Converting Olson to Windows Time

As you can see above, time zones from the service are in IANA time zone format (also known as Olson) as described here: https://dev.maxmind.com/faq/how-can-i-determine-the-timezone-of-a-website-visitor/

So the next order of business was to take the Olson time zone ID and convert it to a Windows time zone ID so that when I was ready to perform the local time calculation, it would be fairly easy using the .NET Framework's TimeZoneInfo class.

After a quick Google, I came across this Stack Overflow article that I based my conversion helper method on:
http://stackoverflow.com/questions/5996320/net-timezoneinfo-from-olson-time-zone

With these pieces on place, I had everything I needed to build out my time-based personalization rules!

Bonus: Testing GeoIP Lookups

To make GeoIP Lookup testing easy, I created a processor that injects a mock ip address obtained from a setting, so you can verify that your dependent rules work as expected.

Note: There is a dated module called Geo IP Tester on the Marketplace, but unfortunately it isn't compatible with Sitecore 8.x due to the changes in the API.

Processor

Patch Configuration

Read More

3-Step Guide: How to trigger an xDB goal using jQuery AJAX with Sitecore MVC

Background

After cruising around the web looking for some code to use to trigger a goal using jQuery AJAX, I discovered that there weren't really any easy to understand, end-to-end, current examples of how to do this using Sitecore MVC.

So, I decided to write up a quick post to demonstrate how to do this in 3 easy steps.

Step 1 - Create MVC Controller

The first step is to create an MVC Controller with an action that you will use to trigger the goal:

 using System;  
 using System.Linq;  
 using System.Web.Mvc;  
   
 using Sitecore.Analytics;  
 using Sitecore.Analytics.Data.Items;  
 using Sitecore.Mvc.Controllers;  
   
 namespace MyNamespace.Controllers  
 {  
   public class AnalyticsController : SitecoreController  
   {  
     private const string DefaultGoalLocation = "/sitecore/system/Marketing Control Panel/Goals";  
   
     [HttpPost]  
     public ActionResult TriggerGoal(string goal)  
     {  
       if (!Tracker.IsActive || Tracker.Current == null)  
       {  
         Tracker.StartTracking();  
       }  
   
       if (Tracker.Current == null)  
       {    
         return Json(new { Success = false, Error = "Can't activate tracker" });  
       }  
   
       if (string.IsNullOrEmpty(goal))  
       {  
         return Json(new { Success = false, Error = "Goal not set" });  
       }  
   
       var goalRootItem = Sitecore.Context.Database.GetItem(DefaultGoalLocation);  
       var goalItem = goalRootItem.Axes.GetDescendants().FirstOrDefault(x => x.Name.Equals(goal, StringComparison.InvariantCultureIgnoreCase));  
   
       if (goalItem == null)  
       {  
         return Json(new { Success = false, Error = "Goal not found" });  
       }  
   
       var page = Tracker.Current.Session.Interaction.PreviousPage;  
       if (page == null)  
       {  
         return Json(new { Success = false, Error = "Page is null" });  
       }  
   
       var registerTheGoal = new PageEventItem(goalItem);  
       var eventData = page.Register(registerTheGoal);  
       eventData.Data = goalItem["Description"];  
       eventData.ItemId = goalItem.ID.Guid;  
       eventData.DataKey = goalItem.Paths.Path;  
       Tracker.Current.Interaction.AcceptModifications();  
   
       Tracker.Current.CurrentPage.Cancel();   
   
       return Json(new { Success = true });  
     }  
   }  
 }  

Step 2 - Register a custom MVC route

The next step is to create a custom processor for the initialize pipeline and define custom route in the Process method similar to the following:

 using System;  
 using System.Collections.Generic;  
 using System.Linq;  
 using System.Web;  
 using System.Web.Mvc;  
 using System.Web.Routing;  
 using System.Web.UI.WebControls;

 using Sitecore.Pipelines;  
   
 namespace MyNamespace  
 {  
  public class RegisterCustomRoute  
  {  
   public virtual void Process(PipelineArgs args)  
   {  
    Register();  
   }  
   
   public static void Register()  
   {  
    RouteTable.Routes.MapRoute("CustomRoute", "MyCustomRoute/{controller}/{action}/{id}");  
   }  
   
  }  
 }  

Add this processor to the initialize pipeline right before the Sitecore InitializeRoutes processor. You can do this with the help of the patch configuration file in the following way:

 <?xml version="1.0" encoding="utf-8"?>  
 <configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">  
  <sitecore>  
   <pipelines>  
    <initialize>  
     <processor type="MyNamespace.RegisterCustomRoute, MyAssembly"/>  
    </initialize>‌  
   </pipelines>  
  </sitecore>  
 </configuration>  

Step 3 - Trigger using jQuery

Finally, trigger the goal by name using a few lines of jQuery:

 $.post("/MyCustomRoute/Analytics/TriggerGoal?goal=tweet" ,function(data){  
      //Do something with data object  
 });  

Read More

Presentation Targets - Chuck Norris's version of Sitecore's Item Rendering

Background

My current project is unique in the way that the Home Page of the site is designed. Basically, the Home Page is a single-page app, where the entire list of products will start appearing as you scroll down the page. They will be grouped by category, and as you keep scrolling down and reach a product list in a new category, the navigation will automatically change to reflect the new category.

It is one of those designs where the client gets all googly eyed over how pretty it looks and you as the Sitecore Architect are left thinking, "how in the world am I going to give my content authors a great experience when building this out in the Experience Editor?"

Well, an obvious approach would be to have one massively long Home Page with a ton of renderings plopped all over the show. But, I was left asking myself, "Self, we need to break this down into separate Product Category pages, so that my Content Authors will have a good experience and the pages will be easier to maintain.

The Problem

Great idea right? But, how would all the content from these pages be dynamically brought over to the Home Page so that we can do this fancy, animated show / hide trick?

I also had to make sure that I would be able to personalize everything based on the approach that I landed on. For example, depending on the visitors' identified persona, or the time of day, I wanted to be able to switch out the order in which the product categories would appear on the Home Page.

True Item Rendering

Doing a bit of research, I came across one of Vasiliy Fomichev's older posts regarding Sitecore's Item Rendering. We share the same underwhelming feeling about Sitecore's implementation of the Item Rendering, and reading further, I was pleasantly surprised to find that his requirements and "true item rendering solution" matched what I was looking for:

The Item Rendering must use the presentation components found in the presentation details of the referenced item

After I downloaded his example project, and fired it up, I realized that this would get me most of the way there; I could set the datasource to any content item that contained presentation components with set datasources, and it would render them in the location where my Item Rendering was placed.

The nice thing too was that I had full Experience Editor support.

So, I could modify the properties of both the Item Rendering and the Renderings within the Rendering (that's a tongue twister).

Sweet! 

Presentation Targets is Born

My next thought to myself was "Self, how awesome would it be if I could drop this on a page somewhere and target specific renderings within specific placeholders that exist on another item?"

This would give me a fantastic amount of flexibility and ultimate re-usability of already built presentation components.

So, I decided to update the project and add the following improvements:

1. A new Presentation Targets Rendering and updated patch file so that both the new rendering and out-of-the-box Item Rendering could live in harmony.
2. Adjusted the code to allow rendering parameters to be passed through from the target renderings.
3. Added the ability to set the placeholders and renderings that you would like to target on the datasource item. These are both set as pipe-separated (|) lists of rendering parameters:

So, what does this give me?

To put it simply; a rendering that can target any renderings inside any placeholders on an existing item and render their content.

As I mentioned earlier, you have the ability to modify the content of the targeted renderings within the location that you have placed the Presentation Targets Rendering within the Experience Editor.

So let's study this with an example use case:

1. You worked very hard to build out a wonderful product carousel, and implemented a series of personalization rules on several of the slides.
2. There is a promotion this week for a series of products that are part of the wonderful carousel, and so your boss wants you to add the carousel to the Home Page.
3. Instead of adding a new carousel rendering to the page, setting up each of the slides with datasources and re-applying your personlization rules from scratch, you could place the Presentation Targets Rendering on the Home Page, set the datasource to the item that has the wonderful carousel, set the placeholder that it exists in, and the rendering id of the carousel.
4. You are done!

The Possibility of Nested Personalization

While working with Lars Peterson and the SBOS team, this topic came a few times during the implementation of a Home Page carousel (yes, I just love carousels if you couldn't tell already).

Looking at the proposed use case: 

• For a specific group of visitors, change the entire carousel's datasource so that it displays one that is relevant to the specific group.
• For visitors that are outside of this group, personalize slide 1 with conditions based on geolocation.

So, basically allowing for the possibility to personalize the datasource of an entire carousel, and within that, personalize each individual slide.

Without Presentation Targets

This is achievable by having more than one carousel rendering where we apply a rule with the "Hide Component” set to only show the carousel with the personalized slides for our targeted visitor.

With Presentation Targets

Granted that you would have to initially set up the presentation of the carousels and slides on a seperate item, using the Presentation Targets Rendering will allow you to achieve this level of personalization because with Experience Editor support, you could set personalization rules on both the Presentation Targets Rendering and the Carousel's Slide Renderings.

Now, isn't that fancy?

Final Thoughts

The Presentation Targets module opens up some new opportunities to explore content reuse and the depths of personalization within the Experience Platform.

I will be sure to share more of my experiences as I dig in a bit deeper.

Full source code is available on GitHub: https://github.com/martinrayenglish/PresentationTargets

Another shout-out to Vasiliy Fomichev on his awesome post to get the fire started!

Read More

Setting up Sitecore's Geolocation Lookup Services in a Production Environment

Background

We have been working with Sitecore’s Business Optimization Services (SBOS) team for quite some time, helping one of our client's stretch the legs of the Experience Platform.

One of the tasks on the list included setting up Sitecore's Geolocation Service so that we could personalize based on the visitor's location. The SBOS team had some pretty slick rules set up on the Home Page of a site, where they switched out carousel slides and marketing content spots based on the visitor's location.

Sitecore Geolocation Lookup Service and MaxMind

There have been some changes with regards to the Geolocation / MaxMind set up because Sitecore launched IP Geolocation directly to their customers via the Sitecore App Center instead of going through MaxMind to use their service. Here is a link to Sitecore's documentation site that contains set up instructions, and how to migrate from MaxMind if purchased and have been working directly with them in the past:
https://doc.sitecore.net/Sitecore%20Experience%20Platform/Analytics/Setting%20up%20Sitecore%20IP%20Geolocation

Setup

After sifting through the documentation, we highlighted the following steps that needed to be completed in order to get up and running, and validate that things were indeed working:

1. Download and install the Geolocation client package from https://dev.sitecore.net/Downloads.aspx
2. Enable ALL configuration files in the CES folder
3. Whitelist geoIp-ces.cloud.sitecore.net and discovery-ces.cloud.sitecore.net
4. Test personalization

NOTE: Depending on your firewall, you may only have the option to whitelist by IP address. If this applies to you, you will need obtain the list of Azure IP addresses from the following link: Azure Datacenter IP Address Ranges. This is what happened to us, and I call tell you that the list is looooooooooooooong!!! Your network guy or gal will hate you!

Unfortunately for us, there was one piece of configuration that isn't documented. It so happened to be one of the most important pieces.

You will know what I am referring to as you read further along.

Testing

With all the pieces in place, we started testing our personalization.

Things worked beautiful on our Staging Server, but Production was a non-starter! So, as good detectives, we started our troubleshooting by looking at the differences between Staging and Production.

Load Balancer / Firewall / CDN woes

Our client uses Incapsula to protect their production websites. It does a great job protecting and caching their various site's to ensure optimal performance. It has however given us some grey hairs in the past when dealing with Sitecore's Federated Experience Manager. But, that's a story for another time.

The Incapsula CDN was the main difference between Staging and Production.

After running several tests with Fiddler and capturing packets using Wireshark, we were able to the determine that the Geolocation service was not obtaining the visitor's actual IP address. Instead, it was passing along Incapsula's IP address.

The reason for this was identified in the CreateVisitProcessor within the CreateVisits analytics pipeline. As you can see below, it was passing over the Request.UserHostAddress value.

This doesn't work when you are behind a load balance or proxy, as described by this article: http://stackoverflow.com/questions/200527/userhostaddress-gives-wrong-ips

Digging further, we discovered another interesting processor in the CreateVisits pipeline called XForwardedFor. Aha! As we know; the "...header field is a de facto standard for identifying the originating IP address of a client connecting to a web server through an HTTP proxy or load balancer."

Looking at the code below, you will notice that it's pulling in a setting, and if it's not empty, it is used as the key to obtain the value from the request's header NameValueCollection.

After digging around, and talking to support, we discovered the config file named Sitecore.Analytics.Tracking.config and setting below:

 <!-- ANALYTICS FORWARDED REQUEST HTTP HEADER  
    Specifies the name of an HTTP header variable containing the IP address of the webclient.  
    Only for use behind load-balancers that mask web client IP addresses from webservers.  
    IMPORTANT: If this setting is used incorrectly, it allows IP address spoofing.  
    Typical values are "X-Forwarded-For" and "X-Real-IP".  
    Default value: "" (disabled)  
 -->  
 <setting name="Analytics.ForwardedRequestHttpHeader" value="" />  

Light at the end of the tunnel

After setting the value to "X-Forwarded-For" as shown below, the magical Geolocation based personalization started working like a champ!

<setting name="Analytics.ForwardedRequestHttpHeader" value="X-Forwarded-For" />

NOTE: We discovered that casing matters when setting the value. "X-FORWARDED-FOR" will NOT work. It needs to be set exactly like I have it above. For more information on this, you can read this Stack Overflow article:
http://stackoverflow.com/questions/11616964/is-request-headersheader-name-in-asp-net-case-sensitive


I hope that this information helps make your Sitecore IP Geolocation configuration go smoothly for your environment!

A special thanks to Kyle Heon from Sitecore for his support through this process.

Read More