Wednesday, December 16, 2015

Sitecore Commerce Connect - Cleaning up the Rusty Shopping Cart

Standard

Background

In my last project, I really got to dig deep into Commerce Connect and can say that the Sitecore team did a fantastic job architecting the module.

As with all E-Commerce implementations, one of our requirements was to have a shopping cart that would persist for a period of time. The problem we were faced with was that the shopping cart needed to exist on the Sitecore side, and Commerce Connect expects the cart to be handled by the External Commerce System. Our External Commerce System was only responsible for handling the checkout process.

Reading through the Sitecore Connect Integration Guide, we discovered the "Storing a copy of the cart locally" section on page 17 that mentioned 2 sample cart repositories that ship with Commerce Connect:
  • EaStateCartRepository - for storing carts in MongoDB  (enabled by default)
  • EaStateSqlBasedCartRepository - for storing carts in SQL

Basically, what this section described was exactly what we wanted to do.

However, the documentation had a very specific caveat; "...these are only sample repositories and should not be used in a production scenario."

So, what could we wrong with the sample code? Surely if it shipped with the module and was enabled by default, it was OK?

Unfortunately, we discovered that the sample EaStateCartRepository had a significant flaw in it during our testing. A bug that could wreak havoc if put in production.

In this post, I will describe the issues with the sample cart repository, and how you can fix it so that it can be used in production.

Side note: I couldn't find a sample of the EaStateSqlBasedCartRepository mentioned in the documentation. Also, it was unfortunate that the sample repository didn't have any documentation describing it's functionality. So, you will need to get out your trusty decompiler to review the code.



About Commerce Connect

Before diving in, I want to take a step backwards to get some important facts highlighted about the module, as there has been a lot of confusion with partners wanting to understand comparisons between Commerce Connect and Commerce Server. This really isn't even an apples to oranges comparison. It's more like a cars to roads comparison.

What Commerce Connect IS NOT

  • A stand-alone E-Commerce solution.
  • A competitor or alternative to Sitecore Commerce Server.
  • An upgrade path for Sitecore Ecommerce Services.
  • A solution that provides out-of-the-box components like you would get with Active Commerce or uCommerce.

What Commerce Connect IS

  • A pluggable framework. 
  • A pass-through to an underlying E-Commerce platform or service. 
  • A common integration layer that external systems can be integrated with.
  • A means of bringing all the aspects of the Sitecore Experience Platform around user engagement to external E-Commerce systems.

With all this being said, if you are looking to implement a solution using the module, you REALLY NEED TO READ the following documentation in order to get a handle of all the features that the module provides and how to implement it correctly:


So, What's Wrong with the Cart Repository Anyway?

The sample cart repository issue that we found stemmed from the first two processors within the CreateOrResumeCart pipeline found in Sitecore.Commerce.Carts.config:

1:  <!-- CREATE OR RESUME CART  
2:        Initiate the creation of a shopping cart and in the process to:   
3:         - Load persisted, potentially abandoned cart, if present;  
4:         - Call resumeCart pipeline to resume loaded cart;  
5:         - Call createCart pipeline to create cart if no cart was found on previous steps.  
6:     -->  
7:     <commerce.carts.createOrResumeCart>  
8:      <processor type="Sitecore.Commerce.Pipelines.Carts.CreateOrResumeCart.FindCartInEaState, Sitecore.Commerce">  
9:       <param ref="eaStateCartRepository" />  
10:      </processor>  
11:      <processor type="Sitecore.Commerce.Pipelines.Carts.CreateOrResumeCart.RunLoadCart, Sitecore.Commerce" />  
12:      <processor type="Sitecore.Commerce.Pipelines.Carts.CreateOrResumeCart.RunResumeCart, Sitecore.Commerce" />  
13:      <processor type="Sitecore.Commerce.Pipelines.Carts.CreateOrResumeCart.RunCreateCart, Sitecore.Commerce" />  
14:     </commerce.carts.createOrResumeCart>  
The FindCartInEaState processor retrieves a CartId for the current user, using the request.UserId shown below:

1:  public override void Process(ServicePipelineArgs args)  
2:  {  
3:     Assert.ArgumentNotNull((object) args, "args");  
4:     CreateOrResumeCartRequest request = (CreateOrResumeCartRequest) args.Request;  
5:     List<CartBase> list = Enumerable.ToList<CartBase>(Enumerable.Where<CartBase>(this.repository.GetByUserName(request.UserId, request.ShopName), (Func<CartBase, bool>) (c =>  
6:     {  
7:      if (c.Name == request.Name)  
8:       return c.CustomerId == request.CustomerId;  
9:      return false;  
10:     })));  
11:     if (list.Count == 1)  
12:      args.Request.Properties["CartId"] = (object) list[0].ExternalId;  
13:     else  
14:      args.Request.Properties["CartId"] = (object) null;  
15:    }  
16: }  
When the RunLoadCart processor runs, it initializes the LoadCartFromEaState pipeline.

1:  public override void Process(ServicePipelineArgs args)  
2:  {  
3:     Assert.ArgumentNotNull((object) args, "args");  
4:     Assert.ArgumentNotNull((object) args.Request, "args.Request");  
5:     object obj = args.Request.Properties["CartId"];  
6:     if (obj == null)  
7:      return;  
8:     CreateOrResumeCartRequest resumeCartRequest = (CreateOrResumeCartRequest) args.Request;  
9:     this.RunPipeline("commerce.carts.loadCart", new ServicePipelineArgs((ServiceProviderRequest) new LoadCartRequest(resumeCartRequest.ShopName, obj.ToString(), resumeCartRequest.UserId), args.Result), args.Request);  
10: }  
This pipeline calls the GetByCartId method of the EaStateCartRepository. It does not pass the UserId as an argument, despite the GetByCartId method requiring a CartId and a UserId to perform the lookup.

The EaStateCartRepository has a private string called lastAccessedUserName, which should be the contact id of the visitor / contact. This variable is set in several methods in the EaStateCartRepository, however it is only used in the GetByCartId method.

1:  public virtual Cart GetByCartId(string cartId, string shopName)  
2:  {  
3:     Assert.ArgumentNotNullOrEmpty(cartId, "cartId");  
4:     Assert.ArgumentNotNullOrEmpty(shopName, "shopName");  
5:     Type type = this.entityFactory.Create("Cart").GetType();  
6:     Cart cart1 = (Cart) null;  
7:     if (!string.IsNullOrEmpty(this.lastAcessedUserName))  
8:      cart1 = (Cart) Enumerable.SingleOrDefault<CartBase>(this.GetByUserName(this.lastAcessedUserName, shopName, type), (Func<CartBase, bool>) (cart => cart.ExternalId == cartId));  
9:     return cart1 ?? (Cart) Enumerable.SingleOrDefault<CartBase>(this.GetAll(shopName, type), (Func<CartBase, bool>) (cart => cart.ExternalId == cartId));  
10: }  
Looking at Sitecore.Commerce.Carts.config, you will notice that only one instance of the EaStateCartRepository will be created by the provider engine:

1:  <eaStateCartRepository type="Sitecore.Commerce.Data.Carts.EaStateCartRepository, Sitecore.Commerce" singleInstance="true">  
2:     <param ref="entityFactory" />  
3:     <param ref="eaPlanProvider" />  
4:    </eaStateCartRepository>  
What this means is that the lastAccessedUserName variable can be the id of a different contact.

In our investigation, the lastAccessedUserName not matching the current contact id (Tracker.Current.Contact.ContactId), caused the GetAll fallback method to fire (line 9 in the GetByCartId method above).

This method is massively expensive as it iterates through all users to retrieve their carts!

1:  /// <summary>  
2:  /// Gets all.  
3:  ///   
4:  /// </summary>  
5:  /// <param name="shopName">Name of the shop.</param><param name="cartType">Type of the cart.</param>  
6:  /// <returns>  
7:  /// The collection of carts.  
8:  ///   
9:  /// </returns>  
10:  private IEnumerable<CartBase> GetAll(string shopName, Type cartType)  
11:  {  
12:   Assert.ArgumentNotNullOrEmpty(shopName, "shopName");  
13:   ID itemId = this.eaPlanProvider.GetEaPlanId(shopName).Item1;  
14:   EngagementPlanItem engagementPlanItem = new EngagementPlanItem(Tracker.DefinitionDatabase.GetItem(itemId));  
15:   ContactManager contactManager = CommerceAutomationHelper.GetContactManager();  
16:   List<CartBase> list = new List<CartBase>();  
17:   foreach (CustomItemBase customItemBase in engagementPlanItem.States)  
18:   {  
19:       foreach (ID id in (IEnumerable<ID>) AutomationContactManager.GetStateContactsIdsPaged(customItemBase.ID))  
20:       {  
21:        Contact contact = contactManager.LoadContactReadOnly(id.Guid);  
22:        if (contact == null)  
23:        {  
24:            if (Tracker.Current.Contact.ContactId == id.Guid)  
25:             contact = Tracker.Current.Contact;  
26:            else  
27:             continue;  
28:        }  
29:        IEnumerable<CartBase> customDataOrNull = CommerceAutomationHelper.GetCustomDataOrNull<IEnumerable<CartBase>>(AutomationStateManager.Create(contact).GetCurrentStateInPlan(itemId), "commerce.carts");  
30:        if (customDataOrNull != null)  
31:            list.AddRange(customDataOrNull);  
32:       }  
33:   }  
34:   return (IEnumerable<CartBase>) list;  
35:  }  
As we ran tests with more and more users, the mismatch started happening all the time, and the expensive GetAll fallback method kept firing, causing the server's CPU usage to spike.

Yikes!


Fixing the Cart Repository 

How Sitecore Should Fix It

To permanently fix this issue, the Sitecore team should make the LoadCartFromEaState pipeline pass the GetByCartId method the ContactId and the lastAccessedUserName variable should be purged from the class.

My Fix

In my case, I created a new cart repository class that inherited from EaStateCartRepository. I then added a new GetCartByIdFallback method where I would retrieve the cart based on the current contact id (Tracker.Current.Contact.ContactId).

1:  private Cart GetCartByIdFallback(string cartId, string shopName, Type cartType)  
2:  {  
3:       if (Tracker.Current.Contact == null)  
4:       {       
5:            return null;  
6:       }  
7:    
8:       string contactId = Tracker.Current.Contact.ContactId.ToString();  
9:    
10:       List<CartBase> carts = this.GetByUserName(contactId, shopName, cartType).ToList();  
11:       Cart cart1 = (Cart) null;  
12:    
13:       if (carts.Any())  
14:       {  
15:            cart1 = (Cart) carts.SingleOrDefault(cart => cart.ExternalId == cartId);  
16:    
17:            if (cart1 != null)  
18:            {  
19:                 return cart1;  
20:            }  
21:       }  
22:    
23:       return null;  
24:  }  
Finally, I overrode the GetByCartId method to use my GetCartByIdFallback instead of the expensive GetAll fallback method if the cart couldn't be found for the lastAcessedUserName, or as mentioned above, the lastAcessedUserName didn't match the current contact id (Tracker.Current.Contact.ContactId).

1:  public override Cart GetByCartId(string cartId, string shopName)  
2:  {  
3:       Assert.ArgumentNotNullOrEmpty(cartId, "cartId");  
4:       Assert.ArgumentNotNullOrEmpty(shopName, "shopName");  
5:       Type type = this.entityFactory.Create("Cart").GetType();  
6:       Cart cart1 = (Cart)null;  
7:       if (!string.IsNullOrEmpty(this.lastAcessedUserName))  
8:       {  
9:            cart1 = (Cart)Enumerable.SingleOrDefault<CartBase>(this.GetByUserName(this.lastAcessedUserName, shopName, type), (Func<CartBase, bool>)(cart => cart.ExternalId == cartId));  
10:       }  
11:       return cart1 ?? GetCartByIdFallback(cartId, shopName, type);  
12:  }  

Final Configuration

To put my new cart repository into action, I simply had to modify the Sitecore.Commerce.Carts.config to use my new repository:

1:  <eaStateCartRepository type="CommerceTest.MyNewCartRepository, CommerceTest" singleInstance="true">  
2:   <param ref="entityFactory" />  
3:   <param ref="eaPlanProvider" />  
4:  </eaStateCartRepository>  

After this, the cart worked perfectly!


Sunday, November 22, 2015

Getting on the 8.1 Train

Standard

Background

I recently got my development team on one of our projects updated from 8.0 Update-5 to 8.1. It was a pretty seamless process, obviously always easier when you are in development as opposed to performing an upgrade on a production site.



Content 

We use Hedgehog's Team Development for Sitecore (TDS), and so all of our content was serialized and available in source control.

Solr 

We are using Solr on the project, and after following these steps, everything was working like a champ:

  • Added the required assemblies from the 8.1 Solr Support package from here: http://bit.ly/1NlFu4W
  • Updated our Global.asax to include the Castle / Solr Support. 
  • Ran Patrick Perrone's Solr switching script http://bit.ly/1YkXPUS
  • Added the new sitecore_marketing_marketingdefinitions index. 
  • Rebuilt all indexes to verify Solr and Sitecore were playing nice.   


MVC Areas 

We have been using a custom implementation to address the unsupported areas issue for quite some time. The switch over to Sitecore controller renderings with support for areas was straightforward. I just removed the pipeline processor in our custom implementation, changed the template type from our custom controller rendering to the new OOTB controller rendering with the area field populated and it worked perfectly.

NuGet and Such 

As most of us are aware, 8.1 supports the latest and greatest versions of MVC and Web Forms. So I had to make sure that my various modules and solution's NuGet Packages were up-to-date with version 5.2.3 of MVC, Razor 3.2.3 and WebPages 3.2.3. One thing I noticed that was interesting was the version of Microsoft.AspNet.Cors was still 5.1.2.

Glass Mapper 

We use Glass Mapper in our implementations, and I had no issues after installing the latest version (4.0.5.54 at the time of writing this) from our friend Michael Edwards: http://bit.ly/1lCnwSW

Castle Windsor 

Castle is our DI framework of choice. For some reason, when I got 8.1 up and running, I started getting the following error when running my IOC controllers:

Looks like you forgot to register the http module Castle.MicroKernel.Lifestyle.PerWebRequestLifestyleModule Add '<add name="PerRequestLifestyle" type="Castle.MicroKernel.Lifestyle.PerWebRequestLifestyleModule, Castle.MicroKernel" />' to the section on your web.config

 Adding the following entries to the web.config fixed the issue:

 <system.webServer>  
  <modules>  
   <add name="PerRequestLifestyle" type="Castle.MicroKernel.Lifestyle.PerWebRequestLifestyleModule, Castle.Windsor" />  
  <handlers>  
   <add name="PerRequestLifestyle" verb="*" path="*.castle" preCondition="managedHandler" type="Castle.MicroKernel.Lifestyle.PerWebRequestLifestyleModule, Castle.Microkernel"/>  

I am interested to understand why this has worked via Microsoft.Web.Infrastructure dynamically registering Castle before instead of me having to explicit register Castle in the web.config.

sc_site 

This parameter has always been around, but now, it is added to the url when you are in Preview or Experience Editor mode within your site(s). The problem I am having is that for some reason, it is always set to "website". We know that "website" is special in Sitecore and necessary for features to work correctly in a multi-site setup. But why is sc_site always "website"? I validated all my site configurations. Looks to be an annoying bug.

UPDATE: Thanks to Kamruz for pointing out the new <setting name="Preview.ResolveSite" value="false"/> config value in Sitecore.config. Changing the value to "true" fixed the issue.



Monday, November 16, 2015

How To Fix Sitecore Connector Version Issues After TDS Upgrade

Standard

Background

It is pretty obvious that if you are doing some form of Sitecore development, you are probably using Hedgehog's Team Development for Sitecore (TDS). It makes life so much easier when working on large projects with several team members.

The Upgrade Problem

I am running Visual Studio 2015, and ran into a peculiar problem after updating to the latest version of TDS. I had previously been running 5.1.0.17 and upgraded to version 5.1.0.20.

I was stuck in an endless loop where my connector was showing this error: "Warning: The version of the sitecore connector is from a different version of TDS." An addition, my output window was displaying this message: "Warning: Connector returned version 5.1.0.17. Looking for version 5.1.0.15."



 Warning: Connector returned version 5.1.0.17. Looking for version 5.1.0.15  
 Connection Test Start: 2015-11-05 09:13:54  
 Connection Test Pass: Content downloaded from http://hfedev/sitecore  
 Connection Test Pass: TDS connection test successful  
 Connection Test Message: Connector version local 5.1.0.15 and msbuild 5.1.0.15  
 Connection Test Pass: MSBuild version correct.  
 Connection Test Pass: Sitecore.Kernel.dll and Web.config found beneath G:\Inetpub\wwwroot\HFEDev\Website  
 Connection Test Pass: TDS service files are correctly installed.  
 Warning: Connector returned version 5.1.0.17. Looking for version 5.1.0.15.  
 Warning: Connector returned version 5.1.0.17. Looking for version 5.1.0.15.  
 Connection Test Failure: Failed to complete test.  
 Exception The given key was not present in the dictionary. (KeyNotFoundException):  
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)  
   at HedgehogDevelopment.SitecoreProject.VSIP.Utils.Support.CallServiceWrapper[T](TdsServiceSoapClient client, SitecoreProjectNode project, Func`2 clientCall)  
   at HedgehogDevelopment.SitecoreProject.VSIP.ProjectTests.AccessGuidTest.Execute()  
   at HedgehogDevelopment.SitecoreProject.VSIP.Dialogs.BuildTestDialog.RunTests()  
 Connection Test Finish: 2015-11-05 09:14:02  
This was pretty weird as I had just upgraded from version 5.1.0.17 to 5.1.0.20.

The Path To Solving The Problem

I started out by taking a bit of an invasive approach to fix the problem, as it had worked for me before after I ran into a similar issue. I navigated to my Sitecore instance, and deleted  the _DEV folder (that contains the TdsService.asmx) and then deleted the TDS assemblies from my bin folder. After this, I tried to install the connector again.



Unfortunately, this didn't help fix the issue, and I now had the following error:


I checked to make sure that my Access Guid in my various projects matched what was in the service's web.config, and they were in fact the same:


Getting Closer

After restarting Visual Studio, and trying my Sitecore Connector again, I started getting the same type of error message that I got initially, except now it was showing that it was looking for version 5.1.0.20 which was the version that should have been installed.

 Connection Test Start: 2015-11-05 09:43:44  
 Connection Test Pass: Content downloaded from http://hfedev/sitecore  
 Connection Test Pass: TDS connection test successful  
 Connection Test Message: Connector version local 5.1.0.20 and msbuild 5.1.0.20  
 Connection Test Pass: MSBuild version correct.  
 Connection Test Pass: Sitecore.Kernel.dll and Web.config found beneath G:\Inetpub\wwwroot\HFEDev\Website  
 Connection Test Pass: TDS service files are correctly installed.  
 Warning: Connector returned version 5.1.0.17. Looking for version 5.1.0.20.  
 Warning: Connector returned version 5.1.0.17. Looking for version 5.1.0.20.  
 Connection Test Failure: Failed to complete test.  
 Exception The given key was not present in the dictionary. (KeyNotFoundException):  
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)  
   at HedgehogDevelopment.SitecoreProject.VSIP.Utils.Support.CallServiceWrapper[T](TdsServiceSoapClient client, SitecoreProjectNode project, Func`2 clientCall)  
   at HedgehogDevelopment.SitecoreProject.VSIP.ProjectTests.AccessGuidTest.Execute()  
   at HedgehogDevelopment.SitecoreProject.VSIP.Dialogs.BuildTestDialog.RunTests()  
 Connection Test Finish: 2015-11-05 09:44:02  
The next thing I did was I add a new empty TDS project to my existing solution. Interestingly enough, the connector worked like a champ with no issues:


I then went ahead and tested the connection on the rest of my projects and they all worked too!

Yeah baby!

Problem solved? Well, not exactly.

The Final Solution

After successfully testing the connection on all of my existing TDS projects, I went ahead and deleted the empty TDS project. I am sure you can guess what happened next right? The other projects started failing to connect again!

So, what was different after I added the new TDS project? Mmmm. It had a different Access Guid than the other projects. That was weird though. How would the other projects connect then?

I went ahead and created another new, empty TDS project, configured it, did a test and made sure that everything passed. Then I copied the Access Guid from the new TDS project and replaced the Access Guids in all my projects with the new Guid. 

After doing this, I did a test on all the TDS projects in my solution, and every one of them worked like a champ.

Finally, I went ahead and I removed the empty TDS project. This time, after removing the empty project,  all my connections continued to work perfectly!



Monday, November 9, 2015

Official Fork of the Improved 301 Redirect Module

Standard

After chatting with Chris Adams about my recent contributions to the improved version of the 301 redirect module (Original version was created by Chris Castle, available on the Marketplace), and the launch of version 2.0, we decided that I would be taking over the maintenance of the module. So if you have any questions or run into issues, you can inbox me at menglish@arke.com or reach out to me on Twitter: @MartinREnglish.



The new repository url is: https://github.com/martinrayenglish/301RedirectModule

Some improvements you will see in version 2.0 include:

Ability to handle redirects for non-Sitecore hostnames
  • New setting for pipe separated list of non-Sitecore hostnames that are bound in IIS that you want to allow redirects from. Eg. "mydomain.org|www.myotherdomain.org"
  • New setting to redirect external host names where there is no match found, to a specific URL

Ability to redirect to relative destination paths using the redirect pattern. Eg:
  • Requested URL: http://www.mydomain.org/nice/blog/2015/12/1
  • Regex match for “^/nice/blog/?”
  • Redirect to http://nice-blog.myotherdomain.com/ set
  • Redirect will go to http://nice-blog.myotherdomain.com/nice/blog/2015/12/1

Geolocation Redirects using MaxMind
  • Ability to use MaxMind’s GeoIP2 Database or GeoIP2 Precision Services to determine visitor location, and redirect to specific URLs if a location match based on ISO Country Code or Continent code is found.

Caching layer for Geolocation Redirects
  • This was introduced to improve performance of Geolocation lookups


Hope that community benefits from the improvements!

Friday, October 30, 2015

Fix for Multiple Versions of Items being indexed from the Master Database

Standard

Background

This reared it's head on a project a couple months back - we were finding multiple versions of items that where being indexed from the Master Database.

After looking around on the web for a solution, I came across this post about inbound and outbound filter pipelines by the Sitecore Development Team: http://www.sitecore.net/learn/blogs/technical-blogs/sitecore-7-development-team/posts/2013/04/sitecore-7-inbound-and-outbound-filter-pipelines.aspx

The Disaster Waiting to Happen 

As Owen Wattley noted in his post, implementing the ApplyInboundIndexVersionFilter to ensure only the latest version goes into the index can cause problems that aren't apparent at first glance.

"...The problem my team found is as follows:

  1. Create an item, version 1 goes into the index because it's the latest version 
  2. Add a new version. Version 2 goes into the index because it's now the latest version. 
  3. Version 1 gets blocked by the inbound filter, meaning the index entry for version 1 DOESN'T GET UPDATED OR REMOVED. In the index it is still marked as the latest version. So is version 2. This means you have 2 versions in your index, both marked as the latest version. 



You have to be very careful with inbound filters because they don't do as you might expect. I expected that if you set "args.IsExcluded" to true then it would REMOVE that entry from the index, but it doesn't - it ONLY ensures that nothing gets ADDED. That's a subtle but very crucial difference. Once we found this problem we quickly removed the inbound latest version filter. "

The Solution

Luckily, Sitecore star Pavel Veller recommended a solution that would help alleviate these issues. I just took his idea and implemented the solution.

As this keeps popping up time and time again in the Sitecore 8.x projects that I have been working on, I wanted to share this implementation with the community.

Hope it helps!

FilterPatchItemCrawler.cs


1:  using System;  
2:  using System.Collections.Generic;  
3:    
4:  using Sitecore.ContentSearch;  
5:  using Sitecore.ContentSearch.Abstractions;  
6:  using Sitecore.ContentSearch.Diagnostics;  
7:  using Sitecore.Data.Items;  
8:  using Sitecore.Diagnostics;  
9:  using Sitecore.Globalization;  
10:    
11:  namespace FilterPatch.Library.ContentSearch  
12:  {  
13:    public class FilterPatchItemCrawler : SitecoreItemCrawler  
14:    {  
15:      protected override void DoAdd(IProviderUpdateContext context, SitecoreIndexableItem indexable)  
16:      {  
17:        Assert.ArgumentNotNull((object)context, "context");  
18:        Assert.ArgumentNotNull((object)indexable, "indexable");  
19:    
20:        this.Index.Locator.GetInstance<IEvent>().RaiseEvent("indexing:adding", (object)context.Index.Name, (object)indexable.UniqueId, (object)indexable.AbsolutePath);  
21:        if (this.IsExcludedFromIndex(indexable, false))  
22:          return;  
23:        foreach (Language language in indexable.Item.Languages)  
24:        {  
25:          Item obj1;  
26:          using (new FilterPatchCachesDisabler())  
27:            obj1 = indexable.Item.Database.GetItem(indexable.Item.ID, language, Sitecore.Data.Version.Latest);  
28:          if (obj1 == null)  
29:          {  
30:            CrawlingLog.Log.Warn(string.Format("FilterPatchItemCrawler : AddItem : Could not build document data {0} - Latest version could not be found. Skipping.", (object)indexable.Item.Uri), (Exception)null);  
31:          }  
32:          else  
33:          {  
34:            using (new FilterPatchCachesDisabler())  
35:            {  
36:              SitecoreIndexableItem sitecoreIndexableItem = obj1.Versions.GetLatestVersion();  
37:              IIndexableBuiltinFields indexableBuiltinFields = sitecoreIndexableItem;  
38:              indexableBuiltinFields.IsLatestVersion = indexableBuiltinFields.Version == obj1.Version.Number;  
39:              sitecoreIndexableItem.IndexFieldStorageValueFormatter = context.Index.Configuration.IndexFieldStorageValueFormatter;  
40:    
41:              this.Operations.Add(sitecoreIndexableItem, context, this.index.Configuration);  
42:            }    
43:          }  
44:        }  
45:        this.Index.Locator.GetInstance<IEvent>().RaiseEvent("indexing:added", (object)context.Index.Name, (object)indexable.UniqueId, (object)indexable.AbsolutePath);  
46:      }  
47:    
48:      protected override void DoUpdate(IProviderUpdateContext context, SitecoreIndexableItem indexable)  
49:      {  
50:        Assert.ArgumentNotNull((object)context, "context");  
51:        Assert.ArgumentNotNull((object)indexable, "indexable");  
52:        if (this.IndexUpdateNeedDelete(indexable))  
53:        {  
54:          this.Index.Locator.GetInstance<IEvent>().RaiseEvent("indexing:deleteitem", (object)this.index.Name, (object)indexable.UniqueId, (object)indexable.AbsolutePath);  
55:          this.Operations.Delete((IIndexable)indexable, context);  
56:        }  
57:        else  
58:        {  
59:          this.Index.Locator.GetInstance<IEvent>().RaiseEvent("indexing:updatingitem", (object)this.index.Name, (object)indexable.UniqueId, (object)indexable.AbsolutePath);  
60:          if (!this.IsExcludedFromIndex(indexable, true))  
61:          {  
62:            foreach (Language language in indexable.Item.Languages)  
63:            {  
64:              Item obj1;  
65:              using (new FilterPatchCachesDisabler())  
66:                obj1 = indexable.Item.Database.GetItem(indexable.Item.ID, language, Sitecore.Data.Version.Latest);  
67:              if (obj1 == null)  
68:              {  
69:                CrawlingLog.Log.Warn(string.Format("FilterPatchItemCrawler : Update : Latest version not found for item {0}. Skipping.", (object)indexable.Item.Uri), (Exception)null);  
70:              }  
71:              else  
72:              {  
73:                Item[] versions;  
74:                using (new FilterPatchCachesDisabler())  
75:                  versions = obj1.Versions.GetVersions(false);  
76:                foreach (Item obj2 in versions)  
77:                {  
78:                  SitecoreIndexableItem versionIndexable = PrepareIndexableVersion(obj2, context);  
79:    
80:                  if (obj2.Version.Equals(obj1.Versions.GetLatestVersion().Version))  
81:                  {  
82:                    Operations.Update(versionIndexable, context, context.Index.Configuration);  
83:                    UpdateClones(context, versionIndexable);  
84:                  }  
85:                  else  
86:                  {  
87:                    Index.Locator.GetInstance<IEvent>().RaiseEvent("indexing:deleteitem", (object)this.index.Name, (object)indexable.UniqueId, (object)indexable.AbsolutePath);  
88:                    Operations.Delete(versionIndexable, context);  
89:                  }  
90:                    
91:                }  
92:              }  
93:            }  
94:            this.Index.Locator.GetInstance<IEvent>().RaiseEvent("indexing:updateditem", (object)this.index.Name, (object)indexable.UniqueId, (object)indexable.AbsolutePath);  
95:          }  
96:          if (!this.DocumentOptions.ProcessDependencies)  
97:            return;  
98:          this.Index.Locator.GetInstance<IEvent>().RaiseEvent("indexing:updatedependents", (object)this.index.Name, (object)indexable.UniqueId, (object)indexable.AbsolutePath);  
99:          this.UpdateDependents(context, indexable);  
100:        }  
101:      }  
102:    
103:      private static SitecoreIndexableItem PrepareIndexableVersion(Item item, IProviderUpdateContext context)  
104:      {  
105:        SitecoreIndexableItem sitecoreIndexableItem = (SitecoreIndexableItem)item;  
106:        ((IIndexableBuiltinFields)sitecoreIndexableItem).IsLatestVersion = item.Versions.IsLatestVersion();  
107:        sitecoreIndexableItem.IndexFieldStorageValueFormatter = context.Index.Configuration.IndexFieldStorageValueFormatter;  
108:        return sitecoreIndexableItem;  
109:      }  
110:    
111:      private void UpdateClones(IProviderUpdateContext context, SitecoreIndexableItem versionIndexable)  
112:      {  
113:        IEnumerable<Item> clones;  
114:        using (new FilterPatchCachesDisabler())  
115:          clones = versionIndexable.Item.GetClones(false);  
116:        foreach (Item obj in clones)  
117:        {  
118:          SitecoreIndexableItem sitecoreIndexableItem = PrepareIndexableVersion(obj, context);  
119:          if (!this.IsExcludedFromIndex(obj, false))  
120:            this.Operations.Update((IIndexable)sitecoreIndexableItem, context, context.Index.Configuration);  
121:        }  
122:      }  
123:    }  
124:  }  
125:    

FilterPatchCachesDisabler.cs


1:  using System;  
2:    
3:  using Sitecore.Common;  
4:  using Sitecore.ContentSearch.Utilities;  
5:  using Sitecore.Data;  
6:    
7:  namespace FilterPatch.Library.ContentSearch  
8:  {  
9:    public class FilterPatchCachesDisabler : IDisposable  
10:    {  
11:      public FilterPatchCachesDisabler()  
12:      {  
13:        Switcher<bool, DatabaseCacheDisabler>.Enter(ContentSearchConfigurationSettings.DisableDatabaseCaches);  
14:      }  
15:    
16:      public void Dispose()  
17:      {  
18:        Switcher<bool, DatabaseCacheDisabler>.Exit();  
19:      }  
20:    }  
21:  }  
22:    

Your index configuration:


1:  <locations hint="list:AddCrawler">  
2:         <crawler type="FilterPatch.Library.ContentSearch.FilterPatchItemCrawler, FilterPatch.Library">  
3:          <Database>master</Database>  
4:          <Root>#Some path#</Root>  
5:         </crawler>  
6:  </locations>  

Monday, October 19, 2015

Sharing Razl Connections between User Profiles

Standard

Background

This is a quick post to demonstrate how you can share your Hedgehog Razl connections between various users who are logging into the same machine to sync content between environments.

I have been using Razl for quite some time, and can't say enough good things about the product. As Nikola Gotsev put is in his blog post, it is jewel that comes in handy to easily move the content between your various environments: http://sitecorecorner.com/2014/10/27/the-amazing-world-of-razl-part-1/.



Setting Up Connections

Our client purchased a copy of the software, and loaded it on their Staging server. We then had a developer set up the connections to each Sitecore instance in the various environments.

Within a few minutes, he could connect and sync content between 3 different environments with a few button clicks.

The Shared Connection Problem

The problem we were faced with was that non of the other developers could see the connections that the first developer had set up under his profile. Would we have to get each developer to set up the connections individually?

The Shared Connection Solution

I found the solution on Razl's FAQ page: http://hedgehogdevelopment.github.io/razl/faq.html

If you navigate to C:\Users\{username}\AppData\Local\Hedgehog_Development\Razl.exe_Url_????, you will see a user.config file that contains all the connection information.






So, to get the connections to show up for a new developer's profile, this is what you need to do:

  1. Each user needs to run Razl once. This will create the "Hedgehog_Development\Razl.exe_Url_????" folder structure and user.config file in the location mentioned above.
  2. Get a copy of the user.config of the developer that initially set up the connections and replace the file in each user's C:\Users\{username}\AppData\Local\Hedgehog_Development\Razl.exe_Url_???? location.
After this, when you they fire up Razl, the connections will show up.


Monday, September 14, 2015

How to Fix Missing Conditions in Engagement Plan Designer After Upgrading to Sitecore 8.x

Standard

Then and Now

Over the course of the last year, we upgraded a client's Sitecore instance from 6.6 to 8.0 Update-2.  I am pretty sure that we were probably the first in the country, if not the world, to get our client running on xDB Cloud.

There were definitely some bumps in the road, but things have been running pretty smoothly for us, and we have been starting to really see the benefits of the new experience platform.



Look Ma, No Conditions!

It wasn't until recently, when we started developing some pretty fancy engagement plans, that we discovered a new issue - when trying to set rules on Engagement Plan conditions, the dialog was completely empty!



Detective Mode Enabled

Lars and I started digging around the tree, and we discovered a legacy Marketing Workflows folder living in the rules location: /sitecore/system/Settings/Rules.


Digging deeper, we discovered that the Engagement Plan Condition Rule's Source field was pointing to one of the Marketing Workflows folders located at /sitecore/system/Settings/Rules.


We knew that we were onto something here, so I checked the links of the legacy Marketing Workflows folder. Presto! I discovered that the Engagement Plan Condition Rule's Source field was linked to it.



Out With the Old, In With the New

At this point, I decided to take an invasive approach by deleting the legacy folder. When I clicked the delete button, the Breaking Links dialog popped up. I selected "Link to another item" in the action options as shown below:



Next, I pointed the item that was linked to it to the proper Marketing Workflows folder:



Lift Off!

After changing the link to point to the proper Marketing Workflows folder, and removing the old folder, the rule set editor dialog was working again:


Thanks to Lars for helping get this issue resolved.

Monday, August 31, 2015

Setting up Sitecore's Geolocation Lookup Services in a Production Environment

Standard

Background

We have been working with Sitecore’s Business Optimization Services (SBOS) team for quite some time, helping one of our client's stretch the legs of the Experience Platform.

One of the tasks on the list included setting up Sitecore's Geolocation Service so that we could personalize based on the visitor's location. The SBOS team had some pretty slick rules set up on the Home Page of a site, where they switched out carousel slides and marketing content spots based on the visitor's location.


Sitecore Geolocation Lookup Service and MaxMind

There have been some changes with regards to the Geolocation / MaxMind set up because Sitecore launched IP Geolocation directly to their customers via the Sitecore App Center instead of going through MaxMind to use their service. Here is a link to Sitecore's documentation site that contains set up instructions, and how to migrate from MaxMind if purchased and have been working directly with them in the past:
https://doc.sitecore.net/Sitecore%20Experience%20Platform/Analytics/Setting%20up%20Sitecore%20IP%20Geolocation

Setup

After sifting through the documentation, we highlighted the following steps that needed to be completed in order to get up and running, and validate that things were indeed working:

  1. Download and install the Geolocation client package from https://dev.sitecore.net/Downloads.aspx
  2. Enable ALL configuration files in the CES folder
  3. Whitelist geoIp-ces.cloud.sitecore.net and discovery-ces.cloud.sitecore.net
  4. Test personalization
NOTE: Depending on your firewall, you may only have the option to whitelist by IP address. If this applies to you, you will need obtain the list of Azure IP addresses from the following link: Azure Datacenter IP Address Ranges. This is what happened to us, and I call tell you that the list is looooooooooooooong!!! Your network guy or gal will hate you!

Unfortunately for us, there was one piece of configuration that isn't documented. It so happened to be one of the most important pieces.

You will know what I am referring to as you read further along.

Testing

With all the pieces in place, we started testing our personalization.

Things worked beautiful on our Staging Server, but Production was a non-starter! So, as good detectives, we started our troubleshooting by looking at the differences between Staging and Production.

Load Balancer / Firewall / CDN woes

Our client uses Incapsula to protect their production websites. It does a great job protecting and caching their various site's to ensure optimal performance. It has however given us some grey hairs in the past when dealing with Sitecore's Federated Experience Manager. But, that's a story for another time.

The Incapsula CDN was the main difference between Staging and Production.

After running several tests with Fiddler and capturing packets using Wireshark, we were able to the determine that the Geolocation service was not obtaining the visitor's actual IP address. Instead, it was passing along Incapsula's IP address.

The reason for this was identified in the CreateVisitProcessor within the CreateVisits analytics pipeline. As you can see below, it was passing over the Request.UserHostAddress value.


This doesn't work when you are behind a load balance or proxy, as described by this article: http://stackoverflow.com/questions/200527/userhostaddress-gives-wrong-ips

Digging further, we discovered another interesting processor in the CreateVisits pipeline called XForwardedFor. Aha! As we know; the "...header field is a de facto standard for identifying the originating IP address of a client connecting to a web server through an HTTP proxy or load balancer."

Looking at the code below, you will notice that it's pulling in a setting, and if it's not empty, it is used as the key to obtain the value from the request's header NameValueCollection.



After digging around, and talking to support, we discovered the config file named Sitecore.Analytics.Tracking.config and setting below:

 <!-- ANALYTICS FORWARDED REQUEST HTTP HEADER  
    Specifies the name of an HTTP header variable containing the IP address of the webclient.  
    Only for use behind load-balancers that mask web client IP addresses from webservers.  
    IMPORTANT: If this setting is used incorrectly, it allows IP address spoofing.  
    Typical values are "X-Forwarded-For" and "X-Real-IP".  
    Default value: "" (disabled)  
 -->  
 <setting name="Analytics.ForwardedRequestHttpHeader" value="" />  

Light at the end of the tunnel

After setting the value to "X-Forwarded-For" as shown below, the magical Geolocation based personalization started working like a champ!

<setting name="Analytics.ForwardedRequestHttpHeader" value="X-Forwarded-For" />

NOTE: We discovered that casing matters when setting the value. "X-FORWARDED-FOR" will NOT work. It needs to be set exactly like I have it above. For more information on this, you can read this Stack Overflow article:
http://stackoverflow.com/questions/11616964/is-request-headersheader-name-in-asp-net-case-sensitive


I hope that this information helps make your Sitecore IP Geolocation configuration go smoothly for your environment!

A special thanks to Kyle Heon from Sitecore for his support through this process.