Setting up Sitecore's Geolocation Lookup Services in a Production Environment

Background

We have been working with Sitecore’s Business Optimization Services (SBOS) team for quite some time, helping one of our client's stretch the legs of the Experience Platform.

One of the tasks on the list included setting up Sitecore's Geolocation Service so that we could personalize based on the visitor's location. The SBOS team had some pretty slick rules set up on the Home Page of a site, where they switched out carousel slides and marketing content spots based on the visitor's location.

Sitecore Geolocation Lookup Service and MaxMind

There have been some changes with regards to the Geolocation / MaxMind set up because Sitecore launched IP Geolocation directly to their customers via the Sitecore App Center instead of going through MaxMind to use their service. Here is a link to Sitecore's documentation site that contains set up instructions, and how to migrate from MaxMind if purchased and have been working directly with them in the past:
https://doc.sitecore.net/Sitecore%20Experience%20Platform/Analytics/Setting%20up%20Sitecore%20IP%20Geolocation

Setup

After sifting through the documentation, we highlighted the following steps that needed to be completed in order to get up and running, and validate that things were indeed working:

1. Download and install the Geolocation client package from https://dev.sitecore.net/Downloads.aspx
2. Enable ALL configuration files in the CES folder
3. Whitelist geoIp-ces.cloud.sitecore.net and discovery-ces.cloud.sitecore.net
4. Test personalization

NOTE: Depending on your firewall, you may only have the option to whitelist by IP address. If this applies to you, you will need obtain the list of Azure IP addresses from the following link: Azure Datacenter IP Address Ranges. This is what happened to us, and I call tell you that the list is looooooooooooooong!!! Your network guy or gal will hate you!

Unfortunately for us, there was one piece of configuration that isn't documented. It so happened to be one of the most important pieces.

You will know what I am referring to as you read further along.

Testing

With all the pieces in place, we started testing our personalization.

Things worked beautiful on our Staging Server, but Production was a non-starter! So, as good detectives, we started our troubleshooting by looking at the differences between Staging and Production.

Load Balancer / Firewall / CDN woes

Our client uses Incapsula to protect their production websites. It does a great job protecting and caching their various site's to ensure optimal performance. It has however given us some grey hairs in the past when dealing with Sitecore's Federated Experience Manager. But, that's a story for another time.

The Incapsula CDN was the main difference between Staging and Production.

After running several tests with Fiddler and capturing packets using Wireshark, we were able to the determine that the Geolocation service was not obtaining the visitor's actual IP address. Instead, it was passing along Incapsula's IP address.

The reason for this was identified in the CreateVisitProcessor within the CreateVisits analytics pipeline. As you can see below, it was passing over the Request.UserHostAddress value.

This doesn't work when you are behind a load balance or proxy, as described by this article: http://stackoverflow.com/questions/200527/userhostaddress-gives-wrong-ips

Digging further, we discovered another interesting processor in the CreateVisits pipeline called XForwardedFor. Aha! As we know; the "...header field is a de facto standard for identifying the originating IP address of a client connecting to a web server through an HTTP proxy or load balancer."

Looking at the code below, you will notice that it's pulling in a setting, and if it's not empty, it is used as the key to obtain the value from the request's header NameValueCollection.

After digging around, and talking to support, we discovered the config file named Sitecore.Analytics.Tracking.config and setting below:

 <!-- ANALYTICS FORWARDED REQUEST HTTP HEADER  
    Specifies the name of an HTTP header variable containing the IP address of the webclient.  
    Only for use behind load-balancers that mask web client IP addresses from webservers.  
    IMPORTANT: If this setting is used incorrectly, it allows IP address spoofing.  
    Typical values are "X-Forwarded-For" and "X-Real-IP".  
    Default value: "" (disabled)  
 -->  
 <setting name="Analytics.ForwardedRequestHttpHeader" value="" />  

Light at the end of the tunnel

After setting the value to "X-Forwarded-For" as shown below, the magical Geolocation based personalization started working like a champ!

<setting name="Analytics.ForwardedRequestHttpHeader" value="X-Forwarded-For" />

NOTE: We discovered that casing matters when setting the value. "X-FORWARDED-FOR" will NOT work. It needs to be set exactly like I have it above. For more information on this, you can read this Stack Overflow article:
http://stackoverflow.com/questions/11616964/is-request-headersheader-name-in-asp-net-case-sensitive


I hope that this information helps make your Sitecore IP Geolocation configuration go smoothly for your environment!

A special thanks to Kyle Heon from Sitecore for his support through this process.

Read More

Mastering Sitecore MVC Forms : CRUD that works on both Management and Delivery servers

The Use Case

This is a very common scenario; you need to build a form that has the ability to create or update items in Sitecore after a user has input data and clicked the submit button.

This sounds pretty straightforward right? Well, it gets a bit tricky when you don't have direct access to the master database. For example, if you are trying to perform CRUD operations on a security hardened content delivery server, you don't have access to the master database, and this will fail.

Most folks will create a new web service on their management server that they could consume to handle this. A good option, but something extra to build and maintain.

In this post, I am going to show you how to make your form logic "smart" enough to be able to create or update items even when they are posting from content delivery servers.

The Form

Let's start out by looking at a simple form where we ask a user to input some basic information so that they can receive a reminder about an event. We intend to use this data to create an item in the master database so that a scheduled task running in the management instance will determine when to email them to remind them about the event.

The Model

The model is pretty simple. As you can tell, the class' properties almost match the form's input.

1:  public class ReminderViewModel : ModelBase<Item>  
2:  {  
3:      public ReminderStrings FormStringsAndLinks { get; set; }  
4:      public Event RemindEvent { get; set; }  
5:      public string EventId { get; set; }  
6:      public string FirstName { get; set; }  
7:      public string LastName { get; set; }  
8:      public string Email { get; set; }  
9:      public bool Consent { get; set; }  
10: }  

FormsStringsAndLinks and ReminderEvent are complex properties used to display some form input strings, links and the event information as shown above.

The Client Script

We are using AJAX to post our form. The following JavaScript snippet is an AngularJS / jQuery mix where there is a validation check on the form after the user clicks the schedule reminder button. If it's valid, we build a view model object, and post it to the SubmitReminder action in my Forms controller.

1:  if (isValid) {  
2:        var reminderViewModel = {};  
3:        reminderViewModel.firstname = $scope.txtreminderfirstname;  
4:        reminderViewModel.lastname = $scope.txtreminderlastname;  
5:        reminderViewModel.email = $scope.txtreminderemail;  
6:        reminderViewModel.eventId = jQuery("#hdnEventId").val();//Angular doesn't like hidden fields with guids  
7:        reminderViewModel.consent = $scope.chkreminderconsent ? $scope.chkreminderconsent : false;  
8:        $http.post("/api/mysite/forms/SubmitReminder", reminderViewModel )  
9:          .success(function () {  
10:            jQuery("#reminder-app").slideUp("slow");  
11:            jQuery("#reminder-set").removeAttr("style");  
12:          })  
13:          .error(function () {  
14:            alert("An error occurred while adding you to our reminder list.");  
15:          });  
16:      }  

The Submit Reminder Action

The action method takes the model object, and then passes it to a business layer method in line 5. If it is successful, it will use some key values to identify the contact in xDB. The xDB piece is beyond the scope of this post.

1:  [HttpPost]  
2:      public bool SubmitReminder(ReminderViewModel reminderViewModel)  
3:      {  
4:        var reminderManager = new ReminderManager();  
5:        var success = reminderManager.CreateEventReminder(reminderViewModel);  
6:        if (success)  
7:        {  
8:          var identifiedContact = new IdentifyContact  
9:          {  
10:            FirstName = reminderViewModel.FirstName,  
11:            LastName = reminderViewModel.LastName,  
12:            Email = reminderViewModel.Email  
13:          };  
14:          ExperienceProfile.ContactIdentifier = reminderViewModel.Email;  
15:          ExperienceProfile.UpdateProfile(identifiedContact);  
16:        }  
17:        return success;  
18:      }  

Nothing special happening in the business layer method. It is strictly a wrapper around the repository in this case.

1:  public bool CreateEventReminder(ReminderViewModel reminderViewModel)  
2:      {  
3:        return _eventRepository.CreateEventReminder(reminderViewModel);  
4:      }  

Handling the CRUD

Config Settings

In our set up, we have an element where we define the application instance type, default database and the url for the management instance.

We also have other config nodes that help to dynamically switch the Solr or Lucene indexes that we are querying against based on the environment we have set. We won't get too much into this, but it's basically a static configuration class that we access within our query methods.

1:  <mysite.com  
2:     applicationInstance="management"  
3:     defaultDomain="mysite.com"  
4:     defaultDatabase="master"  
5:     siteName=""  
6:     managementInstanceUrl="http://cm.mysite.com">  
7:     <indexes>  
8:      <index name="buckets" management="sitecore_master_index" delivery="sitecore_web_index" />  
9:      <index name="products" management="commerce_products_master_index" delivery="commerce_products_web_index" />  
10:     <index name="media" management="scene7_media_master_index" delivery="scene7_media_web_index" />  
11:    </indexes>  
12: </mysite.com>  

On a delivery server, the applicationInstance would be set to "delivery" and the defaultDatabase to "web".

Create Event Reminder

The first method that gets accessed in our repository is CreateEventReminder, as shown above in the quick snippet from the business code.

1:  public bool CreateEventReminder(ReminderViewModel reminderViewModel)  
2:      {    
3:        return Settings.ApplicationInstance == InstanceType.Management ? CreateEventReminderMaster(reminderViewModel) : PostEventToManagement(reminderViewModel);  
4:      }  

Line 3 is a ternary that determines if we are on the delivery or management instance, and either passes the model object to the CreateEventReminderMaster method if we are on the management server, or to the PostEventToManagement method if we are on the delivery server.

Creating the Item in the Master Database

The CreateEventReminderMaster method below, as it's name indicates, creates items in the master database. There is a check to determine if the item already exists in the database, before creating it. Nothing really fancy here.

1:  public bool CreateEventReminderMaster(ReminderViewModel reminderViewModel)  
2:      {  
3:        //We are on the management server  
4:        var masterService = new SitecoreService("master");  
5:        var destinationFolder = masterService.Database.GetItem(new ID(Sitecore.Configuration.Settings.GetSetting("EventReminderFolderID")));  
6:        if (destinationFolder == null)  
7:        {  
8:          return false;  
9:        }  
10:        var itemName = ItemUtil.ProposeValidItemName(reminderViewModel.Email + reminderViewModel.EventId);  
11:        if (!EventReminderExists((itemName)))  
12:        {  
13:          if (!EventReminderExistsFallback((itemName)))  
14:          {  
15:            var newUser = new Event_Reminder  
16:            {  
17:              Name = itemName,  
18:              First_Name = reminderViewModel.FirstName,  
19:              Last_Name = reminderViewModel.LastName,  
20:              Email = reminderViewModel.Email,  
21:              Communications = reminderViewModel.Consent,  
22:              Event = new Guid(reminderViewModel.EventId)  
23:            };  
24:            using (new SecurityDisabler())  
25:            {  
26:              masterService.Create(destinationFolder, newUser);  
27:            }  
28:          }  
29:        }  
30:        return true;  
31:      }  

This is an example of an item that has been created:

Posting Data From the Delivery to the Management Server

This is the magic method.

As it turns out, it's quite simple. All we are doing here is posting the model object back over to the exact same controller and action on the management server. You can see on line 4 where I am building the url to the controller's action.

On line 6, I created a clean DTO so that I didn't run into any issues when serializing my object to JSON.

Finally, I used the WebClient class' UploadData method to post my model back over to the management server as shown on line 14.

1:  public bool PostEventToManagement(ReminderViewModel reminderViewModel)  
2:      {  
3:        //This will get executed if you are on a delivery (CD) server  
4:        var controllerUrl = $"{Settings.ManagementInstanceUrl}/api/mysite/forms/SubmitReminder";  
5:        //Create a clean DTO Object to send over  
6:        var reminderDTO = new {reminderViewModel.FirstName, reminderViewModel.LastName, reminderViewModel.Email, reminderViewModel.EventId, reminderViewModel.Consent};  
7:        using (var client = new WebClient())  
8:        {  
9:          client.Headers["Content-type"] = "application/json";  
10:          // serializing the reminderViewModel object in JSON notation  
11:          var jss = new JavaScriptSerializer();  
12:          var json = jss.Serialize(reminderDTO);  
13:          //Post to controller on cm  
14:          client.UploadData(controllerUrl, "POST", Encoding.UTF8.GetBytes(json));  
15:          return true;  
16:        }  
17:      }  

Summary

With these pieces in place, we can summarize the events that follow after a user has completed their input and clicks the submit button:

Management Server Instance

If the setting indicates that we are on the management server, the object is passed to the CreateEventReminderMaster method where it creates the item in the master database.

Delivery Server Instance

If the setting indicates that we are on the delivery server, the object is passed to the PostEventToManagement method where is posts the model over to the management server where the process starts again.

Once on the management server, the object ends up in the CreateEventReminderMaster method where the item is created in the master database.

Final Thoughts

I hope that you find this concept useful as you start working on your own custom MVC forms that require you to make changes to the master database from all instances.

Please share your thoughts and comments.

Happy Sitecoring!

Read More