Serializing Custom .Net Types for use with the Azure Redis Cache

In my previous post I looked at a real-world example of using the new Azure Redis Cache. One thing that was missing was the storing of custom .Net Types as cache values, which we’ll look at here.

The RedisValue Type

The Microsoft Azure documentation recommends using the StackExchange.Redis API for accessing the Redis Cache. This API stores cache values within the RedisValue type, which has a number of implcit conversions from primitive .Net types (Int32, Int64, Double, Boolean etc.), but conversions from the RedisValue type to the primitive .Net type need to be made explicitily.

For example, setting an int as the cache value is implicit, while retrieving an int from a cache value needs to be cast (the following screenshot is from the MSDN documentation):

Cache Serializer - Get and Set of Primitive Types

In addition to the implicit/explicit conversion of primitive data types, the RedisValue type can store binary data as the cache value. This means that we can serialize a custom .Net type into a byte array and pass that as the cache value within the StringSet() method; we can then retrieve that cache value using the StringGet() method and deserialize the byte array back to the custom type.

To complete the serialization/deserialization, we need a helper class, which is shown below. This class is inspired by the sample on MSDN (Work with .Net objects in the cache) but is tweaked slightly:

Note that during deserialization, if the byte array is null, a default instance of type T will be returned. This caught me out when I was initially testing, so beware that it is returning the value you expect.

Using these helpers with the StackExchange.Redis API’s StringSet() and StringGet() methods to store custom .Net types makes life really easy:

To retrieve a custom type, we call the StringGet() method, passing the cache Key and deserialize the returned byte array into the .Net Guid type; In order to determine whether the cache item was found, I check whether the returned cacheItem equals Guid.Empty, which is the value returned by default(T) for the Guid type from the Deserialize() method.

To store a custom .Net type (in this case a GUID), call the StringSet() method and instead of passing a primitive type as the cache value, we serialize our cache item and pass the resulting byte array.

Azure Redis Cache – A Real World Example

Read the second post in this series: Serializing Custom .Net Types for use with the Azure Redis Cache.

I spend a lot of my time at the moment architecting and writing integration code against Dynamics CRM 2011 and 2013. My current project is based on Azure Worker Roles and sucks CRUD messages from an SFTP Server (yep, FTP), maps them to CRM Entities before finally calling the appropriate CRUD operation on the XRM Service Proxy with the new Entity instance.

I need to repeatedly retrieve an Entity Reference to common ‘reference’ data on Create and Update – data that is loaded once and hardly ever changes (such as Products, Agent Codes, Cost Centres etc.) At the moment, I’m hitting CRM each and every time I need to grab the Entity Reference, which isn’t particularly efficient, slows down the overall solution and will start to incur unnecessary Azure data-egress charges at some point in the life-cycle of the application.

Azure Redis Cache

The Azure Redis Cache is the new cache offering from Microsoft which supersedes the Managed Service Cache (an instance of the AppFabric Cache) and In-Role Cache (a cache based on Cloud Services).

The Azure Redis Cache is based on the open-source Redis cache, an advanced key-value cache and store. More information about the project can be found at

The Azure offering comes in two flavours: Basic (a single-node deployment, primarily for dev/test) and Standard (a replicated cache in a two-node primary/secondary configuration, with automatic replication between the two nodes (managed by the Azure Fabric) and a high-availability SLA, primarily for production use). For the purposes of this blog-post I am using a Basic instance.

A Real World Example

I use an Entity Reference Provider in my code which implements a bunch of simple methods to retrieve the Guid of the Entity I’m interested in – in the example below, I am retrieving the Guid to the Product Entity:

RetrieveProductEntityReference() Method - Pre Cache Implementation

In order to cache-ify this method using the Azure Redis Cache we first need to spin-up  a new instance of the Redis Cache in the Azure Portal (see Getting Started with Azure Redis Cache in the Azure Documentation for further information).

We then create a connection to the Redis Cache using the ConnectionMultiplexer.Connect() method. Be aware that calling this method is expensive and the resulting object should be re-used thoughout your code, hence why I create an instance in the constructor for the provider:

Entity Reference Provider - Constructor

That’s all the setup required to use the Azure Redis Cache and we can now concentrate on retrieving and storing values in our cache. At a high level, the approach will be:

  • Attempt to retrieve a Guid for the referenced Entity from the cache based on a specified cache key.
  • If nothing is found in the cache for the key, we will retrieve the value from XRM itself and then populate the cache with that value.
  • We will finally return the Guid (retrieved either from the Cache or XRM).

The full implementation of the reworked RetrieveProductEntityReference() method with a cache implementation is shown below.

Note that I am keeping things simple here and converting my Guid to a string and vice-a-versa (I plan on showing how to cache custom types in a separate post).

With a connection to the Cache we need to retrieve a reference to the cache database. The GetDatabase() method is a lightweight method call and can be performed as required:

RetrieveProductEntityReference - Breakdown-1

We then define a cache key and attempt to get the item from the cache that has that key:

RetrieveProductEntityReference - Breakdown-2

If the cache item is null, there was no item in the cache with the supplied key (either it has never been set, or a previous cache item with that key has expired).

RetrieveProductEntityReference - Breakdown-3

If that is the case, we go off and retrieve the actual Entity Guid from XRM and assign the Guid to the (previously defined) cacheItem:

RetrieveProductEntityReference - Breakdown-4

We then add the cache item (containing the newly retrieved Guid) to the cache using the cache.StringSet() method, passing the previously defined cache key, item and a timespan which defines when the item will expire in the cache (the expiry timespan is optional):

Finally, we return the Guid contained within the cache item which has either been retrieved directly from XRM or the cache itself:

RetrieveProductEntityReference - Breakdown-6

A Performance Improvement?

Just to give you an idea of numbers, using a very small test sample of 50 iterations over this method, retrieving from the cache takes (on average) 32ms, while hitting the XRM Service Proxy takes 209ms (we are hitting Dynamics CRM 2013 Online).

Putting this into context, if I didn’t use the cache, it would take me 10.45 seconds to perform 50 iterations of this method, vs. 1.8 seconds for the cache implementation (which includes the initial hit to retrieve the data from XRM).

RESTfully GETting JSON Formatted Data with BizTalk 2013

With the new WCF-WebHttp Adapter that shipped in BizTalk Server 2013 we now have the ability to specify which HTTP verbs to use – this opens up the possibility of invoking RESTful style applications.

In this post, I’ll investigate how we can RESTfully invoke an endpoint that exposes data via JSON using the GET verb – we’ll be ‘RESTfully GETting JSON Formatted Data with BizTalk 2013′; I’ll be retrieving stock tickers from Yahoo! Finance in JSON format.

Send/Receive Adapter

The RESTful endpoint is invoked via a solicit/response Send Port configured to use the WCF-WebHttp Adapter, providing the Yahoo! Finance URL and a single HTTP method of ‘GET':

WCF-WebHttp Adapter Configuration


JSON Decode Receive Pipeline Component

The JSON response is passed to a custom Receive Pipeline via the adapter which contains a newly developed ‘JsonDecode’ component. This component invokes the fantastic Json.Net package (see for more information) to do the required heavy lifting and convert the JSON into an Xml representation. The Execute() method of the Pipeline Component looks something like this:

JSON Decode Pipeline Component Source Code

Once we have an Xml representation of the JSON response, we add a Root Node (and associated namespace) so we can leverage our usual magic BizTalk sauce. The root node name and namespace are exposed as properties via the Pipeline Component (think WCF-SQL Adapter):

JSON Decode Pipeline Component Configuration

Simply create a schema that represents your new message and voilà, you’re receiving JSON data and processing it as an Xml message within BizTalk; you’ll also have access to all of the usual property promotion/routing capabilities etc.

JSON in, Xml Out

Now that we have configured our solution, what do the input and output messages look like?

Something like this for the JSON response to our GET request:

JSON Response Data

and our corresponding Xml representation:

JSON Data as Xml Message to be Consumed by BizTalk

Feel free to download a copy of the solution if you’re interested –> Please note that the sample code is for demonstration purpose, and it is NOT production quality code.

Stay tuned for Part 2, where I will look at POSTing JSON data from BizTalk.


Beware of windows.old when upgrading to Windows 8

I have just taken the plunge and upgraded from Windows 7 to Windows 8 (partly because of the £25 upgrade offer). After the upgrade I noticed a severe lack of space on my otherwise roomy C: drive.

A quick check in Windows Explorer reveals a windows.old directory which contains all of my old Windows 7 ‘Windows’ directory, Program Files, Users and PerfLogs directories – and its just short of 20Gb. I’ll be shifting it to an external backup drive shortly…


Including Custom Databases in the Backup BizTalk Server Job

I’m sure its common knowledge that other databases can be backed-up using the Backup BizTalk Server Job, meaning that the transaction consistency that is that is written the the BizTalk databases can be included in your own custom databases.

I thought it was pretty easy to include custom databases, simply by adding an entry into the adm_OtherBackupDatabases table in the BizTalkMgmtDb database. In the screenshot below, I’ve added the BizTalk RosettaNet Accelerator (BTARN) databases:

I thought it was that simple, until I tried it for the first time in real life and was greeted with the following error from SQL Agent (notice the error in bold):

Executed as user: NT AUTHORITYSYSTEM. Processed 17520 pages for database ‘BizTalkDTADb’, file ‘BizTalkDTADb’ on file 1. [SQLSTATE 01000] (Message 4035)  Processed 2 pages for database ‘BizTalkDTADb’, file ‘BizTalkDTADb_log’ on file 1. [SQLSTATE 01000] (Message 4035)  BACKUP DATABASE successfully processed 17522 pages in 10.454 seconds (13.093 MB/sec). [SQLSTATE 01000] (Message 3014)  Processed 2264 pages for database ‘BizTalkMgmtDb’, file ‘BizTalkMgmtDb’ on file 1. [SQLSTATE 01000] (Message 4035)  Processed 3 pages for database ‘BizTalkMgmtDb’, file ‘BizTalkMgmtDb_log’ on file 1. [SQLSTATE 01000] (Message 4035)  BACKUP DATABASE successfully processed 2267 pages in 2.284 seconds (7.751 MB/sec). [SQLSTATE 01000] (Message 3014)  Processed 3280 pages for database ‘BizTalkMsgBoxDb’, file ‘BizTalkMsgBoxDb’ on file 1. [SQLSTATE 01000] (Message 4035)  Processed 2 pages for database ‘BizTalkMsgBoxDb’, file ‘BizTalkMsgBoxDb_log’ on file 1. [SQLSTATE 01000] (Message 4035)  BACKUP DATABASE successfully processed 3282 pages in 2.156 seconds (11.890 MB/sec). [SQLSTATE 01000] (Message 3014)  Processed 256 pages for database ‘BizTalkRuleEngineDb’, file ‘BizTalkRuleEngineDb’ on file 1. [SQLSTATE 01000] (Message 4035)  Processed 1 pages for database ‘BizTalkRuleEngineDb’, file ‘BizTalkRuleEngineDb_log’ on file 1. [SQLSTATE 01000] (Message 4035) Could not find stored procedure ‘BTARNARCHIVE.dbo.sp_BackupBizTalkFull’. [SQLSTATE 42000] (Error 2812) BACKUP DATABASE successfully processed 257 pages in 0.326 seconds (6.158 MB/sec). [SQLSTATE 01000] (Error 3014).  The step failed.

‘Could not find stored procedure ‘BTARNARCHIVE.dbo.sp_BackupBizTalkFull’. Oh really?

It turns out that it isn’t as simple as adding a custom database to the adm_OtherBackupDatabases table – a number of stored procedures and tables also need to be added to your custom database to get the backup job to successfully work, however these can be easily added by executing the two SQL scripts Backup_Setup_All_Procs.sql and Backup_Setup_All_Tables.sql against your custom databases (the scripts can be found in the ‘[BizTalk Installation Directory]Schema’ directory). You may want to force a full backup to ensure you have full backups across all of your databases:

UPDATE [BizTalkMgmtDb].[dbo].[adm_ForceFullBackup] SET ForceFull = 1

Re-run your backup job and you will see your custom databases included in the backup.

The requirement to include additional stored procedures and tables in custom databases is well worth remembering, especially if you are deploying (or re-deploying) custom databases that don’t include these artifacts by default.

The procedure is detailed in more depth on MSDN.

BTARN ‘Service Content’ Error in the RNDisAssembler Component

We came across the following error late last night which was a bit of a show-stopper. We were trying to load a custom PIP (specifically a PIDX OrderChange), but kept hitting this issue time and time again:

Event Type: Error
Event Source: BizTalk Accelerator for RosettaNet
Event Category: None
Event ID: 4096
Source module:
Correlation information:
Receive pipeline rejected incoming message
due to the following RNIF exception:
UNP.SCON.VALERR : A failure occurred while validating the service content.
Event Type: ErrorEvent Source: BizTalk Accelerator for RosettaNetEvent Category: NoneEvent ID: 4096Date: 18/08/2010Time: 10:33:47User: N/AComputer: [COMPUTER NAME]Description:Source module:RNDisAssembler
Correlation information:


Receive pipeline rejected incoming message due to the following RNIF exception: UNP.SCON.VALERR : A failure occurred while validating the service content.

At first I didn’t quite understand the cause of the error – the PIDX OrderChange message contained within the Service Content was valid (as far as I was aware), all of the other messages within the payload looked correct and it was 3am….

It turns out that the RNDisassembler does in-fact attempt to validate the message contained within the Service Content against a deployed schema just like the standard XmlDisassembler. The message that our trading partner was sending did not validate and hence the RosettaNet Accelerator threw this error message; once we had corrected the schema and redeployed, the error went away.

This is certainly one to be aware of if you are developing custom PIP’s to use with the RosettaNet Accelerator: ensure that the message in the Service Content validates against your custom PIP schema!

SCOM can cause Unchecked Growth in SSODB

Ok, that’s a bit of an attention grabbing headline, so let me clarify that statement: SCOM can cause unchecked growth in SSODB if you’re not regularly backing up the SSODB transaction log.

We encountered this one today – a client’s SSODB ran out of space overnight, causing the BizTalk environment to shut-down. On further investigation, it would appear that every time SCOM checks the health of Enterprise Single Sign-On, an entry is recorded in the SSOX_AuditXpLookup table:

ESSO appears to be clever enough to manage the size of this table, truncating it every 30 minutes, however this doesn’t help if you’re not managing the size of the database transaction log through backups. To quote the SQL Server documentation:

If log records were never deleted from the transaction log, it would eventually fill all the disk space that is available to the physical log files. Log truncation automatically frees space in the logical log for reuse by the transaction log.

All the more reason to enable and run the supplied Backup BizTalk Server job to help maintain the health of your BizTalk environment.

If you’re looking for more information on the Backup BizTalk Server Job, take a look at my series of posts on the topic:

Hat-tip to this MSDN Forums posting that helped us identify the issue.