Wednesday, October 7, 2015

Azure - Blob storage – Working with file(s)

Introduction
Blob storage is an azure service to store large size/amount of data in unstructured way in form of binaries in the cloud. This opens up the whole world of files and all their extensions which can be uploaded in the blobs.
This post does not talk about the creation of blobs, connections but explains step by step in uploading the files, downloading the files and moving the files from one blob to another.

Upload file
The first step in the process of uploading a file to Blob is to create Objects of the CloudStorageAccount and the BlobClients in it.

// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(UnityConfiguration.GetConfigurationValue("ConfigKey"));

// Create the blob client object from the storage account
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

You may want to add Retry policy to this blobClient which will retry to access the client in an exponential time if there is any problem encountered in the first try
blobClient.DefaultRequestOptions.RetryPolicy = new ExponentialRetry(TimeSpan.FromSeconds(retryInterval), retryCount);

It is always a best practice to check if the Blob exists with a container, this way you can throw an exception that blob does not exists or create one with the container reference.

// Returns an reference for a given container of the blob
CloudBlobContainer blobContainer = blobClient.GetContainerReference(
UnityConfiguration.GetConfigurationValue("ContainerName"));

// Create container if it doesn’t exists
blobContainer.CreateIfNotExists();


Now that we have completed all the pre-requisites, the next step is to create reference of the CloudBlockBlob and then upload the file

// Create the reference of the block blob from the container
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference(this.AddLocationInfo("Blobname"));

using (var inputStream = new MemoryStream())
{
     document.Save(inputStream); // document is a file object(could be xml, image, etc)
     inputStream.Flush();
     inputStream.Position = 0;

     blob.UploadFromStream(inputStream);
}

Note:
    a.  For XmlDocument document -> document.Save(inputStream)
    b.  For other type of files (.docx, .xlsx, etc) need to save into a stream so that it can be uploaded to blob.

Download file
Downloading a file is simple, you can download a file content into a Stream or as a string or even as a byte array.
The CloudBlockBlob object has method to download the file

// Download the file as memory stream
using (var blobStream = blockBlob.OpenRead())
{
      MemoryStream contentStream = new MemoryStream();
      blobStream.CopyTo(contentStream);
}

// Download the file as string
Encoding encoding;
blockBlob.DownloadText(encoding);

// Download the file into byte array
byte[] content;
using (var memoryStream = new MemoryStream())
{
     blockBlob.DownloadToStream(memoryStream);
     content = memoryStream.ToArray();
}


Moving file from one Blob to another
To move a file from one blob to another, simply use the inbuilt method of the CloudBlockBlob object.

CloudBlockBlob source;
CloudBlockBlob target;

using (var sourceStream = source.OpenRead())
{
     target.UploadFromStream(sourceStream);
}


Please leave your feedback and queries in the comments section. Thank you!

Thursday, September 24, 2015

Compare Object List – Determine Insert, Update and Delete data


I was working on this scenario and breaking my head to come up with simple solution to compare two large list of object (List<object>) and get the result as new (Insert), update and delete data. Found out that there is simple solution and I thought will publish it so that it can help others.

I have put generic code which anyone can copy and start using right away.

New data
/// <summary>
/// Returns the newly added
/// </summary>
/// <param name="existingData">Existing data</param>
/// <param name="newData">New data</param>
/// <returns>List of objects which are newly added</returns>
public List<object> GetInsertData(List<object> existingData, List<object> newData)
{
    // Local objects
    List<object> returnList = null;

    // Prepare the list which has the data to be inserted
    returnList = (from New in newData
                  join Existing in existingData on New.ID equals Existing.ID
                  into DataAdded
                  from dataOld in DataAdded.DefaultIfEmpty()
                  where dataOld == null
                  select New).ToList<object>();

    // Return the data list
    return returnList;
}


Updated data
/// <summary>
/// Returns the data updated
/// </summary>
/// <param name="existingData">Existing data</param>
/// <param name="newData">New data</param>
/// <returns>List of objects which are updated</returns>
public List<object> GetUpdateData(List<object> existingData, List<object> newData)
{
    // Local objects
    List<object> returnList = null;

    returnList = (from New in newData
                  join Existing in existingData on New.ID equals Existing.ID
                  into DataUpdated
                  from dataOld in DataUpdated.DefaultIfEmpty()
                  where dataOld!= null && (New.<Field1>!= dataOld.<Field1> || New.<Field2> != dataOld.<Field2>)
                  select new object
                  {
                       Field1 = New.Value1,
                        Field2 = New.Value2,
                        Field3 = dataOld.Value3,
                        Field4 = dataOld.Value4,
                   }).ToList<object>();

            // Return the data list
            return returnList;
        }


Delete data
/// <summary>
/// Returns the deleted rows
/// </summary>
/// <param name="existingData">Existing data</param>
/// <param name="newData">New data</param>
/// <returns>List of schools which needs to be deleted</returns>
public List<object> GetDeleteData(List<object> existingData, List<object> newData)
{
    // Local objects
    List<object> returnList = null;

    // Prepare the list which has the data to be deleted
    returnList = (from Existing in existingData
                     join New in newData on Existing.ID equals New.ID
                     into NewWithNullForDeleted
                     from subNew in NewWithNullForDeleted.DefaultIfEmpty()
                     where subNew == null
                     select Existing).ToList<School>();

     // Return the data list
     return returnList;
}



Please leave your feedback and queries in the comments section. Thank you!

Tuesday, August 25, 2015

Azure – Application Insights – Configuration & Usage


I still remember the good old days where we had a huge instrumentation database wherein developers would log all the data – URLs, user navigation path, query parameters, exceptions, browser information, region information, etc to help in debugging the issues and generating required reports. Application Insights offers all these capabilities and much more with simple configuration and ready to use detailed reports.

This post is divided into two parts –
a.      Installation & configuration
b.      Usage of the reports 

Installation & configuration –

Prerequisites
·        Visual Studio 2013 or later version(s)
·        Subscription to Microsoft Azure
Visual Studio Application Insights has Free, Standard & Premium pricing tiers. 

             1.      Login to Azure portal and create a new Application Insight resource by selecting ASP.Net Web Application



2.      Make a note of the Instrumentation key
            
            Note: Intentionally erased the details 

These first two steps creates a resource and now it is ready for any application to be configured for monitoring.

3.      Install the SDK 

a.      Right click on your Project and install ApplicationInsights NuGet package




b.      Search for Microsoft.ApplicationInsights.Web and install


               
Note: If “Application Insights for Visual Studio” extension is installed, then on right click of the project you will see an option “Add Application Insights” which is another way of configuring Application Insights.


c.      ApplicationInsights.config file will be added by the NuGet installation. Edit this file and add the below tag just before the file end.

<InstrumentationKey>Your instrumentation key here</InstrumentationKey>

 Note: This above configuration in the ApplicationInsights.config will write the logistics from all the available environments. If the requirement is to configure different Application Insights for each environment then the best solution is to configure the Instrumentation Key in Cloud.config and write the below statement in Global.asax.cs to fetch the key based on environment.

Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.Active.InstrumentationKey = CloudConfigurationManager.GetSetting("InstrumentKeyName");
At this point we are done with the installation & configuration. Running the project (debug mode/release mode/published code) will start writing the logs.

 Usage of the reports –

          1.      Login to Azure portal and browse to Applications Insight.
 



2.      Click through the applications and then through the available reports to see more details.



Various reports are already created based on the ApplicationInsights data log –
·        Failures

This reports all the failures with exception details happened in last 24 hours along with giving additional details about the % of each exception against the total number of exceptions logged.

One great thing about this report –

It digs down from the URL to method name to operating system to even browser version for a generated exception. This is like a whole universe of data points for debugging the issue. This detailed information has helped our team more than once to find the root cause sooner.  

·        Performance

Gives performance number against each URL which will help in determining the low performance pages.

·        Browser

Gives the details about the browser & their versions used to access the application.

This post will not get into the details of each of the report. I request the readers to explore other reports & options.
Add Client side monitoring (Web browsers)

               The SDK installed from NuGet sends the monitoring data from server side code. To add monitoring on client side for web browsers,  
a.      Download the Javascript snippet from Azure portal and put it in every page or the Master Page.

The code snippet will have the Instrumentation Key. 
Please leave your feedback and queries in the comments section. Thank you!

Monday, August 10, 2015

Azure – Best practices, Learnings & Performance tips


Azure in many ways is simple to code and work with but sometimes it gets tricky. Remember when you are stuck with some problem and are searching all over the internet to find solution and wished there was a consolidated list where you can quickly read all the standards, learnings from other people, tips and find all the answers to queries ranging from best practices to performance improvements.

This document tries to put all this under one umbrella with work related to Azure table storage skipping the basic things like connections, coding, etc and it assumes that the reader has good understanding of Azure table storage. 

Best practices

a.      Create a table storage helper class for all the operations that can be performed on the Table storage (Insert/Update/Delete/Get/etc). All the classes in Data Access Layer should call these helper methods to perform any/all operations on Azure tables. This will help in maintaining code.

b.      Data comparisons in Azure are case sensitive, make sure that you address this in all the applicable places. The best/standard way of doing this is to use .ToLowerInvariant(), probably at both inserting & reading the entities from the table storage.

Learnings

a.      Azure table storage Inserts/Updates ie .Insert() or.InsertOrMerge() can sometimes return Bad Request without giving additional details. This exception being very generic, developers try to search on the internet for the solutions with less to no success. I have come up with most common mistakes which developers do which are responsible for these Bad Requests.

·     DateTime

Most common source for Bad Request. Never send DateTime.MinValue to Azure table. DateTime.MinValue is not within the supported DateTime range.

·     DataType

Second most common source. See that the data types of the values being sent to the Azure table are matched with Azure table storage’s data type.

b.      When using .ExecuteQuerySegmented() for scenario of fetching more than 1000 records or pagination, keep it in mind that the table service API will either return maximum of 1000 records or as much as entities as possible in 5 seconds. If the query takes more than this time for execution, it will return with empty entities and a continuation token.

Performance tips

a.      I am sure we all know that it is always faster & a good practice to fetch the data from table storage based on PartitionKey and RowKey but

-        Never combine two or more Partition Keys for fetching records in a query, this results into a full table scan 

Wrong way of doing entities fetch
TableQuery.CombineFilters(
                                TableQuery.GenerateFilterCondition(PartitionKey, QueryComparisons.Equal, value1),
TableOperators.And/Or,
                                TableQuery.GenerateFilterConditionForInt(PartitionKey, QueryComparisons.Equal, value2)
);

 In this scenario, spin up tasks for each PartitionKey filter condition, wait for the tasks to complete and perform operations on top of it. Please find below an example on how this can be achieved.

Correct way of doing entities fetch
var PartitionKeyQuery1 = TableQuery.GenerateFilterCondition(PartitionKey, QueryComparisons.Equal, value1);
var PartitionKeyQuery2 = TableQuery.GenerateFilterCondition(PartitionKey, QueryComparisons.Equal, value2);
var customerTaskA = Task.Run<IEnumerable<CustomerDataEntity>>(() => this.customerDataTable.ExecuteQuery(PartitionKeyQuery1));
                   
var customerTaskB = Task.Run<IEnumerable<CustomerDataEntity>>(() => this. customerDataTable.ExecuteQuery(PartitionKeyQuery2));
// If one or more customer(s) exists for given conditions then process the data
if (customerTaskA.Result != null || customerTaskB.Result != null)
{
   . . .
   . . .
}

Note: When a Task with .Result is called, it internally waits for the task to complete and then executes .Result attribute of the Task.

b.      Try to have the results from Azure table storage in IEnumerable object as long as you can in the code. If there is any filtering, sorting, etc to be applied on the data, use LINQ and perform these operations on IEnumerable object. When you are done with these operations then call .ToList() method of the IEnumerable object. The line of code where you call .ToList() is when it actually executes the Query. This is called Deferred Execution.

So the next time when IEnumerable<Object>.ToList() is taking lot of time or is a performance hit, don’t look for any other places but at the Query which is getting executed.
 This post will be updated periodically with new learnings. Thank you!

*********************************************************************************

Update - 8/21/2015

Learnings 
             
           ·     Unsupported characters
Below characters are not supported for PartitionKey and RowKeys –
i.                 The forward slash (/) character
ii.                Backward slash (\) character
iii.               Number sign (#) character
iv.               Question mark (?) character
v.                Control characters(\t, \n, \r, etc)