Performance degradation – Debugging existing Azure API/App

“Performance issue” is every developer’s nightmare and this is very true when all your components are in cloud. I recently had to go through this tedious journey of figuring out what happened to our existing Web API connected to Azure Table storage that it started taking 20-30 secs to return a response instead of a second.

It is obvious that when an existing Web API starts acting weird and slows down, the analysis to be done is in different direction. It includes but not limited to (sudden increase in) load, memory leaks, CPU utilization, etc. My journey starts from here and goes through different Support teams of Azure, learning things in-between. I have put down these learnings and hopefully someone will benefit from this –

a.       Increase in load/hits

Let’s check if there is significant increase in the load/hits to your API/App compared to the days when it was working as expected. If so, start thinking about increasing the number of instances or implementing the Traffic Manager to distribute the load, etc.

b.       Memory leaks & CPU utilization checks

This should be the next area to check, Azure portal provides these details and are self-explanatory with detailed graphical notations. These are available under “Diagnose and solve problems” section. Sample of one such graph below –

Fig: Memory Usage

Fig: CPU Utilization

If these graphs indicate a spike, start looking at API/App code and fix the faulty code.

c.       NAT limitation check

What is NAT?

                Most often APIs/Apps do have to make outbound calls to other endpoints, these include calls to the Azure SQL DB and/or Azure Storages along with the scenarios of calling other applications over http/https, for example, calling search API or calling one of your other API which implements core logic for your application. In all these cases, the calling API/App is implicitly opening a network socket and making outbound calls. All such calls made from an API/App on Azure App Service to a remote endpoint rely on Azure Networking to set up and manage a table of Network Address Translation (aka NAT) mappings.

Creating & removing the entries in this NAT mapping takes time and there is limit on the total number of NAT mappings that can be established for a single Azure App Service and thus, App Service limits the number of outbound connections that can be outstanding at any given point in time. The maximum connections limits depend on the App Service pricing tier. Details about the pricing tier and its allowed maximum connections can be found below (please do check the latest number on Azure’s website)

-          1,920 connections per B1/S1/P1 instance

-          3,968 connections per B2/S2/P2 instance

-          8,064 connections per B3/S3/P3 instance

-          64K max upper limit per App Service Environment

Based on these details check what is your API/App’s maximum outbound connections at any time and compare it with what your App Service limit. This could be the reason for performance issue as the additional connection requests are put in the queue and processed in the order or are failed intermittently.

The most important point to consider and look for in the code is to check for any “leaky” connections that invariably run into these connection limits. Remember, it is always a good practice to close the connection(s) explicitly after receiving the response.

Other best practices to avoid this situation –

-         Use database connection pooling

-  For making outbound HTTP/HTTPS calls, pool and reuse instances of System.Net.Http.HttpClient or use Keep-alive connections with System.Net.HttpWebRequest.

Note: Remember to increase the System.Net.ServicePointManager.DefaultConnectionLimit because you’ll otherwise be limited to two concurrent outbound connections to the same endpoint.

                Check if your application is under the limit.

d.       Enable Table storage tracing & analysis

-          Enable table storage tracing, the requests made to table service

-  Then download logs from $Logs blob container with Storage Explorer www.storageexplorer.com. The logs will show every request made to storage, including client and server latency and content size, etc.

Note: This cannot be viewed/downloaded from Visual Studio’s Cloud Explorer.

-          Details about the format and how to read these logs can be found here.

e.       Throttling

a.       Table storage APIs

This is a long shot but very much possible. Just like our custom APIs/App services, table storage APIs scale up and down based on the traffic. There is a possibility that table storage APIs were getting throttle at that moment and the APIs were busy in scaling up.

You will be surprised to hear this but, in my case, this was the reason for slow performance where the scaling up of Table Storage APIs had problem and took between 1-2 hours to get up and running. Till then, it was throwing 503 ServerBusy ServerPartitionRequestThrottlingError. I had to raise ticket with Azure Storage team to get this confirmation.

b.       Your API/App

Check if your API/App is getting throttled to cause this performance degradation. If so, this is time to start thinking about implementing throttling logic.

I hope this helped you in finding the reason for your APIs/App’s performance degradation. If you think there are things to be considered, please put them in the comments section and I will include them. Thank you.

References

  -    https://msdn.microsoft.com/en-us/magazine/mt793270.aspx?f=255&MSPPError=-2147217396

Web jobs vs Worker role

      Web jobs and worker roles are used to create and run background tasks.

      Below are the differences between them –

 

Web job	Worker role
Can be configured to run either OnDemand or Continuous or Scheduled	Explicit code needs to be written to configure the run
Deployed on a shared VM	Has a dedicated VM
Cannot be remote desktop but Azure does provide tools with restricted ability to manage the folders required for the deployed web job.	Can be remote desktop when RDP is enabled.
Cheaper	Relatively expensive
Configuration is limited	Has all the set of configurations
Scalable from the Web App which is hosting it	Scalable
Simple to code, test & deploy	Relatively complex

   It is recommended to use Worker role when there are process intense & high resource(s) utilization requirements else for light weight executions use Web jobs.

      Web jobs

     Web jobs are recommended for running lightweight background tasks which requires minimum configuration or customization and less turnaround time to develop & deploy.

      Some of the acceptable file types for web jobs to configure as background tasks

·        .cmd

·        .bat

·        .exe

·        .ps1

Below are the scenarios where web jobs can be perfect fit

·        Data archiving

·        (Asynchronous) logging

·        Polling

Worker roles

Worker roles are recommended for running heavy weight, process intense & high resource(s) utilization.

Worker Roles are hosted on a standalone virtual machine which offers more control to the operating environment. 

Below are the scenarios where worker roles can be perfect fir

·        Startup tasks – to prepare environment to suit requirements

·        Import jobs

·        Background tasks that need to be scaled independently.

·        Sending notifications

File upload in Resource (.resx)

I was working on Azure web job wherein I had to put an excel in the solution and use it after getting deployed. I was struggling with finding the path of the file, tried with “~”, “..\” and all other options suggested over the internet to vain. Later I was introduced to uploading a file in the .resx and fetching/accessing the file just like any other resource.

Below is the step by step procedure for doing so –

a.      Add .resx file in your solution, if not already

b.      Instead of Strings, select Files for upload

c.      Click on Add Resource, select All Files and add file. Save the resx. You will see the uploaded file in the solution under Resource folder

In the below screenshot, you will see that I have added .xlsx file.

d.      The next most important step here is to right click on the file which we have uploaded and click on Properties, set the “Build Action” to “Embedded Resource”. This step will make sure that the executing assembly has the information of this file.

e.      Now the next step would be to access this file in the code. Below is the code for the same,

Assembly assembly = Assembly.GetExecutingAssembly(); Stream st = assembly.GetManifestResourceStream("<SolutionName>.Resources.<FileNameWithExtension>");             if (st.CanSeek) {   st.Position = 0; }

Note:

a.      The GetManifestResourceStream method needs an exact path of the file in the form of a hierarchy ie SolutionName.Resources.FileName.FileNameExtension

The data in the uploaded file is now available in the stream. We can now use it as per our need.

Please leave your feedback and queries in the comments section. Thank you!

Azure - Blob storage – Working with file(s)

Introduction

Blob storage is an azure service to store large size/amount of data in unstructured way in form of binaries in the cloud. This opens up the whole world of files and all their extensions which can be uploaded in the blobs.

This post does not talk about the creation of blobs, connections but explains step by step in uploading the files, downloading the files and moving the files from one blob to another.

Upload file

The first step in the process of uploading a file to Blob is to create Objects of the CloudStorageAccount and the BlobClients in it.

// Retrieve storage account from connection string. CloudStorageAccount storageAccount = CloudStorageAccount.Parse(UnityConfiguration.GetConfigurationValue("ConfigKey")); // Create the blob client object from the storage account CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

You may want to add Retry policy to this blobClient which will retry to access the client in an exponential time if there is any problem encountered in the first try

blobClient.DefaultRequestOptions.RetryPolicy = new ExponentialRetry(TimeSpan.FromSeconds(retryInterval), retryCount);

It is always a best practice to check if the Blob exists with a container, this way you can throw an exception that blob does not exists or create one with the container reference.

// Returns an reference for a given container of the blob CloudBlobContainer blobContainer = blobClient.GetContainerReference( UnityConfiguration.GetConfigurationValue("ContainerName")); // Create container if it doesn’t exists blobContainer.CreateIfNotExists();

Now that we have completed all the pre-requisites, the next step is to create reference of the CloudBlockBlob and then upload the file

// Create the reference of the block blob from the container CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference(this.AddLocationInfo("Blobname"));

using (var inputStream = new MemoryStream()) {      document.Save(inputStream); // document is a file object(could be xml, image, etc)      inputStream.Flush();      inputStream.Position = 0;      blob.UploadFromStream(inputStream); } Note:     a.  For XmlDocument document -> document.Save(inputStream)     b.  For other type of files (.docx, .xlsx, etc) need to save into a stream so that it can be uploaded to blob.

Download file

Downloading a file is simple, you can download a file content into a Stream or as a string or even as a byte array.

The CloudBlockBlob object has method to download the file

// Download the file as memory stream using (var blobStream = blockBlob.OpenRead()) {       MemoryStream contentStream = new MemoryStream();       blobStream.CopyTo(contentStream); }

// Download the file as string Encoding encoding; blockBlob.DownloadText(encoding);

// Download the file into byte array byte[] content; using (var memoryStream = new MemoryStream()) {      blockBlob.DownloadToStream(memoryStream);      content = memoryStream.ToArray(); }

Moving file from one Blob to another

To move a file from one blob to another, simply use the inbuilt method of the CloudBlockBlob object.

CloudBlockBlob source; CloudBlockBlob target; using (var sourceStream = source.OpenRead()) {      target.UploadFromStream(sourceStream); }

Please leave your feedback and queries in the comments section. Thank you!

Compare Object List – Determine Insert, Update and Delete data

I was working on this scenario and breaking my head to come up with simple solution to compare two large list of object (List<object>) and get the result as new (Insert), update and delete data. Found out that there is simple solution and I thought will publish it so that it can help others.

I have put generic code which anyone can copy and start using right away.

New data

/// <summary> /// Returns the newly added /// </summary> /// <param name="existingData">Existing data</param> /// <param name="newData">New data</param> /// <returns>List of objects which are newly added</returns> public List<object> GetInsertData(List<object> existingData, List<object> newData) {     // Local objects     List<object> returnList = null;     // Prepare the list which has the data to be inserted     returnList = (from New in newData                   join Existing in existingData on New.ID equals Existing.ID                   into DataAdded                   from dataOld in DataAdded.DefaultIfEmpty()                   where dataOld == null                   select New).ToList<object>();     // Return the data list     return returnList; }

Updated data

/// <summary> /// Returns the data updated /// </summary> /// <param name="existingData">Existing data</param> /// <param name="newData">New data</param> /// <returns>List of objects which are updated</returns> public List<object> GetUpdateData(List<object> existingData, List<object> newData) {     // Local objects     List<object> returnList = null;     returnList = (from New in newData                   join Existing in existingData on New.ID equals Existing.ID                   into DataUpdated                   from dataOld in DataUpdated.DefaultIfEmpty()                   where dataOld!= null && (New.<Field1>!= dataOld.<Field1> || New.<Field2> != dataOld.<Field2>)                   select new object                   {                        Field1 = New.Value1,                         Field2 = New.Value2,                         Field3 = dataOld.Value3,                         Field4 = dataOld.Value4,                    }).ToList<object>();             // Return the data list             return returnList;         }

Delete data

/// <summary> /// Returns the deleted rows /// </summary> /// <param name="existingData">Existing data</param> /// <param name="newData">New data</param> /// <returns>List of schools which needs to be deleted</returns> public List<object> GetDeleteData(List<object> existingData, List<object> newData) {     // Local objects     List<object> returnList = null;     // Prepare the list which has the data to be deleted     returnList = (from Existing in existingData                      join New in newData on Existing.ID equals New.ID                      into NewWithNullForDeleted                      from subNew in NewWithNullForDeleted.DefaultIfEmpty()                      where subNew == null                      select Existing).ToList<School>();      // Return the data list      return returnList; }

Please leave your feedback and queries in the comments section. Thank you!