Software/System Design

Azure Cosmos DB – TTL (Time to Live) – Reference Usecase

October 9, 2018 .NET, .NET Core, .NET Framework, Analytics, Architecture, Azure, Azure, Azure Cosmos DB, Azure Functions, Azure IoT Suite, Cloud Computing, Cold Path Analytics, CosmosDB, Emerging Technologies, Hot Path Analytics, Intelligent Cloud, Intelligent Edge, IoT Edge, IoT Hub, Microsoft, Realtime Analytics, Visual Studio 2017, VisualStudio, VS2017, Windows No comments

TTL capability within Azure Cosmos DB is a live saver, as it would take necessary steps to purge redudent data based on the configurations you may. 

Let us think in terms of an Industrial IoT scenario, devices can produce vast amounts of telemetry information, logs and user session information that is only useful until we operate on them and take action on them, to be specific up to finate period of time. Once that data becomes surplus, we need an application logic that purges these old records.

With the “Time to Live” or TTL, Microsoft Cosmos DB provides an ability to have your documents automatically purged from database storage after a certian period if time(which you configured)

  • This TTL by default can be set on a document collection level and later can be overridden on a per document basis.
  • Once the TTL is set, Cosmos DB service will automatically remove the documents when its lifetime is over.
  • Inorder to track TTL, Cosmos DB uses an offset field to check when it was last modified.  This field is identifiable as “_ts”, which exists in every document you create.  Basically it is a UNIX epoch timestamp. This field is updated everytime when the document is modified. [Ref: Picture1]

image

[Picture1]

Enabling TTL on Cosmos DB Collection:

You can enable TTL on a Cosmos DB collection simply by using Azure Portal –> Cosmos DB collection setting for existing or during creation of  a new collection)

TTL value needs to be set in seconds – if you need 90 days => 60 sec * 60 min * 24 hour * 90 days = 7776000 seconds

image

[Picture2]

Below is a one of the reference architecture in which Cosmos DB – TTL would be essentially useful and viable to any Iot business case:

image

[Picture3]

Hope that was helpful to get some understanding. For more references visit:  Cosmos DB Documentation

Azure Cosmos DB–Multi Master

October 8, 2018 .NET, .NET Core, .NET Framework, ASP.NET, Azure, Azure CLI, Azure Cosmos DB, CosmosDB, Data Consistancy, Data Integrity, Microsoft, Multi-master, Performance, Reliability, Resilliancy, Scalability, Scale Up No comments

During the Ignite 2018, Microsoft has announced the general availability of Multi-Master feature being introduced to Azure Cosmos DB to provide more control into data redundancy and elastic scalability for your data from different regions with multiple writes and read instances.

What is Multi-Master essentially?

Multi-master is a capability that provided as part of Cosmos DB, that would provide you multiple write regions and provides an option to handle conflict resolution automatically through different options provided by the platform. Most of the major scenarios you would encounter the conflict can be resolved with these simple configurations.

A sample diagram depicting a use case of load balanced web app writing to respective regional master:-

image

With multi-master, Azure Cosmos DB delivers a single digit millisecond write latency at the 99th percentile anywhere in the world, and now offers 99.999 percent write availability (in addition to 99.999 percent read availability) backed by the industry-leading SLAs.

image

Wow! That’s an amazing performance Cosmos DB guarantees to provide so that your mission-critical systems will have zero downtime, if they start using Cosmos DB.

 

How to Enabled Multi-Master support in your Cosmos DB solutions?

Currently multi-master can only be enabled for new Cosmos DB instances using “Enable Multi-Master” option in Azure Portal or through PowerShell or ARM templates or through SDK.

These options are detailed below with necessary examples:

1.) Azure Portal – Enable Multi-region writes and Enable geo-redundancy

image

2.) Azure CLI 
Set the “enable-multiple-write-locations” parameter to “true”

az cosmosdb create \
   –-name "thingx-cosmosdb-dev" \
   --resource-group "consmosify-dev" \
   --default-consistency-level "Session" \
   --enable-automatic-failover "true" \
   --locations "EastUS=0" "WestUS=1" \
   --enable-multiple-write-locations true \

3.) AzureRM PowerShell
In AzureRM PowerShell cmdlet – Set enableMultipleWriteLocations parameter to “true”

$locations = @(@{"locationName"="East US"; "failoverPriority"=0},
             @{"locationName"="West US"; "failoverPriority"=1})

$iprangefilter = ""

$consistencyPolicy = @{"defaultConsistencyLevel"="Session";
                       "maxIntervalInSeconds"= "10";
                       "maxStalenessPrefix"="200"}

$CosmosDBProperties = @{"databaseAccountOfferType"="Standard";
                        "locations"=$locations;
                        "consistencyPolicy"=$consistencyPolicy;
                        "ipRangeFilter"=$iprangefilter;
                        "enableMultipleWriteLocations"="true"}

New-AzureRmResource -ResourceType "Microsoft.DocumentDb/databaseAccounts" `
  -ApiVersion "2015-04-08" `
  -ResourceGroupName "consmosify-dev" `
  -Location "East US" `
  -Name "thingx-cosmosdb-dev" `
  -Properties $CosmosDBProperties

4.) Through CosmosDB SDK
Setting connection policy in DocumentDBClient and set UseMultipleWriteLocations to true.

ConnectionPolicy policy = new ConnectionPolicy
{
   ConnectionMode = ConnectionMode.Direct,
   ConnectionProtocol = Protocol.Tcp,
   UseMultipleWriteLocations = true,
};
policy.PreferredLocations.Add("East US");
policy.PreferredLocations.Add("West US");
policy.PreferredLocations.Add("West Europe");
policy.PreferredLocations.Add("North Europe");
policy.PreferredLocations.Add("Southeast Asia");
policy.PreferredLocations.Add("Japan East");
policy.PreferredLocations.Add("Japan West");

Azure Cosmos DB multi-master configuration is the game changes that really makes it a true global scale database with automatic conflict resolution capabilities for data synchronization and consistancy.

In my later sessions I will write examples to cover how conflict resolutions can be configured and used in realtime scenarios.

Useful Refs:

Azure Cosmos DB – 429 Too Many Requests

October 6, 2018 .NET, Azure, CosmosDB, Document DB, Microsoft, Performance, Reliability, Resilliancy, Scalability, Visual Studio 2017, VisualStudio, VS2017 No comments

Recently while I was doing Performance Testing in one of the APIs interacting with Cosmos DB, I encountered a problem as Azure Cosmos DB API’s started returning Http Code 429.  Http Status Code 429 indicates that too many request been received or request rate is very large. This error would happen when we have concurrent users trying to write or read from same cosmos db collection.

Following diagram covers the architecture of the performance test I am performing:

image

Based on analysis it found out to be the Throttling happening from Azure Cosmos DB, as we make requests that may use more than provisioned Request Units(RU) per second. We were using default Cosmos DB configuration for a fixed collection of 1000 RU’s per second which is sufficient enough for a 500 reads and 100 writes for a 1 kb file. You can refer more about Request Units from Azure Docs.

image

 

 

 

Solution(s):

1. Now first logical step we can do is to get rid off this error by increasing the Throughput for the collection.  I am going to increase to 10000 RU/s maximum allocatable for a Storage Capacity: Fixed.   This should ideally improve the Throughput for 250 or more virtual users hitting.

image

2. Second logical step is to improve the code: Improve the connection parameters in the Document DB SDK –> DocumentDbClient. For this I referred to the Microsoft Docs: Performance tips for Azure Cosmos DB and .NET

Providing optimum values to the following Properties in RetryOption class   to be passed as parameter to Connection Policy.

image

 

In my case I provided a value of 30 to give ultimate results:

new RetryOptions() { MaxRetryAttemptsOnThrottledRequests = 30, MaxRetryWaitTimeInSeconds = 30  }

That should resolve most of the 429 issues when dealing with Cosmos DB SDK

Enterprise Architecture

August 12, 2018 Architectures, Software/System Design, TOGAF No comments

What is an Enterprise Architecture?

In this modern world there is lots of confusion about enterprise architecture, in that sense I would write a short scribble about enterprise architecture or called as EA in short.

I will start with a definition by Architecture and Governance Magazine, Issue 9-4, November (2013) :

Enterprise architecture (EA) is “a well-defined practice for conducting enterprise analysis, design, planning, and implementation, using a comprehensive approach at all times, for the successful development and execution of strategy.

Enterprise architecture applies architecture principles and practices to guide organizations through the business, information, process, and technology changes necessary to execute their strategies.

These practices utilize the various aspects of an enterprise to identify, motivate, and achieve these changes.”

Now that sums up as EA is putting together a practice for translating business goals and strategy into practical enterprise to  Business Process + Information Systems (Data and Applications)  and Technologies within an “Enterprise”.  This also conclude a desired state of the enterprise and facilitate towards its change.

How do you achieve that? Answer is short, through Architecture Governance through a selection of desired Architecture Framework. That makes EA an essential practice that at any organizational level to have all the systems functioning as desired.

Goals of enterprise architecture are:

  1. Effectiveness
  2. Efficiency
  3. Agility
  4. Durability.

Subsets/Layers of enterprise architecture or Architecture domain:

There are four kinds of “architecture” that are commonly accepted as subsets of a well defined Enterprise Architecture system:

  1. Business Architecture:
  2. Data Architecture:
  3. Application Architecture:
  4. Technology Architecture:

image

NIST Enterprise Architecture Model initiated in 1989, one of the earliest frameworks for enterprise architecture. (Courtesy: Wikipedia.)

What is an Enterprise Architecture Framework?

An enterprise architecture framework (EA framework) defines how to create and use an enterprise architecture.

As per Wikipedia there are countless EA frameworks and some of them are categorized as below: (Courtesy: Wikipedia.)

Consortia-developed frameworks:
  • ARCON – A Reference Architecture for Collaborative Networks – not focused on a single enterprise but rather on networks of enterprises.
  • Generalised Enterprise Reference Architecture and Methodology (GERAM)
  • RM-ODP – the Reference Model of Open Distributed Processing (ITU-T Rec. X.901-X.904 | ISO/IEC 10746) defines an enterprise architecture framework for structuring the specifications of open distributed systems.
  • IDEAS Group – a four-nation effort to develop a common ontology for architecture interoperability
  • ISO 19439 Framework for enterprise modelling
  • TOGAF – The Open Group Architecture Framework – a widely used framework including an architectural Development Method and standards for describing various types of architecture.

Defence industry frameworks:

  • AGATE – the France DGA Architecture Framework
  • DNDAF – the DND/CF Architecture Framework (CAN)
  • DoDAF – the US Department of Defense Architecture Framework
  • MODAF – the UK Ministry of Defence Architecture Framework
  • NAF – the NATO Architecture Framework

Government frameworks:

  • European Space Agency Architectural Framework (ESAAF) – a framework for European space-based Systems of Systems
  • Government Enterprise Architecture (GEA) – a common framework legislated for use by departments of the Queensland Government
  • FDIC Enterprise Architecture Framework
  • Federal Enterprise Architecture Framework (FEAF) – a framework produced in 1999 by the US Federal CIO Council for use within the US Government (not to be confused with the 2002 Federal Enterprise Architecture (FEA) guidance on categorizing and grouping IT investments, issued by the US Federal Office of Management and Budget)
  • Nederlandse Overheid Referential Architecture (NORA) – a reference framework from the Dutch Government E-overhead NORA
  • NIST Enterprise Architecture Model
  • Treasury Enterprise Architecture Framework (TEAF) – a framework for treasury, published by the US Department of the Treasury in July 2000.

Open-source frameworks:

Enterprise architecture frameworks that are released as open source:

  • MEGAF is an infrastructure for realizing architecture frameworks that conform to the definition of architecture framework provided in ISO/IEC/IEEE 42010.
  • Praxeme, an open enterprise methodology, contains an enterprise architecture framework called the Enterprise System Topology (EST)
  • TRAK – a general systems-oriented framework based on MODAF 1.2 and released under GPL/GFDL.
  • SABSA is an open framework and methodology for Enterprise Security Architecture and Service Management, that is risk based and focuses on integrating security into business and IT management.

Proprietary frameworks:

  • ASSIMPLER Framework – an architecture framework, based on the work of Mandar Vanarse at Wipro in 2002
  • Avancier Methods (AM) Processes and documentation advice for enterprise and solution architects, supported by training and certification.
  • BRM (Build-Run-Manage) Framework – an architecture framework created by Sanjeev “Sunny” Mishra during his early days at IBM in 2000.
  • Capgemini Integrated Architecture Framework (IAF) – from Capgemini company in 1993
  • Dragon1 – An open Visual Enterprise Architecture Method recently recognized by The Open Group as Architecture Framework
  • DYA framework developed by Sogeti since 2004.
  • Dynamic Enterprise Enterprise architecture concept based on Web 2.0 technology
  • Extended Enterprise Architecture Framework – from Institute For Enterprise Architecture Developments in 2003
  • EACOE Framework  – an Enterprise Architecture framework, as an elaboration of the work of John Zachman
  • IBM Information FrameWork (IFW) – conceived by Roger Evernden in 1996
  • Pragmatic Enterprise Architecture Framework (PEAF) – part of Pragmatic Family of Frameworks developed by Kevin Lee Smith, Pragmatic EA, from 2008
  • Purdue Enterprise Reference Architecture developed by Theodore J. Williams at the Purdue University early 1990s.
  • SAP Enterprise Architecture Framework
  • Service-oriented modeling framework (SOMF), based on the work of Michael Bell
  • Solution Architecting Mechanism (SAM) – A coherent architecture framework consisting of a set of integral modules.
  • Zachman Framework – an architecture framework, based on the work of John Zachman at IBM in the 1980s

Hope that covers the initial concepts of Enterprise Architecture. Later sessions I would write more on an interesting and widely used Enterprise Architecture framework called TOGAF – The Open Group Architecture Framework. 

Read about my previous article in the mean time: TOGAF 9.1 Certified

References:

Azure Cosmos DB – Consistency Levels

June 2, 2018 Azure, CosmosDB, Data Consistancy, Data Integrity, Higher Availability, Microsoft, Reliability, Resilliancy, Scalability No comments

CosmosDB is a planet scale multi model, multi-region NoSQL database service provided as part of Azure Platform. Azure Cosmos DB is designed to provide global distribution for every data model you choose while creating Cosmos DB.  It is promised to provide low latency and various well-defined consistency models to ensure data redundancy and high availability.

In this short diagram I will be covering the different consistency models available with Cosmos DB and their benefits:

image

There said depending on your data criticality and needs of faster accessibility, you can choose between any of the above consistency models. I strongly trusts in session consistency, as it ensures a balance b/w both.  But again it is totally depending on your business case and how critical is your system depends on the accuracy of this data.

Hope you enjoyed this short article!.

Further reads: https://docs.microsoft.com/en-us/azure/cosmos-db/consistency-levels

Azure Functions App–Run OnDemand Serverless code – a path way to Serverless Computing

June 18, 2017 App Service, Azure, Azure Functions, CosmosDB, Microsoft, Resilliancy, Scalability, Windows Azure Development, Windowz Azure No comments

Azure Functions is a new cloud solution from Azure that would let you execute small pieces code or “functions” in the cloud.  This means you do not have to worry about the infrastructure or environment to execute your little piece of code to solve any of your business problems.

functions-logo

Functions can make development even more productive, and you can use your development language of choice.

Benefits:

  • Pay only for the time your code runs and trust Azure to scale as needed.
  • Azure Functions lets you develop serveries applications on Microsoft Azure.
  • Supports wide variety of development language choices , such as C#, F#, Node.js, Python or PHP.
  • Bring your own dependencies – you can bring any of your Nuget/NPM dependencies for your functional logic.

What can we do with Azure Functions?

Azure Functions is a very good  solution for processing data, integrating systems, working with the internet-of-things (IoT), and building simple APIs and micro services.

Functions provides templates to help you  get started with some useful scenarios, including the following:

  • BlobTrigger – Process Azure Storage blobs when they are added to containers. You might use this function for image resizing.
  • EventHubTrigger – Respond to events delivered to an Azure Event Hub. Particularly useful in application instrumentation, user experience or workflow processing, and Internet of Things (IoT) scenarios.
  • Generic Webhook – Process webhook HTTP requests from any service that supports webhooks.
  • GitHub Webhook – Respond to events that occur in your GitHub repositories.
  • HTTPTrigger – Trigger the execution of your code by using an HTTP request.
  • QueueTrigger – Respond to messages as they arrive in an Azure Storage queue.
  • ServiceBusQueueTrigger – Connect your code to other Azure services or on-premises services by listening to message queues.
  • ServiceBusTopicTrigger – Connect your code to other Azure services or on-premises services by subscribing to topics.
  • TimerTrigger – Execute cleanup or other batch tasks on a predefined schedule.

Integration Support with other Azure Services:

Following are the services integration supported by Azure Functions app.

  • Azure Cosmos DB
  • Azure Event Hubs
  • Azure Mobile Apps (tables)
  • Azure Notification Hubs
  • Azure Service Bus (queues and topics)
  • Azure Storage (blob, queues, and tables)
  • GitHub (webhooks)
  • On-premises (using Service Bus)
  • Twilio (SMS messages)

Costing:

Azure functions will be charged based on two pricing plans below:

  1. App Service Plan – if you already have an Azure App Service running with Logic, Web, Mobile or Web Job, you can use the same environment for your Azure functions execution without needing to pay for extra resources.  You will be charged based on regular app service rates.
  2. Consumption plan  – with this plan you only need to pay for how long and how many times your functions runs and computational needs/resource usage during that execution time. Consumption plan pricing includes a monthly free grant of 1 million requests and 400,000 GB-s of resource consumption per month.

You can find further pricing related info here

Support and SLA:

  • Free billing and subscription management support
  • Flexible support plans starting at $29/month. Find a plan
  • 99.95% guaranteed up time. Read the SLA

Useful Links: