Microservices

Recently, there has been a large push away from monolithic application in favor of small, composable, purpose-built micro-services. To me, micro-services are autonomous services that take full responsibility for one business capability. Full responsibility includes presentation, API, data storage and business logic.

Each micro-service is aligned with a specific business function, and only defines the operations necessary to that business function. This may sound exactly like service-oriented architecture (SOA), and indeed, micro-services architecture and SOA share some common characteristics. Both architectures organize code into services, and both define clear boundaries representing the points at which a service should be decoupled from another. However, SOA arose from the need to integrate monolithic applications that exposed an API (usually SOAP-based) with one another. In SOA, integration relies heavily on middle-ware, in particular enterprise service bus (ESB). Micro-services architecture may often make use of a message bus, but there is no logic in the messaging layer whatsoever—it is purely used as a transport for messages from one service to another. This differs dramatically from ESB, which contains substantial logic for message routing, schema validation, message translation, and business rules. As a result, micro-services architectures are substantially less cumbersome than traditional SOA, and don’t require the same level of governance and canonical data modeling to define the interface between services. With micro-services, development is rapid and services evolve alongside the needs of the business.

Moreover the focus of SOA is separation of concern at interface level where micro-services describe the entire deployment scenario.

Another key advantage of the micro-services architecture is that a service can be individually scaled based on its resource requirements. Rather than having to run large servers with lots of CPU and RAM, micro-services can be deployed on smaller hosts containing only those resources required by that service.

As micro-services architecture is a significant shift from monolithic approach it may require restructuring of several components of the architecture landscape. In this article we we discuss some of the more significant ones.

Authentication :

In traditional monolithic application a user would provide a username,password which would be verified and and a user session would be created. All messaging withing the application is usually in context of this user session. In micro-service architecture the entire flow cannot be encapsulated under an user session as it would make the whole application stateful. It needs to be split into external (the API gateway), and internal (between services) authentication. External authentication with user would be handled by an authentication micro-services which can verify a user and issue an authentication token. This token can be passed between micro services to establish authentication.

However this would mean multiple micro services relying on auth server in order to authorize resources. At some point it can run into performance issues due to too many round trips to authentication server. There may be also scalability issues for auth server as number of micro services increases. An alternate approach is to split micro-services into frontend and backend micro-services. Frontend micro-services can use authentication token mechanisms as discussed. Backend micro-services will use App-specific API tokens for each backend service . Every backend micro-service will have its own AppId which can be locally validated by other micro-services.

Service Lookup:

When building micro-services, you have to naturally distribute your application around a network. That means that your services need to be reconfigured with the location of the other services they need to connect to. This reconfiguration needs to be able to happen on the fly, so that when a new service instance is created, the rest of the network can automatically find it and start to communication with it. This process is called Service Discovery.

Apache ZooKeeper is one of tool that can be used for service discovery as it provides a distributed, eventually consistent hierarchical configuration store.

Deployment:

As number of micro-services grow it can be difficult to keep track of which hosts are running certain services. Also if services are implemented in different programming languages, this means the deployment of each service will require a completely different set of libraries and frameworks, making deployment to a server complex.A container technology like Docker provides an ideal solution for this scenario as it creates containers that comprise of just the application and its dependencies. It is this isolation between containers running on the same host that makes deploying micro-service code developed using different languages and frameworks very easy.

Persistence:
In micro-services like isolation of service it is also important to have isolation of data. To ensure loose coupling, each service has its own database (schema). Moreover, different services might use different types of database – a so-called polyglot persistence architecture. For example, a service that needs ACID transactions might use a relational database, whereas a service that is manipulating a social network might use a graph database.
If there is some data that is needed in more than 1 service, there will be one service that is the owner of the data and exposes it over an API.

Tracing & Auditing:
As there are different servers so just looking and finding problem in logs can become big pain.What we need to do is  tag each request and each event with a request-id. This id can be generated when the first request enters the system and can be used for every event and request that is triggered after that. That request-id can be used as a tag in the logs. Logstash with Kibana are open source tools that can be be utilized here.Logstash allow to aggregate logs into one location, and Kibana allow to search log files.So we can trace the entire flow in Kibana using the request-id tag.


Hopefully this high level overview is helpful when designing your own micro-service architectures
There are various strategies for incrementally evolving an existing monolithic application to a micro-service architecture. It also makes sense to iteratively identify components to extract from the monolith and turn into services. While the evolution is not easy, it’s better than trying to develop and maintain an unwieldy monolithic application.

How to build Identity and Access Management In Cloud

Ask a CIO what her top concerns are with cloud computing , and he's almost certainly going to list security as his #1 concern. Cloud Security is a very big issue. At least, it's a very big perceived issue. Though there are several aspects to Cloud security one key aspect is identity and access management. . In case of multiple clouds system administrators spend too much time to create cloud accounts for new users and may forget to terminate accounts for ex-users. User activity without oversight or authorization leads to risk of sensitive data loss and compliance violations. Systems with sensitive or regulated data may be vulnerable without additional layers of protection. 

Federated Identity management is therefore becoming increasingly important as it allows companies to enforce access control policies across multiple applications in a consistent manner, which provides an overall reduced cost of implementation.

As enterprises shifted toward cloud-based services at the turn of the century and software-as-a-service (SaaS) applications became more prevalent, the domain-based identity and access management mechanisms began breaking. This shift created a new need for a secure connection to multiple applications outside of the enterprise perimeter and transformed the perception on identity management.
As enterprises continue to adopt cloud-based technologies outside of their network perimeter, the need for reliable identity management solutions becomes more vital. To address these new use cases, organisations planning to adopt cloud architecture must also create a roadmap to move to  new ways handling identity and access  with cloud.

A good starting point for building a cloud security architecture is the security architecture reference in TOGAF. 

A TOGAF iteration should then be “Cloud Identity and Access”. The Enterprise Architecture team drives the programme and works collaboratively with both the business and the IT department.
For purpose of this blog we limit the scope to identity and access management issues of cloud security.

In the Preliminary Phase a cloud security policy should be in place. A carefully crafted cloud security policy document should answer these questions

• How well does the cloud provider’s security policies and procedures align with the organisation?
• For applications in the cloud, who within the organisation is allowed to modify settings on the cloud that affect performance? 
• How should administrative privileges to the cloud provider be managed? 
• How user consent must be taken care of for storing his data in cloud
• How interoperability of multiple identity providers can be managed?
• How to provide consistent experience across context?
• How user data should and access should be removed once the user is no longer a part of the organisation?

In Architecture Vision Phase define necessary security-related management sign-off milestones of this architecture development cycle. This must also include sign offs with cloud service providers and identity and access providers and determining trust boundaries

Determine and document applicable disaster recovery or business continuity plans/requirements

Any existing disaster recovery and business continuity plans must be understood and their relationship with the planned system defined and documented.Disaster recovery would involve replicating an entire site, including the identity repositories, in addition to replacing hardware or subcomponents.

Identify and document the anticipated physical/business/regulatory environment(s) in which the system(s) will be deployed. All regulations regarding storage of user information in cloud must be clearly understood

In Phase B:Business Architecture set up the businesses process for identity and access management

Identify roles and actors related to user and authorization management.

Determine access levels of users.

In Phase C: Data Architecture, Identity repositories, in particular may be an issue for cloud computing as it pushes information back into siloes, that IT may not have direct access to. It is also recommended to determine user data and privacy classification and to prioritise the risk criteria of what goes in the cloud and what stays on-premise.

During Phase E and F there is a need to focus on these key aspects of Identity and access management

Directory:  Identify a central directory for linking user groups to SaaS.

Access: Integrate externalized access technology with your centralized directory.

User provisioning & de provisioning: The provisioning of new accounts and user profiles

Federation: A common set of policies, practices and protocols in place to manage the identity and trust into IT users and devices across organizations

Audit:  Track user access and actions

A sound architecture will enable re usability of identity and access services for all use cases in public, private and hybrid cloud models. Federation architecture is the first step to identity management to evolve for the cloud to become a trusted computing platform. Organizations thus must have clear roadmap for it before embarking into cloud computing.

Cloud Computing Standards and how they map to a Cloud Reference Architecture

There is currently a lot of discussion about the role of standards in the cloud, along with a large amount of activity in standards development for the cloud. While some parties see the cloud as something completely new that requires an entirely new set of standards, other parties see the cloud as a technology based on existing technologies that already have standards. 

While there are many standards that were developed in support of pre-cloud computing technologies, such as those designed for web services and the Internet, which also support cloud computing, many new cloud computing standards have emerged in recent years and several others are still in draft form.
These standards are now being developed in specific support of cloud computing functions and requirements.. There are many organisation operating in this space but it does not appear there is conscious effort to avoid duplication and contradiction.It is unlikely an all encompassing standard will emerge in cloud.


Based on the conceptual reference model of Cloud Service provider defined by NIST I try to catalogue some cloud related standards and how they map to the reference model. While many standards are generally relevant to these cloud computing areas, the following sections will map those specifically relevant to cloud.

Cloud Service Management includes all of the service-related functions that are necessary for the management and operation of those services required by or proposed to cloud consumers.As illustrated cloud service management can be described from the perspective of business support, provisioning and configuration, and portability and interoperability requirement.

Name of the specification	OASIS Public Administration Cloud Requirements (PACR)
Category	Provisioning/configuration::SLA management
Purpose and function of the specification	Set of common required functional elements, and measurable criteria or qualities that should be present in cloud
URI for the normative text of the specification	https://www.oasis-open.org/committees/pacr/charter.php
The name of the SDO that generated/authored/hosted the specification	Organisation for the Advancement of Structured Information Standards (OASIS)
Which of the categories of Cloud services does the standard address?	SaaS, PaaS, and IaaS

Name of the specification	OASIS Cloud Application Management for Platforms (CAMP)
Category	Provisioning/configuration
Purpose and function of the specification	Management of resources within the Platform as a Service domain.an interoperable protocol that cloud implementers can use to package and deploy their applications.
URI for the normative text of the specification	http://docs.oasis-open.org/camp/camp-spec/v1.1/camp-spec-v1.1.pdf
The name of the SDO that generated/ authored/ hosted the specification	Organisation for the Advancement of Structured Information Standards (OASIS)
Which of the categories of Cloud services does the standard address?	PaaS

Name of the specification	Cloud Infrastructure Management Interface
Category	Provisioning/configuration
Purpose and function of the specification	Management of resources within the Infrastructure as a Service domain. model and protocol for management interactions between a cloud Infrastructure as a Service (IaaS) Provider and the Consumers of an IaaS service
URI for the normative text of the specification	http://dmtf.org/sites/default/files/standards/documents/DSP0263_1.0.0.pdf
The name of the SDO that generated/ authored/ hosted the specification	Distributed Management Task Force (DMTF)
Which of the categories of Cloud services does the standard address?	IaaS

Name of the specification	Topology and Orchestration Specification for Cloud Applications (TOSCA)
Category	Interoprability/Portability
Purpose and function of the specification	Provide interoperable description of application and infrastructure cloud services, the relationships between parts of the service, and the operational behavior of these services
URI for the normative text of the specification	http://www.snia.org/tech_activities/standards/curr_standards/cdmi
The name of the SDO that generated/authored/hosted the specification	Organisation for the Advancement of Structured Information Standards (OASIS)
Which of the categories of Cloud services does the standard address?	SaaS, PaaS, and IaaS

Name of the specification	Open Virtualization Format Specification
Category	Interoperability/Portability::System  Portability
Purpose and function of the specification	An open, secure, portable, efficient and extensible format for the packaging and distribution of software to be run in virtual machines
URI for the normative text of the specification	http://dmtf.org/sites/default/files/standards/documents/DSP0243_2.1.0.pdf
The name of the SDO that generated/authored/hosted the specification	Distributed Management Task Force (DMTF)
Which of the categories of Cloud services does the standard address?	SaaS, PaaS, and IaaS
Known implementations of the specification	VirtualBox, Red Hat Enterprise Virtualization, VMware, IBM POWER server AIX, IBM SmartCloud, OpenNode Cloud Platform, rPath, SUSE Studio

Name of the specification	OASIS Symptoms Automation Framework (SAF)
Category	Monitoring and Reporting
Purpose and function of the specification	Knowledge sharing across domains, allowing consumer and provider to work cooperatively together to ensure adequate capacity, maximize quality of service, and reduce cost.
URI for the normative text of the specification	https://www.oasis-open.org/committees/documents.php?wg_abbrev=saf
The name of the SDO that generated/ authored/ hosted the specification	Organisation for the Advancement of Structured Information Standards (OASIS)
Which of the categories of Cloud services does the standard address?	SaaS, PaaS, and IaaS

Name of the specification	SNIA Cloud Data Management Interface (CDMI)
Category	Provisioning/configuration ::Data  Portability
Purpose and function of the specification	Functional interface that applications will use to create, retrieve, update and delete data elements from the cloud
URI for the normative text of the specification	http://www.snia.org/tech_activities/standards/curr_standards/cdmi
The name of the SDO that generated/authored/hosted the specification	Storage Networking Industry Association International Organization for Standards(SNIA)
Which of the categories of Cloud services does the standard address?	SaaS, PaaS, and IaaS

Name of the specification	CloudAuthZ
Category	Security::Authorization
Purpose and function of the specification	Managing authorizations and entitlements in SaaS, PaaS, and IaaS contexts
URI for the normative text of the specification	https://www.oasis-open.org/committees/cloudauthz/charter.php
The name of the SDO that generated/authored/hosted the specification	Organisation for the Advancement of Structured Information Standards (OASIS)
Which of the categories of Cloud services does the standard address?	SaaS, PaaS, and IaaS

Name of the specification	OASIS Identity in the Cloud
Category	Security::Identity Managent
Purpose and function of the specification	Identity management in cloud computing
URI for the normative text of the specification	https://www.oasis-open.org/committees/id-cloud/charter.php
The name of the SDO that generated/ authored/ hosted the specification	Organisation for the Advancement of Structured Information Standards (OASIS)
Which of the categories of Cloud services does the standard address?	SaaS, PaaS, and IaaS

Name of the specification	Trusted Multi-tenant Infrastructure
Category	Security:: Confidentiality & Integrity
Purpose and function of the specification	Implementation patterns for cloud providers and consumers to implement a trusted computing base using shared multi-tenant infrastructure
URI for the normative text of the specification	https://www.trustedcomputinggroup.org/files/resource_files/54C245B0-1A4B-B294-D0B2E058FF8E5246/TMI_Reference_Model_V1_00R1_0.pdf
The name of the SDO that generated/authored/hosted the specification	TrustedComputingGroup
Which of the categories of Cloud services does the standard address?	SaaS, PaaS, and IaaS

Modeling Cloud Architecture with TOGAF

Cloud computing has emerged as one of the most significant game changers to hit the technology landscape in the recent years.However, the explosion of cloud computing has been a bit of a catch 22 for many enterprises.They  are often struggling to understand the benefits of cloud computing as it appears totally disruptive to their current approach to IT operations.
Enterprise Architects will have a crucial role in cloud adoption strategies for enterprises. Enterprise Architecture deals with the whole enterprise and not only on the IT part, but also it involves the business side. Subsequently EA is for business and IT alignment. Now as Cloud is being presented as the most versatile technology in the IT field, there is greater need among enterprise architects to quickly understand transformations being brought in by the Cloud principles and to do the required twist and tweak on the EA. Increasingly enterprise architects are tasked to tune their EA to be Cloud-ready. 

As most enterprise architects are familiar with TOGAF it is a good starting point to model your cloud architecture roadmap. The TOGAF ADM can help immeasurably in arriving at a viable Cloud strategy that leads to closer business – IT alignment. 

As next-generation services and applications have to be analyzed, architected, and constructed for the cloud landscape from the ground up, the top down approach would be most logical approach.We discuss here the specific steps which need to be adapted in TOGAF for building cloud architectures

Preliminary Phase: -

Identify and Establish Architecture Principles
The architecture principles are an important anchor when establishing the architecture governance. Some Cloud specific architecture principles could be 

• Small stateless services
• Use Scalable Ingredients
• Use Resource Pooling
• Support Multi-Tenancy

Tailoring Of TOGAF Architecture  Views for Cloud Applications
While  TOGAF ADM is one of  the very good approach to architect and design Cloud Applications from top down perspective,but the architectural views defined in this framework need to be twisted and tweaked in order to simplify and streamline cloud enablement. Cloud specific terminologies like 'Virtualization' 'SAAS' 'PAAS' 'IAAS' needs to   incorporated as a part of terminology tailoring.

Architecture Vision: -

The high-level description produced in Phase A will reflect the cloud based nature of the architecture that is envisaged. 

Stakeholders, Concerns, and Business Requirements

There are some concerns that are specific to Cloud in terms of data security , privacy , and transparency which needs to be identified.

Assess Readiness for Business Transformation

Quantify the enterprise’s readiness to undergo changes due to cloud model.
Extend Business Transformation Readiness Assessment of TOGAF to include cloud specific assessments.

• Is the business prepared to account for IT or applications to be metered out as a service? 
• How will processes for change and release fare when allowing certain user groups to self-provision resources? 
• How will compliance controls perform in a highly automated virtual world of shared resources?   
• What business use cases might benefit most from implementing an internal cloud?
• Is your IT Operations ready to adopt IT Service Management?

Business Architecture

While the overall business goals of a cloud enabled application will not change, the Business users themselves will be variable in a multi tenant scenario and hence this view may needs to be adjusted for different groups that the Cloud service will cater .  Especially the questions like  Who Does It, What Do They Do will change for Cloud Applications when compared to traditional enterprise applications.

Business applications are typically delivered from enterprise server machines. Now with the emergence of powerful and affordable cloud servers, business services and applications are to be deployed and delivered from the new environment. The business architecture has to take care of security, privacy, and other quality attributes of cloud-based applications. Governance, visibility, and controllability are other important factors to be given a serious thought. There has to be a kind of synchronization between enterprise resources and cloud-based applications.

Data Architecture

The core Entity Relationship modeling of a cloud application may match that of its  traditional enterprise application counterpart , however  multi tenancy aspect will introduce new variations to the Logical Data Model. Data Security View  will be totally different from a traditional enterprise application.

Application Architecture

Applications to be cloud ready have to undergo some critical changes. The PaaS Platform will abstract several traditional components that are part of the Application Architecture View and hence this view will be different from a traditional enterprise application. This includes custom-built shelf software to be converted into SaaS.  

Technology Architecture

Due to the tenants of cloud application like virtualized server environment ,PaaS platform ,On-demand instances and other virtual storage considerations, this view will be the one that will go through maximum changes for a cloud application when compared to a normal application. 



We see that bringing cloud capabilities to an enterprise is about more than just the latest technology; it is about changing the traditional business and collaboration model with partners, customers, and providers of services to the enterprise.

Modeling Canonical Data with ArchiMate

Almost all large organisations have many applications which were developed based on different data models and data definitions, yet they all need to share data. So the basic idea of a canonical data model is “let’s use a common language when exchanging information between systems.” So, if system A wants to send data to system B, it translates the data into an agreed-upon standard syntax (canonical format) without regard to the data structures, syntax or protocol of system B.  When B receives the data, it translates the canonical format into its internal format and everything is good.

The benefit of this approach is that system A and B are now decoupled; if A needs to be upgraded, enhanced or replaced, as long as it continues to reformat the data for external systems into canonical form, then B or any other system is not impacted. Simple and elegant!  Under this approach, organisations with hundreds of application systems can now happily upgrade and enhance individual systems without the complexity (and cost/risk) of coordinating the analysis, design and testing with all the other systems.

However, there is a catch.  While A and B are no longer coupled to each other, they are now coupled to the canonical physical format.  So now if the canonical model changes, every single system that uses it to send or receive information is potentially impacted.The results endless analysis and discussions among business analyst, architects and designers of any proposed change to the canonical model.

The complexity of data Governance is often avoided by creating  "quick and dirty" service implementations, which focus only on the requirements at hand so as to speed up implementation. Over time, these services create increasingly reductant services and restricts the flexibility and extensibility of the overall SOA.

One symptom of this approach is revealed by the inspection of the data structures being exposed through service messages. If there are many services exposing arbitrary variations of the same business information, or many instances of data mapping running in the enterprise service bus (ESB) and business processes, then it is likely that a simpler and more reusable solution could have been developed. Left unchecked, this approach can lead to a solution so complex that all the benefits of the SOA approach are lost.

A good place to look for possible candidates for the CDM is the Information Structure Viewpoint and Application co-operation viewpoint in ArchiMate model.

A structure like this would indicate possibility of a redundant information in both the customer model.

A target Information model for this could be to have a specialised representation of customer as shown. However it is not evident from this model that customer object is a part of the Canonical data model. We can use group to explicitly  state this fact as shown in the next model.

The customer object is now a part of a group is represents a set of  canonical data objects

Note that canonical data model is often achieved though Enterprise service bus patterns. We do not model the ESB patterns in the above diagram. This model merely points to the presence of a CDM  and not how its implemented.

Starting with ArchiMate -2

In Starting with ArchiMate -1 we had a overview of the the ArchiMate meta models. In this section we discuss the possible relations between the components.

Here is a list of relations in ArchiMate.





All relations have a direction, except Association which is bidirectional (Assignment was also bidirectional in ArchiMate 1.0).
ArchiMate, also offers a mechanism to create ‘shortcuts’ called ‘derived structural relations’. The derived relation between two objects is the weakest relation found on the route

ArchiMate 2.1 released

ArchiMate 2.1 has been released.

Apart from the editorial corrections the major change is change of classification of the concepts within the Business , Application and Technology layers.
In ArchiMate 2.0 the core concepts section describes three types of elements :-active structure elements, behavioral elements and passive structure.

However, within the the layers the concepts are categorized as 

• Structural Concepts

• active entities

• passive entities

• Behavioral Concepts

• Informational Concepts

This was inconsistent with the core concepts. Also the at times it is ambiguous whether an entity is a passive structure or information as both do not invoke behavior (as active entity) or are themselves a behavior.
This has been fixed in 2.1 release and the new classification within the layers stand as

• Active Structure Concepts

• Behavioral Concepts

• Passive Structure Concepts

Impact on your existing models:
All references to business objects and data objects which may be categorized as Structural objects in the metadata of the element must be changed to Passive Structure Object. The Information Object category itself should be renamed to Passive Structure objects.