ESB Toolkit 2.0 : Why do we need it?
It's been over 6 months now, since I've started digging into ESB Guidance 2.0 CTP version ( and later ESB Toolkit) and working on an 'Enterprise Service Bus' solution for an Insurance company that would like to have it's systems integrate seamlessly with various 'Service Providers' on the fly, in a very secure, dynamic, reliable way and yet being loosely coupled.
The resulting system should give the end users, an ablilty to configure various applications to consume services offered by a large array of 'Service Providers'. They should also be able to track the messages that go in and out of the hub, do exception management, fault tracking and generate various reports to make business decisions on the fly, all from a web interface. We've been going back and forth, trying out various technologies to find out the best way to implement such a system. After an exhaustive and a very extensive analysis of various ESB products, we finally settled on 'ESB Toolkit 2.0' and 'BizTalk Server 2009'. The application that we developed using ESB Toolkit looks very elegant and provides solution for every single bit of problem we've set out to solve.
'ESB Toolkit' is Microsoft's flavour of 'Enterprise Service Bus'implemented on BizTalk Server 2009 platform. There are plethora of third party vendors who came up with ESB products to help enterprises design and implement their EAI projects. With ESB Tookit 2.0, Microsoft plans to capture a pie of that market. Before we discuss 'ESB Toolkit', we've got to first understand what is 'Enterprise Service Bus'? What role does it play in integrating applications? What solutions does it provide that traditional EAI integrating solutions doesn't?
'Enterprise Service Bus' is an architectural pattern that enables the development of 'Service Oriented Architectural' applications. That begs the question, What is 'Service Oriented Architecture'? While there is lot of depth to the concept, SOA at a very highlevel could be described as an architectural pattern that will allow you to package functionality in your software as a set of interoperable services in a loosely coupled way. It adds an additional layer to your application that consists of set of services that are provided by it. These services are exposed to consumers and they can use one or more of these services to implement business logic in their programs. For any application to fully implement SOA, it has to support these features :
BizTalk is primarily designed to be an 'Enterprise Application Integration' (EAI) tool. It's primary function is to integrate various applications (Service Providers and Service Consumers) using a 'Hub and Spoke' model. BizTalk acts as a hub which accepts an inbound message and depending upon the need, performs various operations like transformation, enrichment, routing etc., before forwarding it to appropriate subscribers. If it's a two way request, it waits for the subscriber to return the message, perform another set of outbound operations before sending it back to the consumer application. It achieves this functionality by providing a series of 'Receive Ports', 'Receive Locations' and 'Send Ports'. To be able to consume a service, the consumer application should have an endpoint where it can send the request. For this, a BizTalk developer typically provides a 'Receive Port' and a 'Receive Location' which consists of the required endpoint. This receive location will have various 'Pipelines', 'Maps' and other BizTalk components associated with it to perform the necessary inbound operations like 'Transformation', 'Decoding', 'Enrichment' etc., Once these operations are performed, it will be forwarded to the subscribing send ports. It's the job of the send port to subscribe to the messages arriving on the receive ports and based on the context, forward it to appropriate service provider. Send port will have all the necessary details like URL, transport protocol etc., It has it's own set of pipelines, maps associated with it to perform the necessary operations before sending the request and after getting the response. When it finally get the response, it forwards it back to the same receive port through which the response message is sent to the caller.
The problem with this approach is, it gets overwhelming after a certain point. BizTalk should know about every single message that it would process. This means that the corresponding BizTalk application should have schemas for all the inbound and all the outbound messages. It receive ports should be configured to pickup specific messages from exactly one location. Send ports should be configured to specific locations. All these configurations should be done during the design time. All the operations that needs to be performed on inbound and outbound messages should be specified during design time too. If the application ever gets too large and complex, it would be very difficult to sustain this kind of architecture. The BizTalk Server would be flooded with large number of 'Receive Ports', 'Receive Locations', 'Send Ports' that gets very difficult to manage and maintain. This architecture is very tightly coupled and neither flexible nor agile.
While there are lot of ways to provide more flexibility, agilty and interoperabililty to BizTalk applications, it requires lot of work on the part of programmer and the solution isn't normally elegant.
This is where 'ESB Toolkit' comes into the picture and fills all the required gaps!!!
ESB Toolkit 2.0 using BizTalk Server 2009 :
ESB Tookit extends the functionality of 'BizTalk Server 2009' by providing wide range of capabilities to it that didn't exist before. It provides a bunch of tools to developers that they could leverage to build robust, connected, service-oriented applications that incorporate itinerary-based service invocation for lightweight service composition and dynamic resolution of endpoints and maps. They would also help in Web service and WS-* integration, fault management, exception management, business activity monitoring and reporting.
'ESB Toolkit' cannot exist without the BizTalk Server. It is built on top of BizTalk and makes use of various components like 'Pipelines', 'Maps', 'Orchestrations' that come along with BizTalk.
ESB Toolkit solves the above mentioned problem of schemas by releasing the need to have schemas for any of it's mediation components like on-ramp, off-ramp, generic routing and generic transformation agents. The ESB solution should be generic, re-usable, flexible and agile. For that to happen, the various components provided by ESB Toolkit should not be tied down to schemas.
For example, a typical BizTalk solution for consuming a webservice would be something like this. A schema would be exposed as a webservice by creating a virtual directory in IIS and creating the necessary receive ports and receive locations. Then, a send port is created and configured to subscribe to that receive location and send a WCF request to the webservice. For every webservice that needs to be consumed, you've got to create seperate receive location and a send port. Now, if the WSDL of any of the already configured webservices should change, then you've got to open the VS, import the new WSDL, compile the project, deploy it and restart the host instance. This is time consuming and ridiculous way of implementing SOA solution.
If the need to have an XML for the incoming and outgoing messages is removed, then it makes life a lot easier for developers and the resulting solutions would look more elegant. With ESB Toolkit, the OnRamp and OffRamp would be binded to generic/dynamic ports which does not have to validate the incomming message against a schema. Since the webservices already do the validation, it doesn't make sense to add another layer of validation in BizTalk. The only requirment is that the messages should be well formed. They should be valid XML documents. This is the pattern followed by all the Itinerary(Mediation) components in ESB Toolkit. They all have inbuilt support for 'Generic XML' so that they don't need to change when webservices change their schema or WSDL.
ESB Toolkit comes with a bunch of webservices which helps in dynamically resolving endpoints, pullout required transformation files (maps) and process mediation policies. It also provides an 'ESB Management Console' Portal which gives various reports on Faults, Exceptions etc.,. It allows business users to have a look at the fault messages, it's context properties and gives them an ability to change the message and resubmit it. The users would also be able to subscribe to alerts, configure registries and finally trace the message as it goes in and out of the BizTalk Server.
Therefore, ESB Toolkit 2.0 is a very welcome addition to BizTalk Server 2009, and it significantly increases the functionality and capabilities than that were previously offered by BizTalk. This also ensures that the BizTalk server is no longer thought of just as an EAI Toolkit but a very important engine in the design and development of SOA applications.
The resulting system should give the end users, an ablilty to configure various applications to consume services offered by a large array of 'Service Providers'. They should also be able to track the messages that go in and out of the hub, do exception management, fault tracking and generate various reports to make business decisions on the fly, all from a web interface. We've been going back and forth, trying out various technologies to find out the best way to implement such a system. After an exhaustive and a very extensive analysis of various ESB products, we finally settled on 'ESB Toolkit 2.0' and 'BizTalk Server 2009'. The application that we developed using ESB Toolkit looks very elegant and provides solution for every single bit of problem we've set out to solve.
'ESB Toolkit' is Microsoft's flavour of 'Enterprise Service Bus'implemented on BizTalk Server 2009 platform. There are plethora of third party vendors who came up with ESB products to help enterprises design and implement their EAI projects. With ESB Tookit 2.0, Microsoft plans to capture a pie of that market. Before we discuss 'ESB Toolkit', we've got to first understand what is 'Enterprise Service Bus'? What role does it play in integrating applications? What solutions does it provide that traditional EAI integrating solutions doesn't?
'Enterprise Service Bus' is an architectural pattern that enables the development of 'Service Oriented Architectural' applications. That begs the question, What is 'Service Oriented Architecture'? While there is lot of depth to the concept, SOA at a very highlevel could be described as an architectural pattern that will allow you to package functionality in your software as a set of interoperable services in a loosely coupled way. It adds an additional layer to your application that consists of set of services that are provided by it. These services are exposed to consumers and they can use one or more of these services to implement business logic in their programs. For any application to fully implement SOA, it has to support these features :
- Services should be loosely coupled with Operating Systems. Consumers had Providers should not be depended on each other or the operation system they are developed on to communicate with each other.
- Services should be flexible. They need to be accessible through variety of transport protocols.
- Services should be agile. Any change in business requirements should be quickly propagated and deployed without much hassle.
- Services should be re-usable. They should be easily composed into new business processes.
- Dynamic Resolution of endpoints for Routing.
- Content based Routing.
- Routing based on Rules Engine. ( In case of BizTalk, it's BRE).
- Message Transformation using Maps ( In case of BizTalk, it's BizTalk Mapper).
- Message Enrichment.
- Service Aggregation.
- Support for multiple transport protocols ( Http, Ftp, MSMQ, FILE etc.,)
- Business Process Orchestration.
- Business Activity Monitoring.
- Message Logging.
- Reports.
- Alerts.
- Exception Managment.
- Support for Service Registries eg UDDI.
- Itineraries(Mediation Policies), which consists of instructions to ESB regarding what components to use in the underlying engine, to process the message.
BizTalk is primarily designed to be an 'Enterprise Application Integration' (EAI) tool. It's primary function is to integrate various applications (Service Providers and Service Consumers) using a 'Hub and Spoke' model. BizTalk acts as a hub which accepts an inbound message and depending upon the need, performs various operations like transformation, enrichment, routing etc., before forwarding it to appropriate subscribers. If it's a two way request, it waits for the subscriber to return the message, perform another set of outbound operations before sending it back to the consumer application. It achieves this functionality by providing a series of 'Receive Ports', 'Receive Locations' and 'Send Ports'. To be able to consume a service, the consumer application should have an endpoint where it can send the request. For this, a BizTalk developer typically provides a 'Receive Port' and a 'Receive Location' which consists of the required endpoint. This receive location will have various 'Pipelines', 'Maps' and other BizTalk components associated with it to perform the necessary inbound operations like 'Transformation', 'Decoding', 'Enrichment' etc., Once these operations are performed, it will be forwarded to the subscribing send ports. It's the job of the send port to subscribe to the messages arriving on the receive ports and based on the context, forward it to appropriate service provider. Send port will have all the necessary details like URL, transport protocol etc., It has it's own set of pipelines, maps associated with it to perform the necessary operations before sending the request and after getting the response. When it finally get the response, it forwards it back to the same receive port through which the response message is sent to the caller.
The problem with this approach is, it gets overwhelming after a certain point. BizTalk should know about every single message that it would process. This means that the corresponding BizTalk application should have schemas for all the inbound and all the outbound messages. It receive ports should be configured to pickup specific messages from exactly one location. Send ports should be configured to specific locations. All these configurations should be done during the design time. All the operations that needs to be performed on inbound and outbound messages should be specified during design time too. If the application ever gets too large and complex, it would be very difficult to sustain this kind of architecture. The BizTalk Server would be flooded with large number of 'Receive Ports', 'Receive Locations', 'Send Ports' that gets very difficult to manage and maintain. This architecture is very tightly coupled and neither flexible nor agile.
While there are lot of ways to provide more flexibility, agilty and interoperabililty to BizTalk applications, it requires lot of work on the part of programmer and the solution isn't normally elegant.
This is where 'ESB Toolkit' comes into the picture and fills all the required gaps!!!
ESB Toolkit 2.0 using BizTalk Server 2009 :
ESB Tookit extends the functionality of 'BizTalk Server 2009' by providing wide range of capabilities to it that didn't exist before. It provides a bunch of tools to developers that they could leverage to build robust, connected, service-oriented applications that incorporate itinerary-based service invocation for lightweight service composition and dynamic resolution of endpoints and maps. They would also help in Web service and WS-* integration, fault management, exception management, business activity monitoring and reporting.
'ESB Toolkit' cannot exist without the BizTalk Server. It is built on top of BizTalk and makes use of various components like 'Pipelines', 'Maps', 'Orchestrations' that come along with BizTalk.
ESB Toolkit solves the above mentioned problem of schemas by releasing the need to have schemas for any of it's mediation components like on-ramp, off-ramp, generic routing and generic transformation agents. The ESB solution should be generic, re-usable, flexible and agile. For that to happen, the various components provided by ESB Toolkit should not be tied down to schemas.
For example, a typical BizTalk solution for consuming a webservice would be something like this. A schema would be exposed as a webservice by creating a virtual directory in IIS and creating the necessary receive ports and receive locations. Then, a send port is created and configured to subscribe to that receive location and send a WCF request to the webservice. For every webservice that needs to be consumed, you've got to create seperate receive location and a send port. Now, if the WSDL of any of the already configured webservices should change, then you've got to open the VS, import the new WSDL, compile the project, deploy it and restart the host instance. This is time consuming and ridiculous way of implementing SOA solution.
If the need to have an XML for the incoming and outgoing messages is removed, then it makes life a lot easier for developers and the resulting solutions would look more elegant. With ESB Toolkit, the OnRamp and OffRamp would be binded to generic/dynamic ports which does not have to validate the incomming message against a schema. Since the webservices already do the validation, it doesn't make sense to add another layer of validation in BizTalk. The only requirment is that the messages should be well formed. They should be valid XML documents. This is the pattern followed by all the Itinerary(Mediation) components in ESB Toolkit. They all have inbuilt support for 'Generic XML' so that they don't need to change when webservices change their schema or WSDL.
ESB Toolkit comes with a bunch of webservices which helps in dynamically resolving endpoints, pullout required transformation files (maps) and process mediation policies. It also provides an 'ESB Management Console' Portal which gives various reports on Faults, Exceptions etc.,. It allows business users to have a look at the fault messages, it's context properties and gives them an ability to change the message and resubmit it. The users would also be able to subscribe to alerts, configure registries and finally trace the message as it goes in and out of the BizTalk Server.
Therefore, ESB Toolkit 2.0 is a very welcome addition to BizTalk Server 2009, and it significantly increases the functionality and capabilities than that were previously offered by BizTalk. This also ensures that the BizTalk server is no longer thought of just as an EAI Toolkit but a very important engine in the design and development of SOA applications.
Interesting read, thanks.
ReplyDelete