This project is read-only.

Getting Re-Started!

Topics: Developer Forum, Project Management Forum, User Forum
Jul 22, 2007 at 11:57 PM
Hi, I've been following your project keenly for some time now – I had even downloaded the code couple of times, but never really got started. Anyway, I just thought I'll put forward some questions and some pointers/observations (to help make this a better product or at least easier to start with)?

Firstly, some pointers;
- I am sure you know (and I know it isn't easy to document), but just the documentations seems a bit doggy. I can see you have put out a new guide, and a book is forthcoming, but in the documentation you should perhaps try to build on the topics step-by-step. Yes, an ESB is a complex beast to start with, but maybe you can document at different levels of abstractions (like overview, technical, code level) separately. And perhaps use a wiki, and some nice graphs/pictures too.
- I checked out your console too, but it really seemed overly complex and confusing! For example, why put "RAW XML" to show by default, rather than a services grid first? I see the word “service” three times on the navigation tree, and couldn't make out which is the real-deal? These might seem minor quibbles, but it does make an impression. All the same, the configuration console needs to be made easier to approach and more web 2.0'ish.
- Perhaps you can separate the code from a binary (plus console, and samples) release. You don't want everyone to have to download 50+ MBs.

Secondly, I wanted to put forward some questions (as a beginner), see if you can shoot straight at them:
- What is the basic process flow? Is it like a message comes in and you read the envelope to process it, then transform it etc. etc.?
- Is the envelope central to your ESB, is it able to dynamically wrap (non-envelope) inbound messages with pre-configured settings.
- Is usage of client side assemblies mandatory?
- Is the ESB joint to the hip with MSMQ and WSE 3.0 - I see them as a requirements? If so why?
- In the documentation you make a case for "Business Process Centric", that makes me ask how does it sit with Event Driven Architecture? As you may know ESBs is frequently used as a platform to build EDA implementations, and the gratuity level required thereof is that of an event rather than a process.
- How well do you support or plan to support event processing?
- Going forward is WCF at the heart of the ESB? And do you support all types of transport channels for WCF?
- Do you support sessionful services, with dynamic routing and transformations?
- Do you support all types of asynchronous, synchronous and request-response invocations?
- Can you route/match non .NET and other (e.g. POX) types of inputs with different types of consumers? Do you support REST invocation?
- Do you support non-service endpoints e.g. File System?
- What kind of content-based and rule-based routing mechanisms do you use?
- How does your offering compare to other open source ESBs, case in point Mule?

All in all, I see you have a rich feature set, and by just tweaking here there a bit plus making a lot of communal noise, you can build a rich community around your product - which is what I think you want to do? Further, if your solution seems right for my needs, I would be willing to help build a better solution with you.

Cheers,
Rishi
Jul 26, 2007 at 5:27 PM
Edited Jul 26, 2007 at 5:31 PM
(Due to limitations on the text length, my reply is split over 3 posts - see below...)
{quote}
r5 wrote:
Hi, I've been following your project keenly for some time now – I had even downloaded the code couple of times, but never really got started. Anyway, I just thought I'll put forward some questions and some pointers/observations (to help make this a better product or at least easier to start with)?
:) ==> Feedback always welcome. Thanks for taking the time to do so.

Firstly, some pointers;
- I am sure you know (and I know it isn't easy to document), but just the documentations seems a bit doggy. I can see you have put out a new guide, and a book is forthcoming, but in the documentation you should perhaps try to build on the topics step-by-step. Yes, an ESB is a complex beast to start with, but maybe you can document at different levels of abstractions (like overview, technical, code level) separately. And perhaps use a wiki, and some nice graphs/pictures too.
==>I've got loads of documentation that needs updating & sanitizing. I've been doing this since 1999, and many things are still relevant today. It's a balance between developing & documenting I guess. Since I'm the only person on it at this point, and I have many other committments, unfortunately, things are moving slowly.
I will take your comments on board. Using a wiki may not be the best approach for formal doco, however is great for community content I guess. If anyone else joins this project, they can contribute via the codeplex wiki.

- I checked out your console too, but it really seemed overly complex and confusing! For example, why put "RAW XML" to show by default, rather than a services grid first? I see the word “service” three times on the navigation tree, and couldn't make out which is the real-deal? These might seem minor quibbles, but it does make an impression. All the same, the configuration console needs to be made easier to approach and more web 2.0'ish.
==>Hehe - no arguments there. I've publicly stated on the codeplex pages that the Management Console is no more than ad-hoc thrown together pages. It is definitely long overdue for development as a proper web application. Unfortunately, as mentioned above, apart from personal bandwidth, I don't have the best web UI (or any UI for that matter) development skills. I can't afford to spend too much time there myself right now, as I've got loads of server-side features I want to implement.
I'd love it if someone could just take it & run with it. All the code is there & I'm sure the intent is fairly clear.
There are 2 main use cases for the ESB.NET Management Console I guess. The first is a lightweight (DB Server-less) option (can use DB files if required). A second, more complete Management Console could make use of a database for richer functionality. Other than that general requirement, I really don't care what it looks like.

- Perhaps you can separate the code from a binary (plus console, and samples) release. You don't want everyone to have to download 50+ MBs.
==>I do have such a package (~7-8MBytes), however it can't be used to build the entire system. Reason being is that the entire ESB.NET source code has not yet been released to codeplex.
As per the licensing:
"(1) Release runtime only – potentially release under Microsoft Reference License (MS-RL) in future."
, some core components are still under lock & key. I'm still deciding what to do with them. At the end of the day, my objective is to somehow make more money from this long investment. I've no plans yet, and need to hold onto something key till I sort out some direction in this area.
So, unfortunately, giving out part of the source code is next to useless as you can't run it, and can't build it to run it...
I use that package to periodically update the codeplex TFS repository. Only my own TFS repository contains the whole source tree at this time. I'd very much like to open it all up, but have a lot of things to sort out before I can do that. I guess this is one thing restricting adoption as well.
Trust me, I'd love nothing more than to:
1)Release everything to the community
2)Make loads of money from it
But there's still lots of space between the current and desired target states! :)


Secondly, I wanted to put forward some questions (as a beginner), see if you can shoot straight at them:
- What is the basic process flow? Is it like a message comes in and you read the envelope to process it, then transform it etc. etc.?
==> Pretty much. The core ESB reads the Envelope headers and decides how to route to appropriate service(s). It also decides (based upon both request parameters and service configuration), whether the processing is done
synchronously/asynchronously, if sync, where responses go etc.
Any transformation is done by your service specific implementation.
There are a number of places in the pipeline where you can inject common processing. These can be done for all services (eg. various authorization filters, routing etc.), or on a per-service basis.
Your services also have a view of the request pipeline, and can do things accordingly. For example, typically, to take full advantage of all ESB.NET features, you need to configure the service.
There is, however, a ConventionOverConfig Adaptor that makes certain assumptions and gives you the option to forego some features in order to bypass the configuration overhead.
Similarly, the Sequential Workflow Adaptor can be configured (& soon via the ConventionOverConfig Adaptor) used to route service requests to Windows Workflow based orchestrations.


- Is the envelope central to your ESB, is it able to dynamically wrap (non-envelope) inbound messages with pre-configured settings.
==> :) Yes and Yes. In fact, the Envelope is implemented using provider model. There are a couple of envelopes that currently ship. I've also developed a couple more for specific parties that are not public domain.
There's an IEnvelopeProvider interface (not for the feint harted) that you can implement to serialize to pretty much any envelope that caters for the basic Keystroke ESB Architecture's Envelope requirements.
To try & dumb it down, I've not elaborated much here, as I'm a little conscious that it's scary enough to newbies I guess.
I've got some XSLT that I've developed to generate WSDL from the Service Definition. It's up to the client to determine how many service definitions they want to expose in their WSDL. It's basically client-side managed/dynamic WSDL.
For practical regression testing type reasons, WSDL (apart from being a technical artefact) is not always appropriate for business activity services. Applications that have a well-defined boundary are well-positioned to take advantage of WSDL,
but it's not the case for business activity services which are very course grained and are related to a business process which typically spans unrelated application services. Anyway, this mechanism gives the client
control of what dependencies it has, and reduces unnecessary coupling.
There's currently a raw HTTP transport adaptor that takes an XML request generated from the WSDL mentioned above and wraps it up in an envelope before submitting to the general ESB.NET core for processing.
There are of course pro's & con's, but it's intended for the 'point-n-click' clients that require WSDL. Unfortuntely, it's a Work In Progress with a medium priority. There are many ways to generate the WSDL, as well as the
issues in implementing them with WCF. It has, however been started. As you can see, there's a 'ESBHTTPGenericWebServiceTransport.aspx' endpoint that does a bit of work here.
It's by no means complete though.


- Is usage of client side assemblies mandatory?
==>No, however they're highly recommended for clients that intend to use many ESB.NET services.
ALL ESB.NET services are exposed over standards based protocols & data formats. Constructing the envelope is a little tedious though. Also, 'point-n-click' clients will generate some duplicate code which becomes a problem if not addressed early.
The client side assemblies are a way to hide that complexity for large implementations. As with the server dide, the hard work is done for you. Unfortunately, there are currently no wins for small implementations/proof-of-concept/point-n-click type of projects.
ESB.NET comes into it's own when the complexity increases. It does currently have a high barrier to entry, which I'm trying to reduce via the ESBHTTPGenericWebServiceTransport and the like etc.

Jul 26, 2007 at 5:30 PM
- Is the ESB joint to the hip with MSMQ and WSE 3.0 - I see them as a requirements? If so why?
=MSMQ=>Sort of. MSMQ & MSMQ Triggers is used for asynchronous message processing.
The SQL Server Service Broker currently has a couple of limitations that make it not-ideal for large scale deployment. The express versions of SQL Server 2005 have a 1 or 2 CPU limit. This would mean that in a farm and/or federated deployment, you would be bound to loads of SQL Server 2005 licenses
(doesn't go down well with me). Similarly, installation would be much more heavyweight requiring SQL Server at many points.
MSMQ is also a strategic component in terms of async logging (another medium priority task) whereby logging can go there first before taking the SQL hit.
The SQL Server Service Broker (only from some literature I've read) is not as quick as MSMQ. It doesn't offer an active/active load balanced queueing solution, so my desire to jump to it is not that great (although I have it as a low priority background task). Managing queues/backups/recovery in SQL Server
is much nicer than that of MSMQ.
=WSE3=>This is not strictly required. I have it for 2 reasons:
1)To demonstrate backwards compatibility with many existing WSE3 services.
2)Because WCF has issues with client side channel setup time. WSE3 does not have an initial delay as does WCF. Therefore, for simpler cases, WSE3 is actually a better client! Now, given that I want to point my transport adaptor client
to an arbitraty endpoint, with WCF, I will have to start maintining some sort of state which I don't like. It just makes it harder to get right. WSE3 on the other hand is braindead to get right.
If you run the service tests with all logging disabled, you'll notice the WCF trnsport adaptor takes over 150 ms to execute. The majority of this time is spent in setting up the client channel. Because I don't pool the connection or use the static proxy trick,
I cop this hit every time. It's an outstanding WCF issue I need to sort out.
If you change the transport to HTTP or WSE3 though, you'll see the times come down to ~10ms or so.
If you want to test the WCF performance after the channel has been established, in the "generic request sender" page, select a request and set the "Num Execution Times Adaptor" property to say 100. This will create the WCF transport adaptor
and invoke it repetedly. The "Num Execution Times" property on the other hand, will create the transport adaptor each time, create the channel every time, and cop the channel creation time hit every time.
That will clearly demonstrate the current WCF issue.


- In the documentation you make a case for "Business Process Centric", that makes me ask how does it sit with Event Driven Architecture? As you may know ESBs is frequently used as a platform to build EDA implementations, and the gratuity level required thereof is that of an event rather than a process.
ESB.NET supports both service invocation & pub/sub in exactly the same manner.
They're both done via config, and the client controls whether they get sync/async behviour from the ESB.NET server.
None of these though, play any role on the "Business Process Centric" view statement I make.
The statement I make is this:
For every message that enters ESB.NET, it's often desirable to create a service definition which is related to the Business Process Activity that the Business Activity Service is meant to implement.
The ESB can then make use of that service definition to do various things, including that dynamic WSDL generation.
The Service Definition spells out several things about the service that WSDL does not capture. It's also an artefact that is born at Business Process Analysis time, and carries through to runtime. This means that there's little
double (or more) handling of requirements data.
When a particular Business Activity Service executes, an event can also be fired. This is done in pretty much the same way as a service is configured.
The act of publishing an event is merely a transport adaptor configured to forward requests to another dedicated pub/sub instance of ESB.NET, or just a local configuration entry which points to the subscriber directly.
SOA vs. EDA are technology related level architecture issues. By that I mean, they don't come into the Business Architecture in any direct way.
Also, a bit of a confusion over SOA vs EDA sometimes infers that SOA is alternate to EDA. The true intent of SOA does not in any way exclude EDA as a competing or alternate architecture. Well, not in my books anyway.
EDA is a subset of the SOA paradigm.


- How well do you support or plan to support event processing?
==>Already does (see above statement). It's exactly the same as for a service. ESB.NET is message based. All activity is an event with a response of some sort. Any message is usually accompanied by an acknowledgement of sorts.
The acknowledgement is determined by both client message parameters & service specific config.
For example, a service handler that calculates 2 numbers is generally written as a request/reply component.
When configured in ESB.NET, the configuration (which is the actual service implementation), determines if responses should be sent anywhere other than just the calling thread.
Also, the client determines (separate to the service implementation & config) whether they want to wait for ESB.NET to execute the service logic, or to just get an acknowledgement that the message has been delivered.

- Going forward is WCF at the heart of the ESB? And do you support all types of transport channels for WCF?
==>WCF currently is at the heart of WCF. WSE3 is too. They actually run in separate IIS Virtual Directories, and can run independently of one another. It just so happens that the ESBHTTPTransportInternalAsync.aspx transport adaptor that
receives the MSMQ Triggers notifications, is housed in the WSE3 Virtual directory. It could be anywhere though.
I won't remove WSE3 until ALL WCF issues have been adequately resolved.
==>Only the WCF Transports that are standards compliant are supported. It's one of the Keystroke ESB Architectural principles. Any service that's deployed should be callable from anything, anywhere.
Remember that although you can just CHANGE WCF configurtion to change the endpoint, it's nonetheless a CHANGE. During dev, this is not an issue. In prod though, a CHANGE is a CHANGE.
I have no current plans to support Microsoft specific bindings.
ESB.NET however allows you to easily build a transport adaptor. In fact, the WCF layer, as with all transports, is relatively thin. You could build one to do whatever you like. Check out the transport adapter code. It's relatively simple.


- Do you support sessionful services, with dynamic routing and transformations?
==>The ESB.NET core is completely stateless.
Services will need to maintain state themselves. For Example, in prior versions, I had a beta Windows Workflow State Based Workflow adaptor that maintained state.

If a session is desired, then a separate Transport Adaptor may need to be developed, or the existing ones changed to do whatever's required.
One of the principles of the Keystroke ESB Architecture is that services are stateless. State is factored out into a separate ESB.NET instance and isolated as per best practices for Workflow/BPM etc.

- Do you support all types of asynchronous, synchronous and request-response invocations?
==>Pretty much. See above.

- Can you route/match non .NET and other (e.g. POX) types of inputs with different types of consumers? Do you support REST invocation?
==>Only standards compliant transports & data formats (eg. HTTP, SOAP, WS-*) and XML are used. As such, it interoperates with Java or anything that can talk standards.
==>Service implementation is done using .NET, so internally, you're bound to whatever .NET's bound to.

- Do you support non-service endpoints e.g. File System?
==>No, not out of the box. A transport adaptor could be written to do this if desired. Generally, the transport adaptors either host the ESB.NET core directly, or they listen on something and invoke an ESB.NET instance somehow using "Send" Transport Adaptors.
For example, the async processing works by MSMQ Triggers notifications, although that's only used internally (using externally violates architectural principles, although there's no technical limitation).
It'd be relatively simple to do the same with say a file system or DB trigger I guess.

- What kind of content-based and rule-based routing mechanisms do you use?
==>Content based routing is limited to the ESB Envelope Header fields.
==>You can write your own routing logic. For example, you can use the Windows Workflow rules engine to do such logic.
Just configure the Service Handler as an "Always Execute" handler, and it will get executed for every request. You then decide what you want to do with the request, and if you wish to continue the rest of the
pipeline processing.

Jul 26, 2007 at 5:30 PM
- How does your offering compare to other open source ESBs, case in point Mule?
==>I've not used Mule myself. I've used BizTalk, webMethods, infravio, Sonic MQ (ages ago) and Seebeyond's eGate. Other java based open source kits seem to be unstable at this point. We have a few people looking into them
but no luck as yet. I actually suggested Mule as it's something that I've heard of years ago, as opposed to Service Mix & WS02 which are relatively new.

ESB.NET differs from them in that it has that Business Process Centric & Service Definition view. It pretty much mandates the Envelope concept and is purely message based.
As a result, ESB.NET has a relatively high barrier to entry compared to some other ESB's (although many open source ones we've looked just don't work or we can't get them to work - not bagging, just our experience. I'm sure they'll all improve over time.).
That said, ESB.NET, once up & running, just works for you, and there are no real dead-end surprises for large scale deployments. That's what it's designed for, and that's what it's good at.
If you want point-n-click, ESB.NET's not there yet.
If you've got large amounts of data, many course grained services to develop for an organization that needs flexible federated routing support, supports farms, is reliable & scalable up & across, then ESB.NET's a good choice.
It's not a silver bullet, but the Keystroke ESB architecture is implemented in ESB.NET and gives you a realistic ESB that delivers on the SOA and ESB hype. You just have to put some effort into it & understand exactly
what you want before doing anything.
As such, I'm available for consulting to help get people get up to speed on realities of ESB's and what they should be looking at doing, as well as practical ways to implement the vision with ESB.NET.


All in all, I see you have a rich feature set, and by just tweaking here there a bit plus making a lot of communal noise, you can build a rich community around your product - which is what I think you want to do?
==>You're exactly right.

Further, if your solution seems right for my needs, I would be willing to help build a better solution with you.
==>That'd be great. There's many areas that need help. Understnding the Architectural Principles is key, as it pretty much sets the scene for what to do & what not to do, and where I want to take it.
I'd love more community input.



Cheers,
Rishi

{quote}


Thanks for your feedback. Very much appreciated.
Cheers
Minas
Jul 29, 2007 at 12:22 AM
Hi Minas,
Thanks for your 3x reply, it really helped me get a much better picture - though still I have some quiries, but we can get to them later.

From my side, what I wanted was a lightweight ESB solution which is designed for WCF (not exclusively, but primarly). Why WCF? Well, because it offers such a clean seperation from the underlying mechanics, kick-ass design model, and by default supports multiple transport channels (even non .net). These and more WCF attributes provide a great starting point for ESBs to build upon. With regards to you point on perfomance of WCF, I wanted to point out to you this podcast discussing WCF relative to WSE.

http://aspnetpodcast.com/CS11/blogs/asp.net_podcast/archive/2007/06/14/asp-net-podcast-show-94-paul-on-the-wcf-mp3-for-all.aspx

And in as far as I know, WCF should generally yield better perfomance than WSE, there is a study to that effect - you can google for it. Also, I am not quite comfortable with using MSMQ, though I am not aware of the particulars with why you needed to use it, but I am quite sure it can be done away with. There are a lot of other routes to do async message processing.

I don't know if you have checked out Microsoft's Biztalk based ESB Guidance - they also use the envolpe concept to proccess messages, which they call OnRamp or something. And mule is a very mature ESB, I recently heard about a single deployment on 13,000 servers. Even other than mule, there are a lot of very good ESB implementations in the java world, which to me as a non-java guy puts .NET community to a shame! In terms of middlewear I am starting to suspect .NET community is lagging behind, big time!

All in all, I am looking for a liteweight WCF focused ESB solution, with good scalibility and extensibility. Usability and the barrier to entry is a consideration or deliverable too. So, I don't know how we can co-operate on this or something, but it has to a streamlined solution.

Let me know what do think.
Thanks,

Rishi





Jul 29, 2007 at 4:29 AM
Edited Jul 30, 2007 at 1:58 PM
Hi Rishi,
WCF - As I mentioned, WCF is just used as a transport. The service implementation is not directly connected to WCF. The decoupling makes everything cleaner overall & much less technology bound.
As such, you can turn on/off pretty much any WCF feature at the Transport Adaptor Level, and your service implementing code will still work. Your service handlers are plug-ins, and isolated from the transport mechanics.
WCF and WSE run in different IIS Virtual Directories. You can therefore run them in separate app pools or disable one if you like.
W.r.t. WCF performance, as I mentioned in the previous post (although maybe not clearly enough), once the client channel is setup, WCF is a little faster than WSE3 (& obviously a little slower than raw HTTP). You can see that by playing with the Management Console.
Thanks for the link. This WCF thing is something I want to address & get out of the way. WSE3 will then be a legacy type transport option & eventully phased out.
Until that issue is sorted, WSE remains the fastest way to route to ad-hoc endpoints, as required in a generic transport.send type adaptor.
WCF is of course the technology of choice today on the .NET platform - that's without a doubt. Sorry if I implied any different. There's just some issues that I need to address, and that WSE3 currently (with my implementation & by default) does better. I'm sure this will be addressed by MS at some point. Maybe a wrapper class or something.

W.r.t BizTalk ESB Envelope. If you look at it, it's very technology specific. Akin in a way to the SOAP envelope in terms of it's technology focus.
The ESB.NET Envelope is a Business Data type envelope. It focuses on the payload and service, and has some high level routing/endpoint info in it. There's no endpoint bindings etc. included. The SOAP/WSDL crud is defaulted per profile, and buried away in a config file that defines the possible profiles that are available for all services. A generic authorization filter can then determine what services are allowed over which transport profile etc.
This means that an ESB Envelope makes lots of sense to a business person. WSDL & the BizTalk ESB type envelope on the other hand focus on technology aspects, and the majority of data there is meningless to business person. This violates the first Keystroke ESB Architecture principle of being Business Process Centric rather that technology centric. Not that what they do is wrong in any way. It's just that ESB.NET standardizes on some technical issues, and decouples them from service definition, service implementation and data definition. This allows it to stand the test of time w.r.t. moving technologies, whereas a technology focused anything is inherently tied to technology & is therefore directly impacted by changes in the technologies it uses.
This premise for this principle is understood better once you've done the sme thing for a few years, seen disparate technologies work together and had to factor out the essence of what's to be achieved at the business level.
An ESB in the end, is a technical means to a business end. When you look at it in that light, with many non-related systems and technologies, it will give you an idea as to how better factor things out for that particular problem.
Put another way, a typical ESB solution will get a handful of systems working together today.
Yesterday's systems (that can be modified) that don't understand all of today's technologies, but that can run most of yesterday, today & tomorrow's business rules & logic, can still effectively provide services using the Keystroke ESB Architecture in much the same manner that today's systems provide services. Similrly, tomorrow's systems will again be able to provide functionality in the same manner.
Technologies will always change & make our tasks easier. Factoring them actually out makes introducing them to existing systems much easier.
If focusing on technology though, any change has a deep impact to existing systems.

MSMQ - There's a couple of issues here I guess.
1)General reliable messaging & processing. Whilst WCF is good for WS-RM at the transport level, something more's needed at the service implementation level. MSMQ fits the bill here as clearlt something's needed.
MSMQ supports transactions, is multi-threaded & lightweight. V4 supposedly supports the remote transacted reads as well that everyone's been waiting for!
In terms of management though, I guess SQL would have to be the more manageable solution, but the SQL Service Broker is relatively new, a little more heavyweight and artificial CPU limiting because of licensing (SQL Express versions for farm deployments etc.) is always going to be a thorn in someone's side.
2)Async processing - MSMQ was just an easy way to do it. Maybe not the best, but it's fairly clean & light given the fact that it's required by point 1 above.
This is just implementation CRUD though.
There's nothing stopping a switch to something else. Certainly there's no direct dependency between ESB.NET developes services and MSMQ, hence changing it in ESB.NET versions would be transparent to services. Services just flick a flag to determine whether something's async. eg. Client sets the 'callSynchronizationType' flag in the request to determine whether the server will process this request async to a client thread. Similarly, in the service pipeline config, if a particular handler in the pipeline is flagged as ProcessInParallel, then again executed asynchronously.
I certainly don't have any hang-ups pro or against MSMQ. It's just part of the platform, and a logical choice. If something better comes up, then I'll drop MSMQ.
Currently my main gripes with MSMQ are:
1)MSMQ Triggers needing a COM registered component!!! to react to triggers.
2)MSMQ has a central queue service and repository for each server (or OS instance).
3)I'd prefer xcopy type deployment functionality here, rather than to have to call API's to deploy things around.

If I came across something (ideally that was part of the MS technology stack), I'd switch to it at the blink of an eye.
Something that had transparent, active-active clustering/load balancing support would also be nice, but I haven't seen such a beast anywhere. Maybe some of the Windows Grid type of features may do this sort of thing. Gigaspaces maybe? Dunno, but very interested.

Not in anyway meaning to bag anyone, these are some of my experiences in this area in the Java space through various projects I've been part of or close to over the past few years:
Not sure about any good ESB implementations in the java world.
- I heard though, that after Sun bought Seebeyond, their ICANN suite got a nice makeover. I've never used it though, was just told that at a vendor implemented & demonstrated POC, it looked real schmick!
- Other than that, the old seebeyond suite was not too crash hot.
- The webMethods suite does not cut it at all (though it costs a packet).
- People I've worked with found WS02 to be "not all there". Basic functionality works, but other than that, doesn't hang together too well.
- Similarly, ServiceMix is lacking any working WS-* support.
- Never used Mule, nor do I know anyone that had, but have read positive things on the web about it.

- I've not used BEA's Aqualogic, but I've heard it's good. Nothing more than that though.
- I used Sonic MQ years ago. It was so-so. We had lots of problems getting it configured in repeatable manner!
- I've not used the most recent Sonic ESB.
- I've not used IBM's stack.

Can't remember anything else off the top of my head.
Until you actually use something, it's hard to say really. These change over time, and something that was good or bad at some point can change relatively quickly.

In my experience in the Java & open source world, tools are usually cr@p, and it takes good people to sort them out.
What has happened though, is that a lot of good people have taken them on and done good implementations.
It's a similar story on the MS Stack.
Everywhere it seems though, the effort is a once-off or twice at most.
Reason being, the products & market are still maturing.
Technology - as always - is changing, and getting any long-term benefits is difficult. Technology wise, so many things change so quickly, that all the ESB type stacks have trouble keeping up.
Hopefully there will be a bit of a flatline over the next couple of years as WS-* matures a little on the java platform as it hs with .NET/WCF on the MS stack, and some XSD (& maybe? XSLT) type tools are released to general developers my MS.
Apache Axis 2 is out in Java land, but it's not nearly as approachable as WCF is. Onya's MS!!!

My conculsion on ESB's in the java space is therefore negative, especially in the open source space. I've NOT used them all though, as would be next to impossible of course, but from what I've seen so far, I'm nowhere near impressed by any particular product.
So...my point then is... from my experience, the grass doesn't seem much greener on the other side.

I might do some more background reading on mule when I get a chance though.
In any case, any of the suite's that are out there for long enough will eventually come good I guess. For something to stand the test of time, there must be at least something to it.
Jul 29, 2007 at 4:46 AM
Edited Jul 30, 2007 at 1:54 PM
(continued from above...ran out of space again!)

In terms of ESB.NET, usability will be one of my main points of focus over the next year. Once the service definition and dynamic WSDL & related transport/entry points are resolved, I'll be doing some more VS.NET wizard/template work.
The convention over configuration adaptor released recently is also targetted at that problem, and makes ESB.NET service configuration & deployment a little easier for newbies as well as admins migrating things through environments, makes builds & testing a little easier etc., so that one's a nice recent win for me in that area.

I was also considering a DSL a while ago. This got put on hold when the Windows Workflow Beta came out, as it added a nice dimension to ESB.NET.
Once the dust has settled, I will once again consider if such a thing will add much value overall.


In terms of scalability and extensibility, this has been at the forefront of everything in ESB.NET.
Scalability in terms of performance - latencies, throughput, multiple CPU's, web gardens, server farms.
Also in terms of being distributed, federated, parallel development, partitioning ESB features (eg. XSLT vs. Pub-Sub vs. general service execution etc....) into separate instances, servers, forwarding/routing requests, injecting course aspects (eg. authorization, generic pre-post processing), they're all currently supported.
Similarly, all services are plug-ins, envelopes too follow a provider model (although I need to make a small change here to make it read from config. If you see the code, you'll see it's already separated out like that).
There's also an entrypoint manager which was follows the plug-in pattern, but I had issues early on so this is on the back burner.
Transports are pluggable & usually host the ESB.NET core.
The loggers are again configured in and support the federated model.
ESB.NET itself is multi-instancing within the same OS instance.
Deployment is via xcopy & run a script optionally specifying the instance name (which otherwise defaults to 'source').

Windows Workflow support, convention over configuration support, are themselves plug-ins to the core that can be enabled/disabled, or completely configured out.
The old application configuration & exception management blocks are used to support dynamic config changes if so desired, and orthogonal/aspect separated exception handling.
The service and xslt pipeline's allow you to configure services in a powerful manner, and use data inheritance to ensure you only have to specify the same thing only once, but there's still loads of work to be done here.

Just about everything is pluggable and extensible. Hence, if you look at the assemblies, there's lots of small ones.
This helps localize dependencies.
Jul 31, 2007 at 12:43 AM
Hi again, let me start by saying that I got your point that the ESB should be netural yet adaptable to the current trend/tech in communications or middlewear. I guess I am very much impressed by WCF n' so it is my choice for the client/server implementations - though as long as the ESB supports it, that should be fine.

I saw you've put out a new release on codeplex, I downloading it n' am gonna dive into it and let you know how it goes - hope that is fine? Also, I wanted to confirm one-two things, does the ESB support durable n' reliable messaging. And, do you support any WS-* implementation natively, like for pub/sub do use WS-Eventing or something?

And if you want can share any overview of the code structure, it might be helpful in reviewing it - to me and others.

Cheers,
Rishi


MINAS wrote:
(continued from above...ran out of space again!)

In terms of ESB.NET, usability will be one of my main points of focus over the next year. Once the service definition and dynamic WSDL & related transport/entry points are resolved, I'll be doing some more VS.NET wizard/template work.
The convention over configuration adaptor released recently is also targetted at that problem, and makes ESB.NET service configuration & deployment a little easier for newbies as well as admins migrating things through environments, makes builds & testing a little easier etc., so that one's a nice recent win for me in that area.

I was also considering a DSL a while ago. This got put on hold when the Windows Workflow Beta came out, as it added a nice dimension to ESB.NET.
Once the dust has settled, I will once again consider if such a thing will add much value overall.


In terms of scalability and extensibility, this has been at the forefront of everything in ESB.NET.
Scalability in terms of performance - latencies, throughput, multiple CPU's, web gardens, server farms.
Also in terms of being distributed, federated, parallel development, partitioning ESB features (eg. XSLT vs. Pub-Sub vs. general service execution etc....) into separate instances, servers, forwarding/routing requests, injecting course aspects (eg. authorization, generic pre-post processing), they're all currently supported.
Similarly, all services are plug-ins, envelopes too follow a provider model (although I need to make a small change here to make it read from config. If you see the code, you'll see it's already separated out like that).
There's also an entrypoint manager which was follows the plug-in pattern, but I had issues early on so this is on the back burner.
Transports are pluggable & usually host the ESB.NET core.
The loggers are again configured in and support the federated model.
ESB.NET itself is multi-instancing within the same OS instance.
Deployment is via xcopy & run a script optionally specifying the instance name (which otherwise defaults to 'source').

Windows Workflow support, convention over configuration support, are themselves plug-ins to the core that can be enabled/disabled, or completely configured out.
The old application configuration & exception management blocks are used to support dynamic config changes if so desired, and orthogonal/aspect separated exception handling.
The service and xslt pipeline's allow you to configure services in a powerful manner, and use data inheritance to ensure you only have to specify the same thing only once, but there's still loads of work to be done here.

Just about everything is pluggable and extensible. Hence, if you look at the assemblies, there's lots of small ones.
This helps localize dependencies.


Jul 31, 2007 at 11:53 AM
I agree. Today, WCF is the technology of choice with .NET. Hence the usage of it with ESB.NET.

"I saw you've put out a new release on codeplex, I downloading it n' am gonna dive into it and let you know how it goes - hope that is fine?"
==>
Hehe, yeah, that build seems fine so far.
A couple of little (though significant in terms of usability) things that will be coming out in the next build are:
1)Support for Workflow based Service Handlers in the ConventionOverConfiguration adaptor.
2)A Workflow Project Template in the SDK.
It will be about a week away at most I guess.

"Also, I wanted to confirm one-two things, does the ESB support durable n' reliable messaging. And, do you support any WS-* implementation natively, like for pub/sub do use WS-Eventing or something? "
==>Reliable Messaging is supported via WCF. It is one of the two WCF Configured Endpoints that ship "Out-of-the-box". The other is just the plain unreliable transport.
If you go to the management console default uid=admin, pwd =admin , to the generic request sender page, in the "URL and transport settings" tab, on the rhs, select the "WS-Reliable Messaging" radio button.
This will send the message using the WCF transport (& hence implementation) and use the WS-Reliable Messaging profile.

If, in the request (in the envelope header), you set the "CallSynchronizationType" to 1 (async), the message will be placed in a generic receive queue (as defined in config). This is where the durability comes in.
An MSMQ trigger will then notify ESB.NET over the "ESBHTTPTransportInternalAsync.aspx" transport (which is unreliable!), and process the message. A future release will see this use the WCF Send Transport Adaptor to do this, but I'd like to do it after I've sorted out that delay. In this case it's easier, as the endpoint is static.
In earlier versions (C++/VB6/COM based implementations in 1999-2001, I had the synchronous requests go through this, respond via temp. correlation queues etc., but that feature option is not there in the current version. Initially, performance swayed me away. I'd like to put it in as an option again someday though.).

"And if you want can share any overview of the code structure, it might be helpful in reviewing it - to me and others."
==>Very good queston! That's a big one, and I will elaborate in the "Getting Started Guide" as well as in the book whenever that happens!
In the mean time, it goes something like this:
1)For each instance of ESB.NET, you have the following (well, actually only a portion of the folders below are used at runtime ):
2)Each Instance is partitioned into a "Base" and "LOBSystems". The ESB.NET Core stuff goes in "Base". Your services go under "LOBSystems". This leaves the core relatively clean in case you want to update, re-install etc. and makes configuration management easier.

~\ESB ==>ESB Instance Root
++++++++++++++++++++++++++++++++++++++++++++ ESB.NET Core++++++++++++++++++++++++++++++++++++++++++++++++++++
~\ESB\Base ==> Core Codebase root
~\ESB\Base\Solutions\Main\ApplicationAdaptors ==> ESB.NET shared functionality outside of core codebase.
~\ESB\Base\Solutions\Main\Configuration ==> Shared Configuration Data
~\ESB\Base\Solutions\Main\Documentation ==> Released Documentation so far...
~\ESB\Base\Solutions\Main\Include ==> Shared source files that are compiled into multiple assemblies. Some old crud here too. Need clean-up!
~\ESB\Base\Solutions\Main\Interfaces ==> All core internal and external interfaces used and exposed by ESB.NET. Others are under ThirdParty...
~\ESB\Base\Solutions\Main\Logging ==> Loggers. These are invoked by the core Services to log to the DB, Queues or TraceListener. You can add your own by implementing interface and configure in. Application Blocks/Ent Lib not used. Log4Net not used. Potentially in future, but not any time soon.
~\ESB\Base\Solutions\Main\Management ==> Management Console and Transactions.
~\ESB\Base\Solutions\Main\SDK ==> VS.NET Project Templates and Referenced Assemblies to used by user developed services.
~\ESB\Base\Solutions\Main\Services ==> Currently Not Shipped. Core ESB.NET Pipeline and Manager Functionaliity. Context & ContextManager, SystemEntryPointManager, Service Pipeline, XSLT Pipeline...plus other bits'n pieces.
~\ESB\Base\Solutions\Main\Setup ==> Windows Installers, batch files, WSH scripts, SQL scripts
~\ESB\Base\Solutions\Main\StrongNameKeys ==> Strong Name Keys used by Core.
~\ESB\Base\Solutions\Main\Testing ==> Test Service
~\ESB\Base\Solutions\Main\ThirdParty ==>Third Party Components, MS Application Blocks (slightly modified), MvpXml, PowerCollections...
~\ESB\Base\Solutions\Main\TransportAdaptors ==> Entry Points to the system. WCF, WSE3 nd HTTP Transport Adaptors. WCF has 2 profiles setup by default. You can configure others in using the standard WCF config. You then need to modify the Management Console to include that option in the list of WCF Transports in order to use if from the Management Console. All it does is put the profile (WCF Endpoint) in a custom parameter in the request message, and the WCF Send Transport Adaptor then passes it to WCF.
~\ESB\Base\Solutions\Main\Util ==>Utility assembly. Contains some global settings.
~\ESB\Base\Solutions\Main\Version ==> Version number Assembly used for Versioning the Core Codebase and interfaces. Allows you to change version number in one place, rather than having to change in all assemblies. With VS.NET 2005, even the C# assemblies can now use this!
~\ESB\Base\Solutions\Main\XMLSchemas ==>Contains schemas and XML Serialization classes, envelopes & providers, as well as some generic XML utilities. This needs change - planned for a while now - to allow envelope providers to be specified in config.

+++++++++++++++++++++++++++++++++++++++++++Line Of Business Systems Services+++++++++++++++++++++++++++++++++++++++++++
~\ESB\LOBSystems ==> Recommended Custom / User Services Root
~\ESB\LobSystems\ns ==> ns=namespace, as in, namespace is used to do a match... Root for ConventionOverConfiguration based services.



HTH.
Let me know if you want immediate & specific elaboration in any particular area. As I said above, I will put more & more of this in the Quick Start guide.