OK, so high complexity is Evil (and even physicists are getting in on it - here).
In the last few posts I've identified (IMHO) the underlying principles of a well-designed system - high cohesion, low coupling, high conceptual coherence, and good layering.
These are good principles because they aid in reducing system complexity, and low complexity is good.
Today there are many enterprises with a plethora of business processes supported by a host of different systems, perhaps interfacing to one or more legacy applications, all on heterogeneous hardware. To allow information to be distributed, there are numerous point-to-point interfaces using various means such as ETL batch processes, file shares, database shares, messaging, and various styles of RPC.
Just reading that should make you cringe, and most organisations deal with it by making it worse.
Service oriented architecture is the latest attempt at fixing this mess by making a simple observation. Each system (and I'm including the business processes associated with the system) is providing a set of services, and by identifying those services and the contracts by which they are invoked we should be able to abstract the service definition and its interface from the implementation of the service.
Now, the ability to abstract the service definition and interface from its implementation means that you can decouple the systems and introduce something else - like a messaging platform or ESB - to interface the systems. The has the potential to immediately reduce the number of interfaces (there are many traps, outside the scope of this post) by adopting, say, a hub-and-spoke architecture and hence reduce the complexity.
Done badly, of course, it can also increase the complexity! But we won't go there.
Add a few heuristics to take into account the fact that you're dealing with people (such as governance), and a couple of good ideas (like discoverability and interface self-description), and voila you have SOA. Add even-driven ideas (because firing events is a good decoupling mechanism), and you have SOA + EDA.
Most SOA proponents would generally agree that all the above are incorporated into current SOA best practices. Unfortunately, even by adhering to the above principles we're still going to end up with the problems of Company C! In my next post I'll explain why.
Sunday, May 24, 2009
Saturday, April 4, 2009
Managing Complexity
Given that high cohesion and low coupling are necessary but insufficient properties of a well-designed system, what else is required?
We need at least one more - namely 'conceptual coherence'.
I'll attempt to explain myself.
A few months ago I had the pleasure of reading a book by Roger Sessions on Enterprise Architecture that contained a bunch of ideas that resonated deeply with me. The one Big Idea in the book that I liked very much was the concept of complexity, and in particular the importance of minimising it. Its a principle I've tried to follow closely since entering this field (sometimes unsuccessfully, particularly during the early years of my career), and it has served me well.
We need at least one more - namely 'conceptual coherence'.
I'll attempt to explain myself.
A few months ago I had the pleasure of reading a book by Roger Sessions on Enterprise Architecture that contained a bunch of ideas that resonated deeply with me. The one Big Idea in the book that I liked very much was the concept of complexity, and in particular the importance of minimising it. Its a principle I've tried to follow closely since entering this field (sometimes unsuccessfully, particularly during the early years of my career), and it has served me well.
Much of the advice contained in the book follows from his "Five Laws of Partitions", where a partition is a means to compartmentalize a mathematical set. The primary aim when applying these 'laws' is to minimise complexity. In addition to providing a consistent and rational means to analyse a system, the 'laws' provide a foundation for managing and removing complexity and result in heuristics on how to go about analysing and partitioning an existing system.
Those familiar with his background won't be surprised that Roger also makes use of Type (or Classification) Theory - concepts that will be familiar to any skilled (or unskilled, for that matter) OO practitioner. Combining Type Theory with the "Five Laws of Partitions" is really just a means to discover system properties that are 'invariant under analysis' (IUA) - different people analysing the same system or organisation will, using his methods, theoretically devise very similar architectures and be able to rationally explain the reason for its structure.
System properties will be IUA is if the identified types exhibit a high degree of conceptual coherence. To put it a different way - each identified Type in the system must have properties (concepts) that are tightly bound to each other, and simultaneously be loosely bound to other sets of properties. Conceptual coherence is important when conveying information to other people, as conceptually coherent concepts are easy to convey. For example, "Four-legged cat" is a concept everybody will understand. "3-legged cat with gills" doesn't gell too well as the concept of "cat" is being strained.
System properties will be IUA is if the identified types exhibit a high degree of conceptual coherence. To put it a different way - each identified Type in the system must have properties (concepts) that are tightly bound to each other, and simultaneously be loosely bound to other sets of properties. Conceptual coherence is important when conveying information to other people, as conceptually coherent concepts are easy to convey. For example, "Four-legged cat" is a concept everybody will understand. "3-legged cat with gills" doesn't gell too well as the concept of "cat" is being strained.
Now, coming back to why low coupling and high cohesion are important - devising and/or analysing a system architecture using the above rules, while adhering to the principles of high coherence and low coupling, is related to Principal Component Analysis, The Spectral Theory, Eigendecomposition, and Canonical Forms. In particular by analysing a system using a Design Structure Matrix an optimally designed system will ideally have a lower diagonal structure. The ability to determine those aspects of the system that are IUA in essence reflect a 'canonical' form of the system.
When you think about it, reality adheres strictly to this model; elements and compounds are built on molecules, which are built on atoms, that depends on baryons, which in turn are made from quarks. Biological systems are an even better example; with the aid of natural selection and millions of years of evolution, nature has created systems of phenomenal complexity and yet astounding simplicity.
When you think about it, reality adheres strictly to this model; elements and compounds are built on molecules, which are built on atoms, that depends on baryons, which in turn are made from quarks. Biological systems are an even better example; with the aid of natural selection and millions of years of evolution, nature has created systems of phenomenal complexity and yet astounding simplicity.
Tuesday, January 27, 2009
What is SOA?
SOA - Service Oriented Architecture ... what exactly does that mean?
Its an unfortunate fact within the IT industry that there is no single, definitive definition of 'Service Oriented Architecture'. In the early days it was often conflated with Web Services, which is not unreasonable as many resources on the web invariably include Web Services somewhere within their definition of SOA. However, today there are few who still adhere to that view, and some are even amenable to the use of REST.
I'm not going to define SOA here - there's plenty of definitions out there already. Instead, I'm going to take a different tack and consider what we're trying to achieve with SOA. Why are companies attempting to re-engineer their IT infrastructures to be coherent with this vision of 'SOA'?
System Design Principles
System Design is closely related to Systems Theory, being the application of Systems Theory to the development of systems. System Design is a complex subject; however, in its application to software-based System Design it fundamentally comes down to the following principles when defining individual components:
A perfect example of a high-cohesion, low-coupling component is a lego brick. It has high-cohesion by the mere fact that it doesn't tend to spread itself around (unlike, say, air), so you won't find bits of it in places where they don't belong doing things they shouldn't be doing. It has low coupling due to the fact that it physically only has two interfaces that interact with other lego blocks - one that you can plug lego blocks into, and one that can accept lego blocks. Due to this exquisite combination of simplicity and utility, it is capable of being used to construct an infinite number of designs in conjunction with other lego blocks.
However, due to the fact that it has a well-defined set of capabilities (a result of its high-cohesion), this simplicity also imposes limits. A lego block cannot, for example, be used to construct electrical circuits. To build electrical circuits you need resistors, diodes, transistors, capacitors, conductors, and inductors - not lego blocks.
High cohesion and low coupling of components appears to be common in the construction of complex systems, whether they be software systems, computing systems, biological systems, or manufacturing systems. For example, object-oriented design heuristics have high-cohesion and low-coupling as two important properties for well-specified classes, modules and subsystems. In biological systems such as the human body, cells are specialised towards specific functions and predominantly localized to certain organs. The organs, have specialised functions and interface with the remaining body using well-defined interfaces (how else could a surgeon simply replace one heart with another?).
Are high cohesion and low coupling the only principles needed? Not quite ...
Its an unfortunate fact within the IT industry that there is no single, definitive definition of 'Service Oriented Architecture'. In the early days it was often conflated with Web Services, which is not unreasonable as many resources on the web invariably include Web Services somewhere within their definition of SOA. However, today there are few who still adhere to that view, and some are even amenable to the use of REST.
I'm not going to define SOA here - there's plenty of definitions out there already. Instead, I'm going to take a different tack and consider what we're trying to achieve with SOA. Why are companies attempting to re-engineer their IT infrastructures to be coherent with this vision of 'SOA'?
System Design Principles
System Design is closely related to Systems Theory, being the application of Systems Theory to the development of systems. System Design is a complex subject; however, in its application to software-based System Design it fundamentally comes down to the following principles when defining individual components:
- High Cohesion
- Low Coupling
A perfect example of a high-cohesion, low-coupling component is a lego brick. It has high-cohesion by the mere fact that it doesn't tend to spread itself around (unlike, say, air), so you won't find bits of it in places where they don't belong doing things they shouldn't be doing. It has low coupling due to the fact that it physically only has two interfaces that interact with other lego blocks - one that you can plug lego blocks into, and one that can accept lego blocks. Due to this exquisite combination of simplicity and utility, it is capable of being used to construct an infinite number of designs in conjunction with other lego blocks.
However, due to the fact that it has a well-defined set of capabilities (a result of its high-cohesion), this simplicity also imposes limits. A lego block cannot, for example, be used to construct electrical circuits. To build electrical circuits you need resistors, diodes, transistors, capacitors, conductors, and inductors - not lego blocks.
High cohesion and low coupling of components appears to be common in the construction of complex systems, whether they be software systems, computing systems, biological systems, or manufacturing systems. For example, object-oriented design heuristics have high-cohesion and low-coupling as two important properties for well-specified classes, modules and subsystems. In biological systems such as the human body, cells are specialised towards specific functions and predominantly localized to certain organs. The organs, have specialised functions and interface with the remaining body using well-defined interfaces (how else could a surgeon simply replace one heart with another?).
Are high cohesion and low coupling the only principles needed? Not quite ...
Monday, January 26, 2009
Services and Architecture
Ah yes - I remember the first time the term "services" was uttered in my presence, but without "architecture" attached. It was around 2000 or 2001 (my memory is rather sketchy on details!), and I was in a meeting with other 'architects' discussing shared components to be used across the enterprise. Someone cottoned on to the idea that we weren't really talking about components - we were talking about sharing functionality that could be invoked from anywhere; in other words, services. It was an interesting period for me, as it was intellectually challenging. However, it was also incredibly non-productive, as the architecture group seemed to be engaging in little more than intellectual masturbation. Looking back, it was probably the "forming / storming" stage for the group, so no real surprises.
Today, I wonder whether the organization still has meetings where there is lots of high-level discussion around 'Service Oriented Architectures', shared services, and governance ... with little to show for it but the obliteration of vast tracts of the Amazon.
Anyway ... what the company did have, however, was a sufficient number of 'idle' development groups such that almost all new and successful IT implementations were a series of interacting skunkwork projects (in fact, they've recently released an application to a very popular mobile platform).
I'll call this company A.
At the other end of the spectrum, I've come across organizations where they've well and truly gone past the point of discussion and have created an almost endless list of 'services'. Regrettably, there are few services consumed by more than one client as each service has essentially been designed to fit a single purpose. Not only that, but each application uses a different message format for information or services that could relatively easily be shared across applications. It works, to a degree. But without governance, the level of sharing is minimal to zero.
This is company B.
I've personally come across only one organisation where common services were developed and used by others within the organisation. They had all the hallmarks for the beginnings of a successful enterprise-wide SOA implementation - except that their cost to income ratio for IT services was approximately three times that of company A, and many more times that of company B. In addition, senior management was continuously complaining about the lack of flexibility in the provision of new products and services, and couldn't believe how much it was all costing.
This is company C.
Given that company C is, according to current approaches, IT nirvana, why is it faring so badly compared to A and B? To see why, we need to go back to first principles and take another look at why SOA is seen to be the next step for most large IT shops. We're also going to be slightly heretical and question whether SOA in its currently understood form really can deliver all that the vendors and consultants are promising.
I'll leave that for the following posts!
Today, I wonder whether the organization still has meetings where there is lots of high-level discussion around 'Service Oriented Architectures', shared services, and governance ... with little to show for it but the obliteration of vast tracts of the Amazon.
Anyway ... what the company did have, however, was a sufficient number of 'idle' development groups such that almost all new and successful IT implementations were a series of interacting skunkwork projects (in fact, they've recently released an application to a very popular mobile platform).
I'll call this company A.
At the other end of the spectrum, I've come across organizations where they've well and truly gone past the point of discussion and have created an almost endless list of 'services'. Regrettably, there are few services consumed by more than one client as each service has essentially been designed to fit a single purpose. Not only that, but each application uses a different message format for information or services that could relatively easily be shared across applications. It works, to a degree. But without governance, the level of sharing is minimal to zero.
This is company B.
I've personally come across only one organisation where common services were developed and used by others within the organisation. They had all the hallmarks for the beginnings of a successful enterprise-wide SOA implementation - except that their cost to income ratio for IT services was approximately three times that of company A, and many more times that of company B. In addition, senior management was continuously complaining about the lack of flexibility in the provision of new products and services, and couldn't believe how much it was all costing.
This is company C.
Given that company C is, according to current approaches, IT nirvana, why is it faring so badly compared to A and B? To see why, we need to go back to first principles and take another look at why SOA is seen to be the next step for most large IT shops. We're also going to be slightly heretical and question whether SOA in its currently understood form really can deliver all that the vendors and consultants are promising.
I'll leave that for the following posts!
Tuesday, August 19, 2008
Complexity Shmetzity
Software is complex.
We all know that.
So why do we insist on making things even more complex than they need to be?
During my career I've noticed a trend in software development, and its not a trend I think is always intrinsic to the problems that we are trying to solve. You've no doubt cottoned on to the fact that I'm alluding to complexity, and in particular the increasing complexity of the systems that are designed and built.
In software development its always easier to develop a complex solution to a complex problem than it is to develop a simple solution to a complex problem. When designing a solution there are many forces to consider, pushing the design parameters to and fro. You're exploring the solution space, looking for design approaches that hopefully descend down into some kind of optimum (having a good wander around the area is generally a good way to start). Unfortunately, some architects don't seem to see it that way. Instead, they see Mt Olympus over in the distance and want to scale it ... its easy to spot, you know where you're going (even if it is vertically straight up) and dammit, its going to be fun! This way team, I know what I'm doing! We've got all the right tools; we've got design patterns, and services, and layers and ...
Oh dear.
So where am I going with this? Well, nowhere really. Its just a bit of a moan.
We all know that.
So why do we insist on making things even more complex than they need to be?
During my career I've noticed a trend in software development, and its not a trend I think is always intrinsic to the problems that we are trying to solve. You've no doubt cottoned on to the fact that I'm alluding to complexity, and in particular the increasing complexity of the systems that are designed and built.
In software development its always easier to develop a complex solution to a complex problem than it is to develop a simple solution to a complex problem. When designing a solution there are many forces to consider, pushing the design parameters to and fro. You're exploring the solution space, looking for design approaches that hopefully descend down into some kind of optimum (having a good wander around the area is generally a good way to start). Unfortunately, some architects don't seem to see it that way. Instead, they see Mt Olympus over in the distance and want to scale it ... its easy to spot, you know where you're going (even if it is vertically straight up) and dammit, its going to be fun! This way team, I know what I'm doing! We've got all the right tools; we've got design patterns, and services, and layers and ...
Oh dear.
So where am I going with this? Well, nowhere really. Its just a bit of a moan.
Subscribe to:
Posts (Atom)
