Agile & Architecture

With this post I am trying to address the biggest myth about Agile, "We are agile, we don't need architecture or design". Let me start by supporting the myth first. For a  product that has been in the market for 10 years or more, this is not all myth but reality. For a product that has existed for such as long time, there is probably no need to do any design or architecture anymore. It was probably designed, when it was first conceived or when it underwent major upgrade. It may not have been formally documented but its creator must have had a white boarding session and picture on the whiteboard. It may have been formally documented as part of the product documentation for developers. However as it matured and evolved to a certain stage, it stopped changing much. The only change it undergoes nowadays is cosmetic or for superficial features. Under this scenario I would agree to a statement I heard from a supervisor, a self-proclaimed agile champ, when somebody in the team proposed design. "It's a waste of time, we are agile". The supervisor probably hadn't seen the glory days of the product when it underwent major upgrades / releases. Under such circumstances design was inevitable, whether it was documented or not. So, yes, I would agree that for minor product updates, especially superficial or cosmetic ones there is no need to document the design.

However if you are into serious business of creating a new mission critical product or project for developing a new, "green field" system that involves integration with many other internal and external systems coupled with complex business requirements, then you still think architecture can be avoided? You may be able to avoid documentation but even that becomes a risk, if communication is insufficient for some reason like when the development teams are distributed across locations. Lacking documentation becomes a project risk. Neither architecture nor documentation is avoidable for projects of such huge scales. Blueprint becomes necessity. Imagine constructing an airport without having any blueprint, I am afraid that people who believe that architecture & design is a waste of time, since they are "agile" are not into serious business. They are most likely doing work that is so minor in the scale that it doesn't really matter to the enterprise or to anyone. So next time your hear this question of whether the design is necessary, you have every reason to doubt the scale and importance of such a project. And if you are convinced about the scale of the project then you have every reason to doubt the credentials of the person who believes that blueprint is not required.

Talking about the documentation of the architecture, we need to document just enough for the developers and other stakeholders to understand. To the people who say "source code is the best documentation", I would say, "Well try selling an airplane without a maintenance manual." My next question would be, "Have you ever tried to understand a system just by looking at the source code and nothing else?". I am asking this because I have tried that and realized its really very hard. The people who think that "source code is the best documentation" haven't really seen huge systems. Once the LOC crosses a few million lines it start making no sense. I even tried reverse engineering a UML model from the source code and saw a spaghetti mesh right in front of me. If you haven't come across such huge code bases, then try this, find some source code that parses or generates a complex XML document. Now just by looking at the source code try imagining the structure of the XML document. Of course the assumption is the XML document is complex as we are talking about large scale projects. So my advice to those people who say source code is the best document is, "Get Real". Work on real-life projects that involves complexity. And so if we want to document what should we be documenting? And the answer is blueprint. Instead of thousands of words and subjectivity laden textual description, have UML models. Starting with the block diagrams, iteratively go on adding details to it, till you get actionable UML models. Text should only be used for adding what is not obvious from the models instead of describing the model in words. Provide models for each aspect of the system such as functionality, security, maintainability etc.

If you are working on a Agile project that does not have a formal design phase then use Sprint 0 to prepare the block diagram and then in every subsequent sprint go on adding more details. The architecture model should be put on a wiki-page so that the entire team has access to it and can change it as needed. This is agile documentation. One important thing to note about doing architecture in agile - architecture is waterfall to a large extent. I see smile erupting on the faces of the "agilists". By the very nature, architecture deals with things that are hard to change like the technology platform. Just because you are following the agile development process doesn't mean you define or change architecture in every sprint. You have to design the architecture in sprint 0 and then "evolve" it in every sprint by adding details. You cannot change your choice of the technology platform or programming language in every sprint or iteration, can you? And that's why architecture is waterfall. Business requirements and its implementation are agile as these can and often change. However, architecture is like business domain, like you can't change your business domain, as in you can't think of changing from manufacturing cars to gaming consoles overnight, you can't think about changing from Cobol to Swift overnight.

I will suffice it to say here that product development without architecture and design is short sighted, myopic, knee-jerk reactive stint at development. If you are still not convinced please watch:

https://www.youtube.com/watch?v=DngAZyWMGR0
https://www.youtube.com/watch?v=VjKYO6DP3fo

Agile Architecture

In the fight between techno savvy, state of the art, high tech features and economics & profitability. It is the later that always wins. No matter how grand your vision of architecture in the enterprise is, it is the business team that always wins. And therefore does it even make sense to have a vision and technology strategy, enterprise architecture in the first place if we know that it is the viability of the business that wins? I think most will agree that we must have a vision and technology road map, enterprise architecture in place. Without which it is like walking blindly on a ground full of obstacles towards a target location. With rapidly changing & competitive world only those win who offer more to the customers than their competitors. This narrow space of offering just enough to lure the customers and outsmarting the competition is often ruled by technology. Like those Banks who offered mobile banking early on versus those that did not. If the technology never mattered then how come we see so many banks offering mobile banking. And so the conclusion is, it does matter. Well it does matter but we also agree that its very hard to get hi-tech features in the business plan and get budget allocated for it while that technology is not already a common trend in the market . My solution to this problem is, Agile Architecture. Under this approach, you pick low hanging fruits from your enterprise architecture vision and sneak them into the upcoming projects where they are best aligned and can be delivered at the minimal cost. For this you need to keep an eye on the project pipeline and you of course have your vision firmly on your EA radar and then you marry the two when you see a best time & fit. This is what I call Agile Architecture. Architects who have a grand vision that requires management to spend millions of dollars across number of years will never succeed, it is those who master the art of sneaking the components of their vision into run-of-the-mill everyday projects will win. Here's an example, you have a grand vision of Event Driven Architecture where the systems in your organization exchange important events across the enterprise asynchronously but ground reality is some of the systems in the organization still run batch jobs and only way to communicate with these systems is once a day when the batch runs. However you see a project in the pipeline where copies of a certain attribute or an object in different system need to be updated at the same time and stay in synch from that point onward, for example customer's phone number. You take that as a opportunity and propose a very scaled down version of your EDA just for the systems involved in the upcoming project but you ensure that the scalability of the design by using robust communication infrastructure such as a messaging middleware instead of a point-to-point call based interface between the two systems that the project requires. You further justify the additional investment by referring to your vision and benefits that the design will bring eventually. With this you do stand a chance that you will materialize part of your vision and then incrementally & iteratively you can realize your full vision. This is the way of the Agile Architecture.