Lightning Solutions

This post is published on the Confluent blog: https://www.confluent.io/blog/apache-kafka-security-and-governance-with-control-and-data-planes/

With this post I am trying to address the biggest myth about Agile, "We are agile, we don't need architecture or design". Let me start by supporting the myth first. For a  product that has been in the market for 10 years or more, this is not all myth but reality. For a product that has existed for such as long time, there is probably no need to do any design or architecture anymore. It was probably designed, when it was first conceived or when it underwent major upgrade. It may not have been formally documented but its creator must have had a white boarding session and picture on the whiteboard. It may have been formally documented as part of the product documentation for developers. However as it matured and evolved to a certain stage, it stopped changing much. The only change it undergoes nowadays is cosmetic or for superficial features. Under this scenario I would agree to a statement I heard from a supervisor, a self-proclaimed agile champ, when somebody in the team ...

It was late 2007, world was on the brink of credit crisis. I was a lead architect on a project for a large American bank on a consulting assignment. The project required generating and consuming SWIFT messages and I was thinking hard on how to make it easy for my team to exchange SWIFT messages as part of the system that was being developed. After searching around I could not find a good software or a framework that would do the job and so I set out to design one myself. I soon realized that composing SWIFT messages element by element was going to be hard as it required the developers to understand the rules and grammar of SWIFT. If I were to ask my developers to learn SWIFT and then work on the module of the system that would be responsible for exchanging SWIFT messages, I know I would be on a suicidal mission. Most of the developers in my team were novices, let alone understanding SWIFT they didn't even know enough programming. You know how the offshoring companies work, its all...

In this post I am attempting to summarize the SOA patterns that were published in the book, "SOA Patterns" by Amon Rotem-Gal-Oz. This is based completely on my understanding of the matter presented in the book and there is a possibility of a gap in my understanding, if so, please feel free to correct me by leaving a comment. Please also note that this is not the only book on SOA patterns but there is another one, "SOA Design Patterns" by Thomas Erl. The reason I decided to read the book by Amon Rotem-Gal-Oz is because it has a foreword from one of my favorite authors, Gregor Hohpe, one of the author of Enterprise Integration Patterns. With this post let me discuss the patterns presented by Rotem-Gal-Oz. For the patterns from Thomas Erl I may write another post. The very reason I am writing this post is to make life easier for those who don't have time to read  through 300 page book on SOA patterns but want to have a very high level & quick understanding o...

We were working on a batch job and the requirement was to finish processing all the transaction in a database within a maintenance window of less than a hour. The requirement was to read hundreds of thousands of records, call a service and validate and enrich those records using reference data returned by the service (Back then we had no web services, we had CORBA or IIOP (Java RMI/IIOP) to invoke those services). Now the challenge was the time window that was very brief. We figured out that reading the records sequentially was not an option and we had to process those records in parallel. So I proposed we could use batch jobs running in multiple processes spread across multiple nodes to read and process the records from the database, the slaves or Yes, the "Mappers"!  as in MapReduce. Now how could we prevent the parallel batch jobs from stepping on each others toes and avoid picking up the same set of records for processing. How could we avoid this duplication? use row l...

I have been a practicing software solutions architect for more than 8 years now and my total IT industry experience is close to 15 years. During these years I have designed the architecture of many small to large systems. I come across people from varying backgrounds during my work and many of whom even though industry veterans did not quite understand the term Architecture. For many of them Architecture means block diagrams that you prepare after requirements and before design, while others thought that Architecture is a simplified view of design for people from non-technical background and yet others thought that it was a warm-up exercise before design. All these myths are valid and I don't blame the myth-holders. The main reason behind these myths is that people do not understand what is Architecture and how it is designed and what exactly Architects do. And so here goes an attempt to clear myths around what exactly is Architecture by showing how it is prepared more so fro...