Do Agile and be agile

There is agile and there is Agile

One is an adjective and the other is a proper noun.

agile  adj /ˈædʒaɪl/: able to move about quickly and easily; able to think, understand and respond quickly

In business terms, agile refers to dealing with new situations or changes quickly and successfully. And for people dwelling in software craftmanship, it means your mindset and behaviors are inspired by the 4 values and 12 principles stated in the 2001 Agile Manifesto.

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

For example, if the requirement changes midway through a development process, it should be accommodated, perhaps not in the current iteration but in the next. Because the agile mindset believes in collaboration and responding to change over a strict plan no longer reflecting reality.

Note that no place in the manifesto dictates what you need to do to be agile. If a dozen of people who all share this set of values come together and start building software, fat chances are that they will agree on the general direction where they should be going but fight each other at every turn on the fine details. It is because the manifesto is literally a set of loosely defined beliefs. It lacks all the trademarks of the software development process we have grown accustomed to: iterative development, sprint planning, feature backlog, a small amount of WIP, etc.

That is where big 'A' Agile comes in. An Agile process is a methodology, a clearly defined system of practices, whose principles are consistent with the manifesto. There are different flavors of Agile process, the big 4 are: Scrum, Kanban, Extreme Programming, and Lean Software Development (ordered by popularity, judged by myself). What they all have in common is that they take the 4 values and 12 principles and derive a framework encompassing the entire life cycle of software development, from the first seed of an idea to production release. Each of the practices in these frameworks is useful in its own right, but together they are greater than the sum of the parts. They embrace each other and create greater values. Take a humble burndown chart for example, it illustrates a project's progress toward the finish line, but it embraces story point estimation (which is the unit of the chart), user story writing (each story delivers a piece of independent value, and can be released on its own right), and minimize WIP (stories can be tested and accepted as soon as the implementation is done).

src: http://scrumbook.org.datasenter.no/

In the same way constitution and law work, small 'a' agile is inspiring but Agile is what brings people together and allows them to collaborate somewhat efficiently. You are agile and do Agile. Agile was the best thing happening in software development... 20 years ago.

What happened in the last 20 years?

Big 'A' Agile became a victim of its own success. Agile started as a small movement among software development enthusiasts and gained so much traction that it became an industry on its own in which everyone wants a piece. The following awesome map demonstrates how the small movement became a conglomerate of management processes.

There are so many Agile flavors now, I mean, look at the map, that when one claims to do Agile in one form or another, and everyone does, she might as well not say anything at all. To do Agile used to mean the development process follows a certain distinguishable pattern, today it is the equivalent of saying I am breathing.

One more, look at the top left corner of the map, it was made by a consultant. Deloitte, McKinsey, BCG, Scrum Alliance, and a nameless army of certified scrum masters and their dogs have turned Agile adoption into a consulting industry. To ensure their own values and usefulness, these consultants, consciously or not, are the reasons why there are so many Agile flavors, so convoluted, and foreign to even software development professionals. It is believed that was the same reason why religious rites are complicated and foreign to the majority of the population, the priestly people needed to demonstrate their "usefulness". 

Agile under the influence of a profit-seeking industry has been reduced to an empty shell of its former idealization.

FOMO and forceful adoption, many companies now do Agile as if it is just another checkbox in a list. People do Agile while refusing to be agile, they follow the practices mindlessly. We write stories that can't be released on their own, a cluster of stories needs to be deployed and rolled back together. Every 2 weeks we do a Sprint planning but keep both deadlines and scopes, no sign of flexibility is observed. The retrospective meeting is either skipped or used as a gossip forum without improving the working environment or process. 

I have been ranting like agile was pure and good and Agile got spoiled by human greed. But that is not all of it.

The design faults of agile and where we are heading post-agile is a topic I would like to explore in a future article. But look, created some 20 years ago, Agile was made in a world different from the one we are living in today. Back then software was less complicated, written by a smaller team, and managers were unfamiliar with software development hence the need to continuously demo and showcase.

Agile believes collaboration between builders is the key to successful working software and looks down on heavy investment in processes and documents. But as team size gets bigger, complete collaboration also gets expensive. As in, it takes a lot of time if we insist there is no big design phase and the best design is the one emerging during the implementation process. We want to look down on documentation because it is an artifact of bureaucracy but documents make a project long-term maintainable and allow different teams to work with each other, not working software. The world post-pandemic also sees a rise in decentralized teams, collaboration without documents in that context is simply not making the most out of the setting. And we think contracts do nothing good but promote constraints, yet nobody bats an eye at service contract.

It has been a long way to say that in 2023, Agile is not dead but it is less relevant than what it used to be. Seems like agile, as in the set of values and principles captured in the manifesto, shares the same fate despite its lingo difference. And that is the way things should be. In a world that keeps changing, change is the only constant.

ISO 27001:2013 Audit

At the beginning of May, my company went through a 4-day audit for ISO 27001:2013. I was responsible for some parts of the certification process and wanted to write down some thoughts on what I considered an interesting occasion.

Before we proceed, I must make it clear that I wasn't in charge of the entire ISO certification process, that would be our IT Risk & Compliance Manager - Satya. I did however design several systems and engineering processes examined in the audit. I think I can offer insight into what one can expect if she is an engineer going through the certification.

What ISO 27001 is and Why it matters

ISO 27001 is a certification of information security. It sets out the specification of an ISMS (information security management system) and covers people, processes, and technology. First introduced in 2005, it has since received 2 revisions, 2013 and 2022. We decided to go with the 2013 specification because ISO 27001:2022 was out in October 2022 when we were 3 months into the preparation.

Because we are a B2B SaaS, our sales cycle is considerably lengthy with procurement being a major time hoarder. Any enterprise is responsible to ensure that any data subprocessor (which we are) complies with its existing technical and legal obligations. That means our team has to go through pages and pages of questions - both technical and legal, some left you head scratching for hours (very unhealthy especially if you are bald) - just to prove that we know what we are doing when it comes to data security.

International certifications like ISO 27001 cut this cycle short, in the same way IELTS/TOEFT allows you to skip certain English classes. And after decades of enjoying little to no regulation, the last five years have seen a rise in data regulations like data cannot leave a certain geography region or the right to be forgotten, and it is only getting stricter from here. Hence it was a no-brainer for us to do it while we were still at the size where radical changes were still feasible.

Business gains aside, I also wanted to learn from this opportunity. I have been quite verbal about how Parcel Perform is the most complicated system I have ever built and that whatever I knew about large-scale systems, I learned from this very experience through trials and errors and sleepless nights. I couldn't reliably objectively tell whether my work was a state of the art or a big ball of mud, much like any proud mother looking at her child. Once in a while, it is nice to get external validation as well as a chance to learn what is missing.

What to expect from ISO 27001 certification

ISO 27001 is a risk management process. It is more of a thought framework than a checklist. I find it similar to a design pattern. You learn the problem space a pattern excels at then apply it with your own tweak. Between 2 projects though, the implementations of the same pattern might look different. ISO 27001 does the same for risk. Instead of dictating what (not) to do, it gives you a framework to think about risk.

The framework goes like this

1. Given the nature of your business, register the risks associated with it.
2. Assert the level of severity of the risks on your business.
3. Implement a prevention plan for such risks.
4. Post prevent plan, re-assert the new level of severity. The final level of severity must be low enough to allow the business to comply with its SLA

Samples, these were 3 risks we identified and addressed.

All documents related to this risk registration, including policies, practices, system designs, incident records, and whatnot is written, gathered, categorized, and submitted to the audit. Depending on the maturity of an organization, the time this duration takes varies.  For us, it was a few months. As a part of this practice, we had to define some policies and processes that didn't exist before, like Cryptography Policy and Third-party and Supplier Risk Management Policy. One tends to think of information security as protecting against hackers, or at least I do because I watch too many Hollywood movies growing up. There are indeed elements of defending against attack but nothing too paranoid. For example, leaving your computer open and unattended for hours is not safe but a 5-minute lock screen is good enough, despite one of my team member enthusiasm to explain how much he can do in the time window. ISO chooses to focus more on the complete governance view point rather than dictating the precise implementation. Overall though, we got to observe a good case of convergent innovations. Many of our in-house processes responded well to ISO best practices such as the engineering process from inception to deployment, or the access control mechanism.

Upon submission, on-site audits were scheduled. By default, ISO 27001 certification is issued per location. Singapore and Vietnam are places where our product is built and data analyzed so these are the main targets of the audit. Sales offices were skipped as data procession is not a part of their functions.

The audit exists essentially as proof: the documents describe robust processes, yet would it crack under pressure and are we doing what we preach? The audit was performed in a series of on-site interviews between the auditor and the people in charge over particular topics like access control, software development process, incident management, and more. These were just the ones in which I played an active role. I noticed that the audit conducted these interviews in two ways. For small isolated topics such as access control, we went through the entire process before he probed with questions and we countered with reasons why the decisions we made fit our circumstances the best. For longer topics, like an end-to-end software engineering process, he went straight to the Q&A dance. That left us mildly disappointed because we even had mock interviews for those long sessions :)

Findings during the audit are categorized into major non-conformity, minor non-conformity, and suggestions. In theory, as long as you can address all the non-conformities before the final audit day, you are good. For a multi-site audit, the end-to-end time can be a few weeks. In practice, a major non-conformity is pretty much a no-go. It indicates the lack of a mission-critical process without which information security effectiveness can't be achieved. Such a process takes time to form and even longer to put into practice and become an organization's second nature. In fact, I believe not having enough time in practice is the number one reason a quick attempt to rectify a major non-conformity is rejected. Minor non-conformities are quite easier, they indicate points of improvement in an established process and enjoy a much higher chance of being accepted if you can get them in in time. Suggestions are just what they are, non-consequential pieces of advice from the auditor based on his experience with other organizations.

We didn't discover any non-conformity, major or minor, so we are expecting our certificate to come through soon. But that isn't the end of it. Under normal circumstances, the ISO certificate must be renewed annually and only lasts for 3 years before the organization has to go through the full circle again. For ourselves, because ISO 27001:2013 is set to expire in 2025, we will be resetting our certificate with 2022 specifications next year.

Conclusion

Certification being a lucrative business as it is, I do find the whole process to be a positive learning experience.

• There are the thought framework approach and the big to-do list approach when it comes to certification.
• Hand waving can take you places but ISO like many certifications is one where the journey is more important than the destination.
• There are many things that we have already done right with the system and the organization around it, the external validation is a great encouragement to our effort.
• Our documentation collection can see more rigorous standards, though everything was produced correctly, it took a bit of time to gather them for the audit proof.