Taipei - 13 years later

The HSR train is cruising through the south of Taiwan at 200km an hour. I have been on the island this last week and now heading to Kaohsiung for the way back. Inside the car, the stability is excellent, there is nothing but an eerie sound reminding me of the neck-breaking speed a few cm outside. Through the window, a constant flow of paddy fields scrolls by. Unlike in the US where machine power beats nature into submission, the horizon is perfectly flat, and all the fields are endless rectangular shapes. In this part of the world, agriculture forms around the ancient landscape. The rice fields are small and entangled in a net of canals even smaller. Embracing the late summer sun, the fields are green and the stems are round, juicy, and straight as there are no grains to bear yet. It is like looking at the skin of a cantaloupe. My mind though has stayed in Taipei for the last few days, where autumn was hard at work, putting her gentle touch on everything. The sun painted everything with the color of honey, the air was clear, and the breeze was crispy. In that crystal clarity, my younger years came back to life.

The metropolis of Taipei is a conglomerate combination of three cities - it is usual for an Asia country to have one big city overshadowing everything else in the land. In that concrete jungle, I lived abroad for the first time, got my first full-time job - an internship, and learned to juggle between that and a Chinese language degree. Because of course, Taiwan was strict on immigrant workers and they had better things to do than issuing visas for low-level interns. The only way I could stay on the island for more than a month in that situation was to become a uni student. Or married to a Taiwanese sugar mommy. It was tough, exciting, and mesmerizing in the following years, the internship, not the marriage.

Taipei is 2229km away from Saigon. It is not far from the length of Vietnam, 1650km from the tip of the north to the southernmost peninsula. The distance seems much further than that though. There have always been other places to be, work to do, and COVID put everything on ice for a while. This is my first time coming back to Taiwan after 13 years.

I made exactly 15000NT a month back then. I never for once asked if it was a fair income. For what it's worth, I also didn't stop to think if I had the right clothes for Taipei weather or if people there spoke English till I was on the airplane. The 20-year-old me ran on adrenaline rush much more than rationality if it hasn't been clear to you yet. By the way, no, I didn't have the right clothes, it got to -2 Celsius during Winter, and yes you could survive with only English if you only stayed near the city center. I made my financial plan (or rather the lack of it) around this precious number. The office was a converted studio and after dark it actually became my studio. I had a pull-out mattress. There was a cracked XBox in the office and a 7-11 on the ground floor. The boss, Jake, lent me his daughter's old bicycle. I never owned a game console and 7-11 didn't come to Vietnam some 6-7 years later. For what I cared, I was the richest kid in the world. Life was bliss.

A few days ago, I had a reality check. A Uniqlo storefront entry-level staff makes around 36000-42000NT. I was not rich. Actually, if poverty was a horizontal bar, I was killing it in a limbo dance. That explains why I had a vague memory of the MRT system, I was on the bicycle most of the time. I went to many museums and historical sites because the student discount was high. I rarely made it out of the city and the fine dining scene of Taipei was as foreign to me as I was to this country. Yeah I was pretty broke, wasn't I?

I didn't mind it then, and I still don't mind it now, post-reality check. When I think about the whole deal now, I don't feel like I was given the short end of the stick. I had a reasonably comfortable place to sleep whose rent I didn't have to pay. Besides my compensation and the old bicycle, Jake also paid for my tuition fee and often took me out for lunch. I was just low on disposable income. But to be frank, that was an afterthought. The 20-year-old me was blessed with the exposure I never experienced. Every day was a new situation I hadn't faced before. Every weekend was an adventure away. Every project was a stark contrast to the unrealistic school assignments that by then were all that I knew. Luxury and consumerism were not only out of my reach, they were also out of my mind. Without even being conscious about it, I was able to experience a learner mindset in its purest form. I managed to carry these starry eyes, and a dash of stupidity if I must admit, throughout my 20s. I am glad I got my first step right.

On another note, if Taipei had a face, it would be probably the face of a man in his 50s. A man who still maintains all the best qualities from his younger years, but subtly through the cracked skin around the eyes or the way he stands up, it feels like his best years were slipping away. Singapore would be in his late 30s, a force of nature with so many ideas and the energy to see it through. Saigon a kid in his early 20s, all talks, so few deeds.

The pace in Taipei is slower, and people know how to enjoy themselves. It is quite easy to find a park in a random neighborhood and both banks of the Keelung are reserved for outdoor activities. You don't see the elderly working, you are more likely to find them doing taichi in parks. The nightlife is vibrant with night markets. Alcohol and recreational drugs, for better or worse, are more accessible. And the entertainment industry punches above its weight. But I can't help to think of Taiwan as a country trapped in the past. The country was founded on a false hope that one day the Republic of China would be whole again. And today people long for the Taiwan Miracle that probably wouldn't happen for the second time. East meets West, future hope fuses with nostalgia, the more I learn about Taipei, the more elusive the words I want to find to describe my fondness.

During this trip, the one thing I thought a lot about yet still failed to fulfill was to see Jake for what could have been the last time. Jake was my boss at Cogini. He gave me first an internship and second a chance to run Cogini's office in Saigon. I wouldn't be who I am to be without the opportunities Jake presented and I eagerly grasped. Jake however was also an embodiment of the story a great engineer and a greater friend doesn't always make a great manager. That's the story for another time. I wanted to meet Jake because really how many friends of 13 years one can have. My excitement was met with an unfortunate event, Jake went back to the US last year to look after his mom. It makes sense though, his daughters went there for universities a few years ago and now he can be closer to his family. I would miss him though, now that the harsh reality has set in, I probably wouldn't be able to see him again.

I don't know when would be the next time I will be in Taipei, the world is so big and there are so many things to see. I sure hope it won't take another 13 years in the making. And that next time, I will just stay in Taipei for weeks. To find my younger self. To soak in all that nostalgia that by now has become the city's identity in me. To see that I am one more time its citizen.

The flight back to Vietnam is in a few hours.

Do Agile and be agile

There is agile and there is Agile

One is an adjective and the other is a proper noun.

agile  adj /ˈædʒaɪl/: able to move about quickly and easily; able to think, understand and respond quickly

In business terms, agile refers to dealing with new situations or changes quickly and successfully. And for people dwelling in software craftmanship, it means your mindset and behaviors are inspired by the 4 values and 12 principles stated in the 2001 Agile Manifesto.

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

For example, if the requirement changes midway through a development process, it should be accommodated, perhaps not in the current iteration but in the next. Because the agile mindset believes in collaboration and responding to change over a strict plan no longer reflecting reality.

Note that no place in the manifesto dictates what you need to do to be agile. If a dozen of people who all share this set of values come together and start building software, fat chances are that they will agree on the general direction where they should be going but fight each other at every turn on the fine details. It is because the manifesto is literally a set of loosely defined beliefs. It lacks all the trademarks of the software development process we have grown accustomed to: iterative development, sprint planning, feature backlog, a small amount of WIP, etc.

That is where big 'A' Agile comes in. An Agile process is a methodology, a clearly defined system of practices, whose principles are consistent with the manifesto. There are different flavors of Agile process, the big 4 are: Scrum, Kanban, Extreme Programming, and Lean Software Development (ordered by popularity, judged by myself). What they all have in common is that they take the 4 values and 12 principles and derive a framework encompassing the entire life cycle of software development, from the first seed of an idea to production release. Each of the practices in these frameworks is useful in its own right, but together they are greater than the sum of the parts. They embrace each other and create greater values. Take a humble burndown chart for example, it illustrates a project's progress toward the finish line, but it embraces story point estimation (which is the unit of the chart), user story writing (each story delivers a piece of independent value, and can be released on its own right), and minimize WIP (stories can be tested and accepted as soon as the implementation is done).

src: http://scrumbook.org.datasenter.no/

In the same way constitution and law work, small 'a' agile is inspiring but Agile is what brings people together and allows them to collaborate somewhat efficiently. You are agile and do Agile. Agile was the best thing happening in software development... 20 years ago.

What happened in the last 20 years?

Big 'A' Agile became a victim of its own success. Agile started as a small movement among software development enthusiasts and gained so much traction that it became an industry on its own in which everyone wants a piece. The following awesome map demonstrates how the small movement became a conglomerate of management processes.

There are so many Agile flavors now, I mean, look at the map, that when one claims to do Agile in one form or another, and everyone does, she might as well not say anything at all. To do Agile used to mean the development process follows a certain distinguishable pattern, today it is the equivalent of saying I am breathing.

One more, look at the top left corner of the map, it was made by a consultant. Deloitte, McKinsey, BCG, Scrum Alliance, and a nameless army of certified scrum masters and their dogs have turned Agile adoption into a consulting industry. To ensure their own values and usefulness, these consultants, consciously or not, are the reasons why there are so many Agile flavors, so convoluted, and foreign to even software development professionals. It is believed that was the same reason why religious rites are complicated and foreign to the majority of the population, the priestly people needed to demonstrate their "usefulness". 

Agile under the influence of a profit-seeking industry has been reduced to an empty shell of its former idealization.

FOMO and forceful adoption, many companies now do Agile as if it is just another checkbox in a list. People do Agile while refusing to be agile, they follow the practices mindlessly. We write stories that can't be released on their own, a cluster of stories needs to be deployed and rolled back together. Every 2 weeks we do a Sprint planning but keep both deadlines and scopes, no sign of flexibility is observed. The retrospective meeting is either skipped or used as a gossip forum without improving the working environment or process. 

I have been ranting like agile was pure and good and Agile got spoiled by human greed. But that is not all of it.

The design faults of agile and where we are heading post-agile is a topic I would like to explore in a future article. But look, created some 20 years ago, Agile was made in a world different from the one we are living in today. Back then software was less complicated, written by a smaller team, and managers were unfamiliar with software development hence the need to continuously demo and showcase.

Agile believes collaboration between builders is the key to successful working software and looks down on heavy investment in processes and documents. But as team size gets bigger, complete collaboration also gets expensive. As in, it takes a lot of time if we insist there is no big design phase and the best design is the one emerging during the implementation process. We want to look down on documentation because it is an artifact of bureaucracy but documents make a project long-term maintainable and allow different teams to work with each other, not working software. The world post-pandemic also sees a rise in decentralized teams, collaboration without documents in that context is simply not making the most out of the setting. And we think contracts do nothing good but promote constraints, yet nobody bats an eye at service contract.

It has been a long way to say that in 2023, Agile is not dead but it is less relevant than what it used to be. Seems like agile, as in the set of values and principles captured in the manifesto, shares the same fate despite its lingo difference. And that is the way things should be. In a world that keeps changing, change is the only constant.

ISO 27001:2013 Audit

At the beginning of May, my company went through a 4-day audit for ISO 27001:2013. I was responsible for some parts of the certification process and wanted to write down some thoughts on what I considered an interesting occasion.

Before we proceed, I must make it clear that I wasn't in charge of the entire ISO certification process, that would be our IT Risk & Compliance Manager - Satya. I did however design several systems and engineering processes examined in the audit. I think I can offer insight into what one can expect if she is an engineer going through the certification.

What ISO 27001 is and Why it matters

ISO 27001 is a certification of information security. It sets out the specification of an ISMS (information security management system) and covers people, processes, and technology. First introduced in 2005, it has since received 2 revisions, 2013 and 2022. We decided to go with the 2013 specification because ISO 27001:2022 was out in October 2022 when we were 3 months into the preparation.

Because we are a B2B SaaS, our sales cycle is considerably lengthy with procurement being a major time hoarder. Any enterprise is responsible to ensure that any data subprocessor (which we are) complies with its existing technical and legal obligations. That means our team has to go through pages and pages of questions - both technical and legal, some left you head scratching for hours (very unhealthy especially if you are bald) - just to prove that we know what we are doing when it comes to data security.

International certifications like ISO 27001 cut this cycle short, in the same way IELTS/TOEFT allows you to skip certain English classes. And after decades of enjoying little to no regulation, the last five years have seen a rise in data regulations like data cannot leave a certain geography region or the right to be forgotten, and it is only getting stricter from here. Hence it was a no-brainer for us to do it while we were still at the size where radical changes were still feasible.

Business gains aside, I also wanted to learn from this opportunity. I have been quite verbal about how Parcel Perform is the most complicated system I have ever built and that whatever I knew about large-scale systems, I learned from this very experience through trials and errors and sleepless nights. I couldn't reliably objectively tell whether my work was a state of the art or a big ball of mud, much like any proud mother looking at her child. Once in a while, it is nice to get external validation as well as a chance to learn what is missing.

What to expect from ISO 27001 certification

ISO 27001 is a risk management process. It is more of a thought framework than a checklist. I find it similar to a design pattern. You learn the problem space a pattern excels at then apply it with your own tweak. Between 2 projects though, the implementations of the same pattern might look different. ISO 27001 does the same for risk. Instead of dictating what (not) to do, it gives you a framework to think about risk.

The framework goes like this

1. Given the nature of your business, register the risks associated with it.
2. Assert the level of severity of the risks on your business.
3. Implement a prevention plan for such risks.
4. Post prevent plan, re-assert the new level of severity. The final level of severity must be low enough to allow the business to comply with its SLA

Samples, these were 3 risks we identified and addressed.

All documents related to this risk registration, including policies, practices, system designs, incident records, and whatnot is written, gathered, categorized, and submitted to the audit. Depending on the maturity of an organization, the time this duration takes varies.  For us, it was a few months. As a part of this practice, we had to define some policies and processes that didn't exist before, like Cryptography Policy and Third-party and Supplier Risk Management Policy. One tends to think of information security as protecting against hackers, or at least I do because I watch too many Hollywood movies growing up. There are indeed elements of defending against attack but nothing too paranoid. For example, leaving your computer open and unattended for hours is not safe but a 5-minute lock screen is good enough, despite one of my team member enthusiasm to explain how much he can do in the time window. ISO chooses to focus more on the complete governance view point rather than dictating the precise implementation. Overall though, we got to observe a good case of convergent innovations. Many of our in-house processes responded well to ISO best practices such as the engineering process from inception to deployment, or the access control mechanism.

Upon submission, on-site audits were scheduled. By default, ISO 27001 certification is issued per location. Singapore and Vietnam are places where our product is built and data analyzed so these are the main targets of the audit. Sales offices were skipped as data procession is not a part of their functions.

The audit exists essentially as proof: the documents describe robust processes, yet would it crack under pressure and are we doing what we preach? The audit was performed in a series of on-site interviews between the auditor and the people in charge over particular topics like access control, software development process, incident management, and more. These were just the ones in which I played an active role. I noticed that the audit conducted these interviews in two ways. For small isolated topics such as access control, we went through the entire process before he probed with questions and we countered with reasons why the decisions we made fit our circumstances the best. For longer topics, like an end-to-end software engineering process, he went straight to the Q&A dance. That left us mildly disappointed because we even had mock interviews for those long sessions :)

Findings during the audit are categorized into major non-conformity, minor non-conformity, and suggestions. In theory, as long as you can address all the non-conformities before the final audit day, you are good. For a multi-site audit, the end-to-end time can be a few weeks. In practice, a major non-conformity is pretty much a no-go. It indicates the lack of a mission-critical process without which information security effectiveness can't be achieved. Such a process takes time to form and even longer to put into practice and become an organization's second nature. In fact, I believe not having enough time in practice is the number one reason a quick attempt to rectify a major non-conformity is rejected. Minor non-conformities are quite easier, they indicate points of improvement in an established process and enjoy a much higher chance of being accepted if you can get them in in time. Suggestions are just what they are, non-consequential pieces of advice from the auditor based on his experience with other organizations.

We didn't discover any non-conformity, major or minor, so we are expecting our certificate to come through soon. But that isn't the end of it. Under normal circumstances, the ISO certificate must be renewed annually and only lasts for 3 years before the organization has to go through the full circle again. For ourselves, because ISO 27001:2013 is set to expire in 2025, we will be resetting our certificate with 2022 specifications next year.

Conclusion

Certification being a lucrative business as it is, I do find the whole process to be a positive learning experience.

• There are the thought framework approach and the big to-do list approach when it comes to certification.
• Hand waving can take you places but ISO like many certifications is one where the journey is more important than the destination.
• There are many things that we have already done right with the system and the organization around it, the external validation is a great encouragement to our effort.
• Our documentation collection can see more rigorous standards, though everything was produced correctly, it took a bit of time to gather them for the audit proof.

2022 Tech downturn

Tldr; The market is going through a period of downturn as a chain reaction of the world's economy. Lavish startup life is over, lean time is here. Product-market fit will be tested. But tech is here to stay and top engineers are highly sought after.

2022 was not a great year for tech. The industry was plagued with widespread layoffs, weak earning calls, plummeting stock prices, and an investment market that shifted from equity to debt. It was another link in a chain reaction of the world's economy: pandemic aftershocks, the war between Russia and Ukraine, oil prices, energy concerns, and weaker buying power. It just happened that this link hit me the closest.

How exactly did the tech sector get tangled in this whole mess? I think firstly that is what you get from a flat world (not earth), everything is more connected than ever. The very definition of "tech" or "big tech" is ambiguous. It can be anything from social networking companies to EVs and phone makers. Really, the concept of the tech sector is no longer as relevant as it used to be because every company is a tech company to some degree. The whole world economy didn't do well and tech was an integral part of it.

Secondly, the industry has operated with a free flow of cheap money for decades. Since the collapse of the housing bubble in 2008, the FED had kept a low interest rate for almost a decade - an unprecedented event in the 60-year-plus history of the organization. Even the increase in 2015 was described at the time as "a vote of confidence in the American economy". For a long time, the availability of cheap money meant there were strong cases for borrowing money from banks and making profits via investment, including tech ventures. The high ride however ended with the interest rate hike as the FED tried to control the US inflation. I however entered the workforce in 2010, as so did many other startup founders and workers. We thought tech as an infinite source of growth was a norm. We have never experienced a real lean time before. 

The third point, the greater force of macroeconomy throws much of future forecast out of the window. Boardgames such as Agricola or Splendor, or computer games such as Factorio are known to be resembling what it is like to run a business. And in all of them, a common theme is that the move you want to do now must be planned a few turns ago. Much of what a business does today relies on what it is expected to deliver in the future. So when the future prediction is skewed, it drags the businesses into the mud.

And oh boy are we bad at predicting 2022. I mentioned no one was expecting a war between European countries in the 21st century but there were more. Social isolation throughout the pandemic yielded great returns for tech companies in 2020 and 2021. But then the demand dropped as the world reopened. Turned out, it took more than a pandemic to alter behavior. E-commerce and food delivery offered great convenience and indeed enjoyed a higher adoption rate compared to pre-pandemic but they didn't replace traditional means. Working from home didn't become the dominant working mode. And consuming power was limited to necessities as the world braced for economic difficulties.

The combination of cheap money and an over-focus on future delivery means most startups had chosen growth over profit for a long time. It had always been about acquiring the biggest slice of the market pie and only then turning to increase the profit margin. You had to because all your competitors were doing the same. There would only be a little room to grow if you had a tiny slice of the market regardless of how healthy your margin was. The cheap money guaranteed that if slow and steady was your strategy you would end up in a hostile takeover. That also means sometimes companies found themselves running faster than they should have. The widespread layoff we saw was the direct response to a decline in forecast demand and a need to reduce the cash flow till a better time.

It was easy to pick up a sage voice and describe the crazy world that was 2022 with much hindsight clarity. Being on the ground, running a team of 100+ head counts, and experiencing the first lean time sucked monkeys balls though.

Alright, so that was how we got here. Where do we go from here?

The last time tech recessed in the 2000s, it went down in a blast. NASDAQ was tech-heavy and it took 8 years to climb back to where it was previously. Yet there are reasons to believe this downturn wouldn't be as bad. During the dot com bubble, we were in the exploration stage. There was this internet thing that was supposed to be a new era but nobody was quite sure what it could do so everything went. Great ideas mixed with crazy evaluation fueled by FOMO money. Pets.com was arguably the most famous flop. Kozmo and Webvan burned through billions of dollars as their grocery delivery models were not sustainable. Think Tools AG evaluated at CHF 2.5B without having a product. The noise was so bad that after the burst people believed the whole internet thing would just fade away, contributing to the long recovery. 

The time around, other than the crypto scene which is going down the exact same path, the rest of the market is a lot more mature. Companies stay much closer to reality, solve real problems, and have clear plans for monetization (at least I hope so). Tech is here to stay and people are just preparing to weather the storm.

That being said, 2023 is not going to be a great year to raise equity. People who set out for a funding round would be less likely to receive favorable evaluations and terms. The formula used to be that you can plan for a round every 18 - 24 months, one led to another on the basis of momentum, and hope that Tiger or SoftBank will come in with a massive Series C or D and allow you to have some sort of secondary exit and just build the momentum, and maybe a SPAC will you liquidity quickly. Didn't work out quite well for Grab and Sea.

At the same time, the general decline in demand across B2B and B2C remains a threat. The product-market fit (pain killer vs vitamin) is once again brought to the forefront. Startups solving more critical problems are more likely to survive. Long-term strategic product programs might be put on hold to make room for initiatives contributing immediately and directly to the bottom line. Some unfortunately will run out of time before things got better. While others are presented with a unique opportunity if the product-market fit is good in the new reality.

Interestingly for startups to remain competitive, they need to invest in technology. The performance of a tech team grows slowly, depends much on ad-hoc knowledge management, and is the bottleneck of any innovation. Even though layoffs were spreading, the core of tech teams was protected. Ones that do not spend in the near term will undoubtedly fall behind in the medium term and risk not being around in the long run.

The spending pattern might change. Software development in particular has always been one of those fields where the performance gap between top and average performers is tremendous. The term 10x engineer has been saturated to the point now it is more of a meme, but there is a kernel of truth there. In sports, running 1% faster than the next guy results in 10x compensation. But that's it, if a 1% gain costs 10x more, might as well get 10 of the slower ones. Engineering is different. A top performer can provide solutions that bad ones can't come up with, regardless of how many of them, and doesn't cost 10x as much. Then computer automation and scalability come in to multiply this productivity difference many magnitudes more. With cash flow running lower than before, recruitment is no longer a number game. Companies would pay top dollar for a few strategic positions but otherwise not double their team size any time soon.

The startup boom has lasted for a decade. It has also faced so many scares, each time more money and power were poured in. But maybe it really is different this time.

2022 - Could this year have been better? Yes.

It was a lovely afternoon at the beginning of December when I wrote these lines.

Vy was at her class in the uni and I had the entire apartment just for myself.  So it was quiet. Golden sunshine cast on everything, the breeze was pleasing, and the view over the canal was gorgeous. Despite that. I felt uneasy. Felt like there was something I should be doing, that I was losing my time but I couldn't figure out which. And that had been how I felt this whole year - things were dashing by and I was just trying to catch up.

A little reminder for future people what kind of year 2022 was. It was the year when Tuan-Anh got married, Soc was born, and the whole world finally came out of Covid-19 after three years of social isolation and economic stagnation hoping for post-pandemic growth. Things perhaps would finally feel normal again. Except Putin said fuck that and evaded Ukraine with his little special military operation. The oil price was all over the place. No country seemed to agree with one another. In Vietnam, there were widespread layoffs because orders in the West were drying up. And in Saigon just last October you literally could not find gas for your motorbike.

The master's degree.

No, it wasn't me. That was why Vy was at the uni on a lovely Sunday afternoon. She got a scholarship at the same place I got my bachelor's.

I was told that a master's degree is like doping. You use it when your career hits a plateau and you need a little push to go to the next level. And for a really long time, it was one of the most popular pieces of life advice I propelled. Watching Vy though, I learned something new. Once you pass a certain point in your life, you can no longer be schooled. That is, you can no longer entertain someone telling you where to look, what to do, and how to do it. You can no longer fathom the thought of slaving your days and nights away for a thankless assignment solving an imaginary problem just so some old dude can put a no less imaginary score on it. Guess I would not be educatable for a while.

Vy was crushing it though. She worked hard in her day job, got home, and then put on the second shift for her study, be it preparing for the next day's slides, writing assignments, or working out problematic drama queens in her group. You would not believe how childish some of these master's students could be. Perhaps that constituted their educatability. Anyhow, like any good partner, as Vy slaved away at school, I had been pulling my weight in the kitchen. To be clear, I was not setting up a gym in the kitchen. I looked after the place, prepared meals, and might or might not write a programming assignment recently. Felt like that was the least I could do to support her. She was working so hard, I wish she would be successful. Also, my retirement plan depended on that.

I got anxious though, it was hard to watch someone working that hard and not do a round of introspection. Was I growing in my career, that I had been better than who I was last year? Was I prepared to continue this path for another decade? Should I be doing something else other than sitting here writing? To be frank, I was a bit scared, like in those recurring dreams where just as you realized the exam was today, you also realized you haven't studied shit. No? Just me? There was a fear of missing out, a fear that something wrong was brewing and by the time you realized it was too late to rectify. I was sure English had a phobia name for that, or German.

The apartment

During the pandemic, however, I learned to appreciate the importance of a proper home and how rewarding it could be if you got it right. For a long time, all that I did in terms of housing was alternating between piggybacking in one of my previous companies' apartment, sleeping on the floor at offices, and for the most part, shuttling between a cheap tiny rental unit close to my office during the week and my parents' second home at the weekend. While I thoroughly enjoyed the short commute and my parents never collected rent or embossed any restriction on my freedom, neither place gave me the certainty of ownership. I found it hard to put words into this feeling. Say, the sense of ownership at its core is the difference between working on someone else's code and just wanting to get shit done vs working on your own code, something you know intimately, and wanting not only to do the right thing but also to do it right. For my non-developer friends, err... it's a rental.

The criteria for my "repository" were straightforward: close to my office (and hoped that the office was not moving), bare concrete for minimal waste and maximum customization, and a view that didn't constantly remind me I was living in stacked boxes. For the kind of budget I had, that didn't leave a lot of options, I started looking by Oct 2021 and got the paper done by Nov.

Because the place was concrete and nothing else, I went through the whole process of briefing my needs, finding an architect, feedback loops (3 months), and construction (another 3 months). All in all, it was almost a full year from when I bought the place until I moved in. As the nature of construction dictated, the process was waterfall where mistakes in one phase would get really expensive to fix or even unfixable in later phases. With the same attention I used to examine a technical design, I 3D-rendered the apartment in my mind, suggested modifications, and browsed the second page of Google to find the right material. Only desperate people visit the second page of Google. There were plenty of things to think through, like the placement of built-in electrical sockets or the layout of cabinets, as I had neither the budget to redo nor the skills to DIY. 

From start to end, the journey was exciting and fun. Building up a physical space left a tangible spark of joy that was usually missing from my profession - software engineering. But because of how involved it was, for a while it was imprinted in my mind that at any point in time there was something else in the design I have neglected, a modification made, and an article read. I would often find myself on a fine afternoon like today and wonder, should I be doing something else?

It had been 2 months since I moved in and the imprinting was fading away. While it lasted though, I realized that, for me, the fear of either leaving something out or being left out that would lead to cumulative damage was real and primal. I can reason to myself that this too shall pass, that in the grand scheme of things it wouldn't matter, that I have done the best that I could but I can't shake off the feeling. There had been a number of moments this year where I got an acute "tic" that something wrong was happening without me knowing and by the time I knew what it was, it would have been too long.

The tech downturn

2022 was not a great year for tech. The industry was plagued with widespread layoffs, weak earning calls, plummeting stock prices, and an investment market that shifted from equity to debt. It was another link in a chain reaction of the world's economy: pandemic aftershocks, the war between Russia and Ukraine, oil prices, energy concerns, and weaker buying power. It just happened that this link hit me the closest.

I noted down some of my thought on the downturn in another post where I said the greater forces of the macroeconomy were making it really hard to go against where the industry was going. Didn't help with the emotions though, did it? No one working in a startup would be happy with just cruising along the "industry average" path. Startup people are competitive and hold on to exceptionalism. Without those traits, who in his right mind would invest years of hard work charging against bigger corporates with deeper pockets and higher headcounts? So this setback was a blow of defeat.

Here I had my phantom fear materialized. Had I written better code, spent more time with my colleagues, or initiated better technical investment, could I have soared above the average? I hoped so. I wished I could do better because this feeling that I was not enough, that I was asking for more than I could give was the most tangible sense of helplessness I experienced in a really long time.

--

By the time I got to these lines, the sun was on the horizon and I had already moved to the shade of a small dock overseeing a peaceful turn of a canal. I know this canal well, I have traversed countless trips on it and yet its beauty never fell to capture my attention. I always feel attached to bodies of water. The tranquility made me nostalgic. I recalled the time when I was on kayak often, when life was easier, and when the net outcome of action was more direct and calculatable. But life moves on. Tomorrow I would board the first flight to Singapore to discuss how we could best weather the coming storm. 2023 will be challenging but did I say startup people are stupidly competitive?

Stay strong.

Prioritizing development decisions

Great startup stories tend to share the same mold: a great founder dreamed of an equally great vision and followed it fearlessly till the world was conquered. History is written by the victors, of course, but it is relatively common for people to have a rough idea of what they want to build before starting the work. That’s the easy part.

But constructing a concrete step-by-step plan to deliver not even the vision but a mere release is hard work. A good plan needs to take advantage of both business and development expertise without letting one overpowers the other. If the business makes all the calls, the development time might be painfully long and the product crashes when traffic starts to peak. If the development decides, we might have a technical wet dream of solving a non-existent problem. That’s where the planning game comes in.

I can’t decide if a game or a dance is a better metaphor for depicting the collaborative nature of planning. Business and development, each possesses knowledge unavailable to the other and is unable to produce the entire plan. The work can only be done by combining the strength of both sides. In economics, a “game” refers to a situation where players take their own actions but the payoff depends on the actions of all players. Game Theory suddenly sounds less conspiratorially adventurous, doesn’t it? Dance on the other hand isn’t used as much in research literature so I ended up siding with the economists. That’s a sidetrack.

In an Agile team, a planning game looks like this:

1. The Product Owner decides the scope of the plan. Based on the purposes of the projects, the Product Owner prepares a set of use cases and explains why they are valuable problems to solve and why they should be done first.
2. The whole team breaks each use case down into stories. The idea is usually that anything requiring the team to do something other than normal company overhead needs a story.
3. The developers “size” the stories. They estimate the time each would take or its complexity. And then group stories that are too small, split ones too big, and decide what to do with stories they can’t estimate.
4. The Product Owner prioritizes the stories. Some stories won’t be worth adding, either unimportant or too far in the future.

There are many things that can be said about the planning game, from Work In Progress should be minimized, the releases should be small and often, to the best answers for “why does it cost so much?”. Those are stories for another time.

In this piece, I want to discuss specific friction in step 4 of the game where one story is prioritized over another. The stories laid out in step 2 do not necessarily project the same values to different team members. Some are pretty straightforward, implement feature X to earn Y money, the contract was signed. While others are more tricky such as implementing plug-and-play UI components so that future web pages are built faster. The second category usually comes from the development team who is one layer away from the users and so perceives values differently from the Product Owner. That is the breeding ground for misalignment.

Product Owners want to release a solid, usable product. They also have to balance that with the desire to save money and meet market windows. As a result, they sometimes ask developers to skip important technical work. They do so because they aren’t aware of the nuances of development trade-offs in the same way the developers are.

Some developers note down all the development options like a shopping list, “outsource” Product Owners to choose, and then roll in agony at the wrong decisions. If such a strategy didn’t work for the guys at the Pentagon, it wouldn’t work anywhere. Just as Product Owners are the most qualified to decide the product direction, developers are the most qualified to make decisions on development issues. Don’t delegate the decision, take the matter into your own hand. If a development decision isn’t optional then it shouldn’t be prioritized either. Just do it.

Instead of:

Our notifications is crucial at informing customers the health of their business. To make the data pipeline behave transactionally, we have several options. Please let me know how should we prioritize them.

• Experiment with Flink’s TwoPhaseCommit, this is new to us so it would take time and be hard to estimate.
• Get Sentry to cover all the projects, this is a passive measure as we passively wait for exceptions.
• Add a check at the end of the pipeline to make sure no duplicated notifications are generated, the check will have to handle its own state.
• Move the final stage of the pipeline to Django, it is a web framework that supports transactional requests by default and we are familiar with it.

Try this:

Our notifications is crucial at informing customers the health of their business. The data pipeline is long and consists of multiple nodes, each needs to successfully finish its work to produce a notification. To achieve this notion of exactly-once delivery, we need the pipeline to behave transactionally and every exception to surface swiftly. That is done via Flink’s TwoPhaseCommit and Sentry integration. The work will be done at the beginning of the project as it is easier to handle when the code base is still small. TwoPhaseCommit in particular is new to us so we will have a couple of spike stories to understand the technology.

When there is a business choice to be made, don’t ask Product Owners to choose between technical options. Instead, interpret the technology and describe the options in terms of business impact. To continue our notification example, before any notification is sent, there is a need to make sure the data we have is the latest. The conversation can go like this:

We are thinking about adding another Kafka queue to request the latest data. We then need to join the request flow with the future trigger with some sort of sliding window, will also need to thinking about out of bound data. Our other option is to set not one but two future triggers so that one can request data and the other handles notifications. Which would you prefer?

Try this instead:

We have two choices for ensuring a notification always works on the latest data. We can use a deterministic approach or an empirical approach. The deterministic approach would add a new data request flow right before the notification is sent. The notification is processed after the data request flow so we always sure the latest data is used. But because technically data procession and future notifications are asynchronously independent from each other, it would require several more stories for us to join them together. The empirical approach won’t take any extra work. We observe that it usually takes less than 5 minutes for a data request, so we can set two future triggers instead of one, 10 minutes apart from each other. The first one request data, the second notification. But the margin of error is larger because sometime there can be delay in data request. Which would you prefer?

And finally, no software engineering discussion would be completed without a talk about code refactoring. In the context of the planning game, it is mostly about justifying the refactoring effort. While it is tempting to do a “spring cleaning” hoping to refactor the whole thing back into shape, the sad truth is halting the development of working software for refactoring is hardly justifiable. Refactoring effort deals with risk (the old code can implode at any time) and potential (the new code is easier to work on). Those values are intangible compared to the usual subjects of a business decision (new features lead to a new set of customers lead to greater revenue).

What do we do? Boy scout rule “always leave the campground cleaner than you found it.” Whenever you need to implement a new feature or fix a bug you see if that part needs improvement. Refactoring shouldn’t be a separate phase, it is part of everyday development. Once you nurture this culture of quality, there is nothing to justify.

None of the above suggests the easiest way to avoid friction is to keep the business side in the dark while going on waving the engineer's magic wand. Communication remains the key to any successful project. There is more to a project's success than just business decisions, and working out a way to be a (constructive) part of the conversation is more powerful than a baseless delegation.

What makes an internship meaningful?

I got 2 kick-starts in my career, a part-time job in 2009, and an overseas internship in 2010. I dropped a production database on the first day into the former, and was so ill-prepared in the latter I didn’t look up the place’s climate and local language till I was already on the plane. Both were privileges. They catapulted me into worlds I couldn’t fathom and shaped my career in the following decade. In turn, as I have built startups, I have also been on and off in building internship programs. My number one concern is to replicate the magics I got to experience for the next generation. I am writing this for you who are seeking an internship opportunity.

What makes an internship meaningful? I ask this a lot, and the most common answer I receive is “a new experience outside of school”. That’s right. Much of what we do in life is to propel ourselves into new experiences. As a species, we are a bit of an adrenaline junkie (and serotonin, caffeine, and cocaine). But it is not particularly good at answering the question.

A new experience is undoubtedly crucial for an internship, but if it is solely what makes an internship meaningful, wouldn’t picking up something entirely opposite to your education offer the most exposure? I am a software engineer by train, but the course that awed me the most in college was a general elective, macroeconomy. It expanded my horizon, but I wouldn’t have considered an internship in a central bank, it would have ventured too far away from what I wanted to do. An internship should represent a certain level of relevancy to your long-term career.

An internship is only as meaningful as the mentorship it can provide. The transition to the workforce is the transition from theoretical information into practical knowledge. The rigid structure of the curriculum makes it hard to grasp some key concepts of software engineering. At the end of my degree, I was still not sure if my code was clean - there was no need to reopen last semester’s assignments, my architecture was flexible to changes - no assignment lasted for more than a semester, or my software could be released in small, iterative circles - all assignments were fixed time-box. And it took years for me to get comfortable around these concepts. It was frustrating navigating in uncertainties. I wish someone was there telling me what I did wrong. In the language of the Dunning Kruger effect, my descent from the peak of Mt Stupid to the valley of Despair was rough.

A good mentor can ease the journey. Mentorship provides a sense of trusted guidance a personal friend can give with the bonus rule of being the manager. They help you to transit from a student to a professional.

Lastly, an internship implies getting out of the sandbox. At school, you are learning in isolation, a designed system, a less glamorous version of The Matrix. You spend your days solving imaginary problems and gaining imaginary successes. None of which reliably predicts your success at work, otherwise everyone would be boasting their GPAs now. And if you slip here and there, other than some awkward group assignments, there is hardly any difference in the grand scheme of things.

At work, however, there are consequences to your actions. There might be works whose input relies on your delivery. There might be deadlines that are not arbitrary choices dictated by a course’s curriculum but coordinative milestones to align other efforts. And there might be actual end-users of whatever you are delivering. An internship that can’t offer this sense of ownership is nothing but yet another simulation box.

Compared to the previous two elements, giving someone who has zero experience ownership is considerably more complicated. Many internship programs are decided by management or HR.

• Management wants to establish a partnership with a university and an internship is a low effort investment.
• Next year recruitment might get harder if none happens this year. Out of sign, out of mind.
• Doing the dean's office a favor. True story.

While there is nothing wrong with these reasons, too often they are used as excuses to force the internship upon the team adopting you. A forced internship can go wrong in a couple of ways. 

(1) You can be treated as junior developers and go through the same onboarding experience. The thing is, internships are typically short, and by the end of it, you would come back to school to continue your studies, usually, right at the point the onboarding has just been over. The internship is then an overwhelming experience for you and a waste of time for the team.

(2) You might not be ready to work on production code yet so a common solution is to make up an internship project nobody needs. The general lack of coordination between those who want to host an internship and those who are in charge of it results in uncreative half-assed ideas, such as cloning a working piece of software, building an internal tool whose specifications were a few Slack messages, conducting an experiment that the mentor has not made any progress recently, etc. 

A better internship experience can be built with buy-in from the team. An internship fulfilling the needs of the team can be a constructive experience for everyone involved. It could be to give someone new to management a chance to run his own (intern) team before reaching out for a more substantial responsibility. It could be an isolated project with actual stakeholders who have good incentives to invest in its success. And it could also be an experiment, a research topic, but goals, direction, and companionship should come with it. In short, seek out to align the success of the team and the success of the internship.

Too many internship hours were wasted on demo code no one reads and pet projects no one uses. I hope yours would be better.