tag:blogger.com,1999:blog-3081910872960403022024-03-14T09:50:42.178+07:00Life as an engineerI have my opinions. You have yours. Take everything with a grain of salt.Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.comBlogger103125tag:blogger.com,1999:blog-308191087296040302.post-83073889339450208032023-12-31T20:48:00.005+07:002024-01-01T08:50:42.989+07:002023 - A lemonade stand<p>Achoo!!!</p><p>I had been under the weather the last couple of days as a sneeze echoed across the apartment the moment my nose met the smell of fresh ink of crispy paper. I had had this thick notebook for over a year, perhaps two. Skimming through the pages, I found interview notes I had long forgotten, sketches of the apartment that had not been built then, and outlines of essays long and short I might never publish. It was the perfect amount of nostalgia to start a usual year-end retrospective.</p><p>What kind of year was 2023, you asked future readers? 2022 was a straight-up dumpster fire. 2023 was harder to describe. It was a mixed bag. The earth had its hottest summer since the Anthropocene - the era of man. The war in Ukraine progressed to a stalemate. And another one started in Jerusalem (or had it ever ended?). In Vietnam, car registration took more than a week, and hospitals ran out of medicines and equipment as a result of attempts to address corruption in these fields, thus proving the corruption in this country was systematic and superficial reactions wouldn't make anything better. The domestic bank interest increased, but housing prices remained high, if not higher, so the real estate market was broken. But it was also the year OpenAI became the world's sweetheart. India landed on the moon while SpaceX made important progress on its prep for Mars. Vietnam FDI continued hanging on to the tailwind of manufacturing moving out of China. 2023 was a year that life kept a steady intake flow of lemons but some had already started adding soda into their lemonade.</p><h4 style="text-align: left;">An engagement ring</h4><p>I was down on my knee and proposed to Vy by the bank of Sun Moon Lake, Taiwan. Taiwan has always been a special spot in my heart as my adult life started there. I hope Vy would appreciate that ceremonial meaning. I first planned to propose in a fine morning at Taiwan National University which was my hood the entire time I was living here. That fine morning a system incident broke out and I was stuck in the hotel room till it was time for a train.</p><p>The main plan was out of the window, I played by ear. That day we went on for a bicycle tour around the lake, 30-something km. If I proposed at a random spot in the middle of the ride, we wouldn't be able to revisit the exact spot later. I bid my time. Later in the afternoon, toward the end of the ride, we reached a patio area waving around the bank, I thought this must be the place they put a giant pin saying "Sun Moon Lake" on any map. There was no chance in hell I couldn't miss that spot even with my faulty memory.</p><p>I think I did well across superficial social criteria for a proposal. Meaningful location. Surprise. Ring. All checked. Vy kept complaining that of all the days we were in Taiwan, I picked the day she dressed worse and was sweaty... Could have been worse, darling. She also said she didn't see this coming. Didn't see it coming? We poured our life savings into an apartment over the last couple of years. What was that woman thinking!?</p><p>Still, this was a big move. Conceptually I understood that it would set the course of my life but that didn't necessarily mean I got it intuitively. Like when you are signing up for a 50-year life insurance contract without a single clue where your life is going next year. You know what you are doing, doesn't mean you can fathom the impact it has on your life for better or worse. But I have been compatible with Vy the whole time and more often than not we managed to come up with a solution for what life gave us. Doesn't mean I am looking forward to troubles though, can certainly do better without some of those. Hope this will fare better than life insurance. My office is next to a Manulife building and oh boy there is something like a weekly rally there.</p><p>So yeah I think I am getting married at some point.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhO9Ol3R6iI35frNU4ajzBY-h6KR7kCe7isuR1F6gCqnGw_T9wSgchTd1LWdb_lY3g2S_dVIj_eo75xVD-YAPiVuPt8oHw-HLmVnm6LKiO65AKzWcEzXkhktkjSiFH_R9McIR3dX2tCljzkf8hH1HJb2zgCYMs8ARcPWdLZJd4u3cJRYhj6IY-pj9oFD6b4/s2561/a.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="2561" data-original-width="2268" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhO9Ol3R6iI35frNU4ajzBY-h6KR7kCe7isuR1F6gCqnGw_T9wSgchTd1LWdb_lY3g2S_dVIj_eo75xVD-YAPiVuPt8oHw-HLmVnm6LKiO65AKzWcEzXkhktkjSiFH_R9McIR3dX2tCljzkf8hH1HJb2zgCYMs8ARcPWdLZJd4u3cJRYhj6IY-pj9oFD6b4/w354-h400/a.jpg" width="354" /></a></div><h4 style="text-align: left;">Health</h4><p>The last quarter of this year was not a great time for me. I had mild pain where my backbone and my hip meet. It wasn't too bad that it brought agony to my life - my work has already done that - but was enough to let me know its existence. I haven't played badminton for the last 4 months. I missed a mountain marathon to which I made a public commitment to my colleagues. Heavy lifting would intensify my pain 5-10 minutes after the act. My running routine is now down to 5-6 km a week and I usually tip-toe the whole run as I am afraid a bad move is all what it takes to make the matter worse. Oh and everyone is telling me to do more swimming. I do, I try, oh gosh I hate swimming in a (small) pool. Before I can get tired, I will realize I am moving back and forth in a pool of water literally not getting anywhere. Once that feeling kicks in, I get 5min before the exercise becomes terminally boring!</p><p>As of this writing, I haven't experienced constant pain in the last 2 weeks. I am exercising more often. I start to play badminton again in January. I hope I will get better and can do serious running again. I am watching my diet. 5 days a week my supposedly healthy food comes in plastic containers. I hope they are biodegradable but knowing Vietnam that is a best-effort commitment only. At a ripe old age of 34, I have had enough signals from the universe that I am far from invincible and if I am not careful, I won't live to see my 100th birthday or I might not like how I get there. </p><h4 style="text-align: left;">The apartment, again</h4><p>Urggg no, I am not moving to a new place. Last month marked the first full year of Vy and I living in that apartment. I absolutely adore the place. It was designed to the tee so naturally it met all our needs and made living there a bliss. It is also visually pleasing, we got compliments from our friends and family. It showed up in a magazine at some point.</p><p>There was one problem though. We wanted a polished concrete floor. Our architect suggested a microtopping floor. I looked it up, and it seemed to be a nano layer to protect the floor... so a construction technique I guessed. I was fine with that as I assumed it was just one of the methods a concrete floor could be polished. Opposed to my best intention, the first iteration of the floor seemed soft. Like wax. Dragging a heavy chair on it left a dent. You probably picked up from the previous sentence that there was more than one iteration of it. The second was an absolute disaster, it looked like a painting of a 3-year-old, patches of color didn't blend together and the floor was uneven, worse yet, it peeled. Like my skin after a day of sunburn, except no one expected that from a frigging floor.. The third iteration fixed most of the problem, except that it still peeled in small freckles when in contact with wheels which my coffee table and office chair happen to possess in plenty.</p><p>This was the point where I threw in the tower. Redoing the floor was a major operation, we had to leave the apartment empty for at least a week, vacant our belongings, and spend the following couple of weeks dusting the whole place. We are letting go of the hu-tieu cart and getting a carpet for my home office. I had always been fond of the cart and the only time a carpet is clean is when it comes out of the wrapper. But my idea of a house is more of a utility than a work of art. I might like the place a bit less (hardly) but if I can cruise through my day stress-free of wrinkly peeling floor, that seems fair.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeJhYJyTsYiXJRryA6kw8s4jam-ns7XEYOqHMFZxvVocGQnplIWnxuK9FK93ykIWFjzwX2Rs1wq7GPx0X0Nmil9QIUXEvsYilbccLf6xvGNwJuBOM8OXZm1iTYrpEEgSgIsT3wFeJNMCGcJG3nSv9Rhf35Fd-HqKmtS-K4sPisK7BeM7puKwBcYaljje5x/s4080/PXL_20230715_022231417.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3072" data-original-width="4080" height="301" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeJhYJyTsYiXJRryA6kw8s4jam-ns7XEYOqHMFZxvVocGQnplIWnxuK9FK93ykIWFjzwX2Rs1wq7GPx0X0Nmil9QIUXEvsYilbccLf6xvGNwJuBOM8OXZm1iTYrpEEgSgIsT3wFeJNMCGcJG3nSv9Rhf35Fd-HqKmtS-K4sPisK7BeM7puKwBcYaljje5x/w400-h301/PXL_20230715_022231417.jpg" width="400" /></a></div><h4 style="text-align: left;">Vehicles</h4><p>In 2023, I seem to have more vehicle-related stories than average.</p><p>I was left strangled in the highland of Vietnam when my rental - a docile Honda XR150 - ran out of oil, literally. There was no black smoke from the exhaust nor any leak on the engine case, I must have gotten a unit whose maintenance schedule was neglected. Because we started the trip with 2 bikes, we managed to ship the broken one back and continued the rest of the trip 2-up. Still, this has never happened to me before so it left a distinct mark in my memory. So much so I could cite the precise behavior of the bike and the riding experience up till the point of breakdown, I won't bore you with the details though.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPngFNhIM1QeDox-QnicQ7YTPllTiQk32QWppf4ZNpfWD5k8u1HZnMFfIhfoiiWOlT3QVFzhZHOaqw1dUwcbQe0WT2BKDGN6mo18eEx_KHEYr_CDm3F5kRw-HklRf0fcJzr5p9B4FQaIheBKDFe7QcREjP7al9ZbBvnLg0m99SOvPKRt9gTesuyHSd85qd/s2010/Screenshot%202023-12-31%20at%208.45.20%E2%80%AFPM.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1104" data-original-width="2010" height="220" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPngFNhIM1QeDox-QnicQ7YTPllTiQk32QWppf4ZNpfWD5k8u1HZnMFfIhfoiiWOlT3QVFzhZHOaqw1dUwcbQe0WT2BKDGN6mo18eEx_KHEYr_CDm3F5kRw-HklRf0fcJzr5p9B4FQaIheBKDFe7QcREjP7al9ZbBvnLg0m99SOvPKRt9gTesuyHSd85qd/w400-h220/Screenshot%202023-12-31%20at%208.45.20%E2%80%AFPM.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Hero to Zero</td></tr></tbody></table><p>I bought a used bicycle from TuanAnh's wife as the maternity left her little time and it was sad to have the thing collecting dust in the basement. I used it for my commute for a while. So far so good, till I realized the security at the office building was prepared to handle motorbikes and cars, anything else threw them off. They couldn't even decide where the bike could be parked so every morning someone would tell me to park it in a different place. Things gradually got worse from there as more ad-hoc rules were introduced. I got a lot of stress in my line of work and the last thing I wanted was to have my workday ruined before it started just because an incompetent guard decided to be creative that day. Now the bike is collecting dust in my basement...</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9PwcBaWY1SdX9pEUiHGqQ2lKV4lGTu3YcUojLWmx8rdig73Z8SxBW1oiLPXS7U73OlgR4yzSIrQ2yxcyQ0GZt_5yZ6HKal86uIy62-faYd-ScHUQ6k3n_pRtBMJcUoSfsFieSjDtVKLWJr93AS5WZo5U349tesMTRN3MHAgQMkwqwWZqDsc5YqSC2XJ2P/s1006/Screenshot%202023-12-31%20at%208.41.16%E2%80%AFPM.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1006" data-original-width="972" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9PwcBaWY1SdX9pEUiHGqQ2lKV4lGTu3YcUojLWmx8rdig73Z8SxBW1oiLPXS7U73OlgR4yzSIrQ2yxcyQ0GZt_5yZ6HKal86uIy62-faYd-ScHUQ6k3n_pRtBMJcUoSfsFieSjDtVKLWJr93AS5WZo5U349tesMTRN3MHAgQMkwqwWZqDsc5YqSC2XJ2P/s320/Screenshot%202023-12-31%20at%208.41.16%E2%80%AFPM.png" width="309" /></a></div><p>Lastly, after years of procrastination, I finally got my driver's license. I was in no rush, have you visited Saigon? I live here and everyday commute on a car is such a hassle I can't picture myself doing so. But when my father retired a few years ago, I co-financed with him to get a car. It made me feel better that my parents could travel around with comfort and safety. The car was an absolutely positive addition to their lives. I saw that given the right environment, driving could be enjoyable and I thought that Vy would appreciate some road trips where she didn't need to don half a dozen safety gears. That was enough of a nudge.</p><h4 style="text-align: left;">Work</h4><p>I am not sure that I have done a good job this year. People tend to overestimate what can be done in an hour, but routinely massively underestimate what can be done in a year. At times, I couldn't help but feel like Parcel Perform was not moving as aggressively as we could. However, in retrospect, there is clear evidence that we have made bold strides in improving our organization and system.</p><p>We consistently invested more than 20% of our developer time into various tech investment topics. None of the incidents in 2023 were rooted in performance bottlenecks. We managed to downsize the busiest database and spend less on infrastructure than in 2022. All that while sustaining a 50% YoY increase in traffic.</p><p>We gained more and more understanding of DDD and the importance of bounded context to our squads. We have been discussing, communicating, and refactoring to make boundaries clearer and more intuitive to ensure the size and complexity of our projects don't become our weakness. We are gradually getting better at running projects across squads and departments. We learned that there are times for crystal clear interfaces between teams, and there are times when interfaces are just blockers preventing people from making progress together. The little book <a href="https://www.amazon.com/s/?ie=UTF8&keywords=team+topologies&index=aps&tag=hydglogoo-20&ref=pd_sl_8p3fe4xnh7_e&adgrpid=81500867974&hvpone=&hvptwo=&hvadid=585479290475&hvpos=&hvnetw=g&hvrand=14386685909181519723&hvqmt=e&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=1028808&hvtargid=kwd-723035983914&hydadcr=27987_14525428" target="_blank">Team Topologies</a> certainly helps.</p><p>We still have a long way to produce a world-class software team. Heck I don't even know what world-class means nor do I care. What I mean is that we can still do better. We need to internalize what we have learned the hard way into muscle memory of the organization - which means not just within our tech hub but across Parcel Perform as a whole. It will be hard to invest developer time at a greater ratio for tech investment, the quantity is there, it is now time for quality. Training is back on the agenda for both leadership and technical content. On average, we have one squad on the verge of collapse a year. A squad can be still recovered from that stage, but it won't be quick. It takes a lot of time and dedication, and nothing guarantees the effort will yield results. We can get better at all of those.</p><p>---</p><p>I think 2023 was an eventful year, at least for me. It was still bloody hard but the feeling of helplessness had vanished. Gone was the feeling that we were dealing with forces beyond our control, as it felt in 2021 and 2022. The world was an uncertain place, it was also a fertile ground to sow effort. The better of us human have already been back on their feet changing the world. What the hell are we waiting for?</p>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-29583876287678667992023-11-19T21:48:00.006+07:002023-11-24T13:37:25.505+07:00Taipei - 13 years later<p>The HSR train is cruising through the south of Taiwan at 200km an hour. I have been on the island this last week and now heading to Kaohsiung for the way back. Inside the car, the stability is excellent, there is nothing but an eerie sound reminding me of the neck-breaking speed a few cm outside. Through the window, a constant flow of paddy fields scrolls by. Unlike in the US where machine power beats nature into submission, the horizon is perfectly flat, and all the fields are endless rectangular shapes. In this part of the world, agriculture forms around the ancient landscape. The rice fields are small and entangled in a net of canals even smaller. Embracing the late summer sun, the fields are green and the stems are round, juicy, and straight as there are no grains to bear yet. It is like looking at the skin of a cantaloupe. My mind though has stayed in Taipei for the last few days, where autumn was hard at work, putting her gentle touch on everything. The sun painted everything with the color of honey, the air was clear, and the breeze was crispy. In that crystal clarity, my younger years came back to life.</p><p>The metropolis of Taipei is a conglomerate combination of three cities - it is usual for an Asia country to have one big city overshadowing everything else in the land. In that concrete jungle, I lived abroad for the first time, got my first full-time job - an internship, and learned to juggle between that and a Chinese language degree. Because of course, Taiwan was strict on immigrant workers and they had better things to do than issuing visas for low-level interns. The only way I could stay on the island for more than a month in that situation was to become a uni student. Or married to a Taiwanese sugar mommy. It was tough, exciting, and mesmerizing in the following years, the internship, not the marriage.</p><p>Taipei is 2229km away from Saigon. It is not far from the length of Vietnam, 1650km from the tip of the north to the southernmost peninsula. The distance seems much further than that though. There have always been other places to be, work to do, and COVID put everything on ice for a while. This is my first time coming back to Taiwan after 13 years.</p><p style="text-align: left;">I made exactly 15000NT a month back then. I never for once asked if it was a fair income. For what it's worth, I also didn't stop to think if I had the right clothes for Taipei weather or if people there spoke English till I was on the airplane. The 20-year-old me ran on adrenaline rush much more than rationality if it hasn't been clear to you yet. By the way, no, I didn't have the right clothes, it got to -2 Celsius during Winter, and yes you could survive with only English if you only stayed near the city center. I made my financial plan (or rather the lack of it) around this precious number. The office was a converted studio and after dark it actually became my studio. I had a pull-out mattress. There was a cracked XBox in the office and a 7-11 on the ground floor. The boss, Jake, lent me his daughter's old bicycle. I never owned a game console and 7-11 didn't come to Vietnam some 6-7 years later. For what I cared, I was the richest kid in the world. Life was bliss.</p><p style="text-align: left;">A few days ago, I had a reality check. A Uniqlo storefront entry-level staff makes around 36000-42000NT. I was not rich. Actually, if poverty was a horizontal bar, I was killing it in a limbo dance. That explains why I had a vague memory of the MRT system, I was on the bicycle most of the time. I went to many museums and historical sites because the student discount was high. I rarely made it out of the city and the <strike>fine</strike> dining scene of Taipei was as foreign to me as I was to this country. Yeah I was pretty broke, wasn't I?</p><p style="text-align: left;">I didn't mind it then, and I still don't mind it now, post-reality check. When I think about the whole deal now, I don't feel like I was given the short end of the stick. I had a reasonably comfortable place to sleep whose rent I didn't have to pay. Besides my compensation and the old bicycle, Jake also paid for my tuition fee and often took me out for lunch. I was just low on disposable income. But to be frank, that was an afterthought. The 20-year-old me was blessed with the exposure I never experienced. Every day was a new situation I hadn't faced before. Every weekend was an adventure away. Every project was a stark contrast to the unrealistic school assignments that by then were all that I knew. Luxury and consumerism were not only out of my reach, they were also out of my mind. Without even being conscious about it, I was able to experience a learner mindset in its purest form. I managed to carry these starry eyes, and a dash of stupidity if I must admit, throughout my 20s. I am glad I got my first step right.</p><p style="text-align: left;">On another note, if Taipei had a face, it would be probably the face of a man in his 50s. A man who still maintains all the best qualities from his younger years, but subtly through the cracked skin around the eyes or the way he stands up, it feels like his best years were slipping away. Singapore would be in his late 30s, a force of nature with so many ideas and the energy to see it through. Saigon a kid in his early 20s, all talks, so few deeds.</p><p style="text-align: left;">The pace in Taipei is slower, and people know how to enjoy themselves. It is quite easy to find a park in a random neighborhood and both banks of the Keelung are reserved for outdoor activities. You don't see the elderly working, you are more likely to find them doing taichi in parks. The nightlife is vibrant with night markets. Alcohol and recreational drugs, for better or worse, are more accessible. And the entertainment industry punches above its weight. But I can't help to think of Taiwan as a country trapped in the past. The country was founded on a false hope that one day the Republic of China would be whole again. And today people long for the Taiwan Miracle that probably wouldn't happen for the second time. East meets West, future hope fuses with nostalgia, the more I learn about Taipei, the more elusive the words I want to find to describe my fondness.</p><p style="text-align: left;">During this trip, the one thing I thought a lot about yet still failed to fulfill was to see Jake for what could have been the last time. Jake was my boss at Cogini. He gave me first an internship and second a chance to run Cogini's office in Saigon. I wouldn't be who I am to be without the opportunities Jake presented and I eagerly grasped. Jake however was also an embodiment of the story a great engineer and a greater friend doesn't always make a great manager. That's the story for another time. I wanted to meet Jake because really how many friends of 13 years one can have. My excitement was met with an unfortunate event, Jake went back to the US last year to look after his mom. It makes sense though, his daughters went there for universities a few years ago and now he can be closer to his family. I would miss him though, now that the harsh reality has set in, I probably wouldn't be able to see him again.</p><p style="text-align: left;">I don't know when would be the next time I will be in Taipei, the world is so big and there are so many things to see. I sure hope it won't take another 13 years in the making. And that next time, I will just stay in Taipei for weeks. To find my younger self. To soak in all that nostalgia that by now has become the city's identity in me. To see that I am one more time its citizen.</p><p style="text-align: left;">The flight back to Vietnam is in a few hours.</p><p style="text-align: left;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg7fMWdDVlHV1jNNeJmYHznVBL0BL3GdcvXCUSD9kdhPVHnBR4b3IHb9r-o9_Fhr1mMCdPQfNCQG1vJmJuWGUzkfOhRGAzBcg5DO1DJ5xHpg-XoAwrZLFQIMl1GmuPldzxXsVOPg-leLOf7E5tcLhYNzGuTVfGkBEsrYtdieaKrpz5AfoECRJQO_gch0z6q" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1701" data-original-width="3025" height="360" src="https://blogger.googleusercontent.com/img/a/AVvXsEg7fMWdDVlHV1jNNeJmYHznVBL0BL3GdcvXCUSD9kdhPVHnBR4b3IHb9r-o9_Fhr1mMCdPQfNCQG1vJmJuWGUzkfOhRGAzBcg5DO1DJ5xHpg-XoAwrZLFQIMl1GmuPldzxXsVOPg-leLOf7E5tcLhYNzGuTVfGkBEsrYtdieaKrpz5AfoECRJQO_gch0z6q=w640-h360" width="640" /></a></div><br /><p></p><p style="text-align: left;"><br /></p>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-11936627859754621102023-07-16T23:10:00.004+07:002023-07-17T11:52:59.340+07:00Do Agile and be agile<h2 style="text-align: left;">There is agile and there is Agile</h2><p>One is an adjective and the other is a proper noun.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p>agile <span style="font-family: courier;">adj /ˈædʒaɪl/</span><span style="font-family: inherit;">: </span>able to move about quickly and easily; able to think, understand and respond quickly</p></blockquote><p>In business terms, agile refers to dealing with new situations or changes quickly and successfully. And for people dwelling in software craftmanship, it means your mindset and behaviors are inspired by the 4 values and 12 principles stated in the 2001 Agile Manifesto.</p><p></p><ul style="text-align: left;"><li>Individuals and interactions over processes and tools</li><li>Working software over comprehensive documentation</li><li>Customer collaboration over contract negotiation</li><li>Responding to change over following a plan</li></ul><div>For example, if the requirement changes midway through a development process, it should be accommodated, perhaps not in the current iteration but in the next. Because the agile mindset believes in collaboration and responding to change over a strict plan no longer reflecting reality.</div><div><br /></div><div>Note that no place in the manifesto dictates what you need to do to be agile. If a dozen of people who all share this set of values come together and start building software, fat chances are that they will agree on the general direction where they should be going but fight each other at every turn on the fine details. It is because the manifesto is literally a set of loosely defined beliefs. It lacks all the trademarks of the software development process we have grown accustomed to: iterative development, sprint planning, feature backlog, a small amount of WIP, etc.</div><div><br /></div><div>That is where big 'A' Agile comes in. An Agile process is a methodology, a clearly defined system of practices, whose principles are consistent with the manifesto. There are different flavors of Agile process, the big 4 are: Scrum, Kanban, Extreme Programming, and Lean Software Development (ordered by popularity, judged by myself). What they all have in common is that they take the 4 values and 12 principles and derive a framework encompassing the entire life cycle of software development, from the first seed of an idea to production release. Each of the practices in these frameworks is useful in its own right, but together they are greater than the sum of the parts. They embrace each other and create greater values. Take a humble burndown chart for example, it illustrates a project's progress toward the finish line, but it embraces story point estimation (which is the unit of the chart), user story writing (each story delivers a piece of independent value, and can be released on its own right), and minimize WIP (stories can be tested and accepted as soon as the implementation is done).</div><div><div class="separator" style="clear: both; text-align: center;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_Ql_Xn3zeHi0V6-agdYAcUaJXJ0KcpnNQ3h134_fBv6b59qGXBW5Nki2VqdR2SbNKrCU7vq09hTHdO2nHUlLSxfdSFlh436ua70DRbvswCse9EbxPhL6qWZLmH3ZDAWrIdp0YRrt59TXJ3mn8kxx8_6bb1GvxOeHz7ymo_Uf5RMvaFQE-07t-5hsmaM5Z/s2155/SprintBurnDown_Post.jpg" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1115" data-original-width="2155" height="332" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_Ql_Xn3zeHi0V6-agdYAcUaJXJ0KcpnNQ3h134_fBv6b59qGXBW5Nki2VqdR2SbNKrCU7vq09hTHdO2nHUlLSxfdSFlh436ua70DRbvswCse9EbxPhL6qWZLmH3ZDAWrIdp0YRrt59TXJ3mn8kxx8_6bb1GvxOeHz7ymo_Uf5RMvaFQE-07t-5hsmaM5Z/w640-h332/SprintBurnDown_Post.jpg" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">src: http://scrumbook.org.datasenter.no/</td></tr></tbody></table><div class="separator" style="clear: both; text-align: center;"><br /></div></div></div><div>In the same way constitution and law work, small 'a' agile is inspiring but Agile is what brings people together and allows them to collaborate somewhat efficiently. You are agile and do Agile. Agile was the best thing happening in software development... 20 years ago.</div><div><br /></div><h2 style="text-align: left;">What happened in the last 20 years?</h2><div>Big 'A' Agile became a victim of its own success. Agile started as a small movement among software development enthusiasts and gained so much traction that it became an industry on its own in which everyone wants a piece. The following awesome map demonstrates how the small movement became a conglomerate of management processes.</div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjse8_r6xJOJYycLnfRDWZkjTZo_DPDclB_FPsyABomO8XZSUVq1W-CSt2WeVPZqzfOOwJA7zEeBLQjALBVtrZdymaebfGjpQngDOBB64Pg1BvX99Ff0tQBRZW9y4b-twLbbDiSL1kx4JDIlOlGP_r76Ms7fgmKH-9mvoEzaESjHw-7LWw_zsIFP4swxocQ/s2048/agile-placemat-v9-1-2048.webp" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1536" data-original-width="2048" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjse8_r6xJOJYycLnfRDWZkjTZo_DPDclB_FPsyABomO8XZSUVq1W-CSt2WeVPZqzfOOwJA7zEeBLQjALBVtrZdymaebfGjpQngDOBB64Pg1BvX99Ff0tQBRZW9y4b-twLbbDiSL1kx4JDIlOlGP_r76Ms7fgmKH-9mvoEzaESjHw-7LWw_zsIFP4swxocQ/w640-h480/agile-placemat-v9-1-2048.webp" width="640" /></a></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">There are so many Agile flavors now, I mean, look at the map, that when one claims to do Agile in one form or another, and everyone does, she might as well not say anything at all. To do Agile used to mean the development process follows a certain distinguishable pattern, today it is the equivalent of saying I am breathing.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">One more, look at the top left corner of the map, it was made by a consultant. Deloitte, McKinsey, BCG, Scrum Alliance, and a nameless army of certified scrum masters and their dogs have turned Agile adoption into a consulting industry. To ensure their own values and usefulness, these consultants, consciously or not, are the reasons why there are so many Agile flavors, so convoluted, and foreign to even software development professionals. It is believed that was the same reason why religious rites are complicated and foreign to the majority of the population, the priestly people needed to demonstrate their "usefulness". </div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Agile under the influence of a profit-seeking industry has been reduced to an empty shell of its former idealization.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">FOMO and forceful adoption, many companies now do Agile as if it is just another checkbox in a list. People do Agile while refusing to be agile, they follow the practices mindlessly. We write stories that can't be released on their own, a cluster of stories needs to be deployed and rolled back together. Every 2 weeks we do a Sprint planning but keep both deadlines and scopes, no sign of flexibility is observed. The retrospective meeting is either skipped or used as a gossip forum without improving the working environment or process. </div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">I have been ranting like agile was pure and good and Agile got spoiled by human greed. But that is not all of it.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">The design faults of agile and where we are heading post-agile is a topic I would like to explore in a future article. But look, created some 20 years ago, Agile was made in a world different from the one we are living in today. Back then software was less complicated, written by a smaller team, and managers were unfamiliar with software development hence the need to continuously demo and showcase.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Agile believes collaboration between builders is the key to successful working software and looks down on heavy investment in processes and documents. But as team size gets bigger, complete collaboration also gets expensive. As in, it takes a lot of time if we insist there is no big design phase and the best design is the one emerging during the implementation process. We want to look down on documentation because it is an artifact of bureaucracy but documents make a project long-term maintainable and allow different teams to work with each other, not working software. The world post-pandemic also sees a rise in decentralized teams, collaboration without documents in that context is simply not making the most out of the setting. And we think contracts do nothing good but promote constraints, yet nobody bats an eye at service <i>contract</i>.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">It has been a long way to say that in 2023, Agile is not dead but it is less relevant than what it used to be. Seems like agile, as in the set of values and principles captured in the manifesto, shares the same fate despite its lingo difference. And that is the way things should be. In a world that keeps changing, change is the only constant.</div></div></div><p></p>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com1tag:blogger.com,1999:blog-308191087296040302.post-16174479089415473712023-05-31T16:53:00.001+07:002023-06-04T09:13:36.185+07:00ISO 27001:2013 Audit<p>At the beginning of May, my company went through a 4-day audit for ISO 27001:2013. I was responsible for some parts of the certification process and wanted to write down some thoughts on what I considered an interesting occasion.</p><p>Before we proceed, I must make it clear that I wasn't in charge of the entire ISO certification process, that would be our IT Risk & Compliance Manager - Satya. I did however design several systems and engineering processes examined in the audit. I think I can offer insight into what one can expect if she is an engineer going through the certification.</p><h3 style="text-align: left;">What ISO 27001 is and Why it matters</h3><p>ISO 27001 is a certification of information security. It sets out the specification of an ISMS (information security management system) and covers people, processes, and technology. First introduced in 2005, it has since received 2 revisions, 2013 and 2022. We decided to go with the 2013 specification because ISO 27001:2022 was out in October 2022 when we were 3 months into the preparation.</p><p>Because we are a B2B SaaS, our sales cycle is considerably lengthy with procurement being a major time hoarder. Any enterprise is responsible to ensure that any data subprocessor (which we are) complies with its existing technical and legal obligations. That means our team has to go through pages and pages of questions - both technical and legal, some left you head scratching for hours (very unhealthy especially if you are bald) - just to prove that we know what we are doing when it comes to data security.</p><p>International certifications like ISO 27001 cut this cycle short, in the same way IELTS/TOEFT allows you to skip certain English classes. And after decades of enjoying little to no regulation, the last five years have seen a rise in data regulations like data cannot leave a certain geography region or the right to be forgotten, and it is only getting stricter from here. Hence it was a no-brainer for us to do it while we were still at the size where radical changes were still feasible.</p><p>Business gains aside, I also wanted to learn from this opportunity. I have been quite verbal about how Parcel Perform is the most complicated system I have ever built and that whatever I knew about large-scale systems, I learned from this very experience through trials and errors and sleepless nights. I couldn't reliably objectively tell whether my work was a state of the art or a big ball of mud, much like any proud mother looking at her child. Once in a while, it is nice to get external validation as well as a chance to learn what is missing.</p><h3 style="text-align: left;">What to expect from ISO 27001 certification</h3><div>ISO 27001 is a risk management process. It is more of a thought framework than a checklist. I find it similar to a design pattern. You learn the problem space a pattern excels at then apply it with your own tweak. Between 2 projects though, the implementations of the same pattern might look different. ISO 27001 does the same for risk. Instead of dictating what (not) to do, it gives you a framework to think about risk.</div><div><br /></div><div>The framework goes like this</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg71sYiw1TACqov9KL7qgCQzRHUPTf0JlJ7Guxa6kcZwQ86U3ABFyN9C6BAAoDoDnXvRVG4LNfyhrYWf7b0G0N2Mt4aJtNBrkF-NQCxr4fkHmeJ965e3GEB4xtFeq0aJ9Pzfo-fhlMXlIaJpqZhchbATTU35a2EcRFzjTXVm6g2i1w3ddL9pDmJXjoLLQ/s2732/IMG_0713.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1234" data-original-width="2732" height="290" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg71sYiw1TACqov9KL7qgCQzRHUPTf0JlJ7Guxa6kcZwQ86U3ABFyN9C6BAAoDoDnXvRVG4LNfyhrYWf7b0G0N2Mt4aJtNBrkF-NQCxr4fkHmeJ965e3GEB4xtFeq0aJ9Pzfo-fhlMXlIaJpqZhchbATTU35a2EcRFzjTXVm6g2i1w3ddL9pDmJXjoLLQ/w640-h290/IMG_0713.jpg" width="640" /></a></div><div><ol style="text-align: left;"><li>Given the nature of your business, register the risks associated with it.</li><li>Assert the level of severity of the risks on your business.</li><li>Implement a prevention plan for such risks.</li><li>Post prevent plan, re-assert the new level of severity. The final level of severity must be low enough to allow the business to comply with its SLA</li></ol><div>Samples, these were 3 risks we identified and addressed.</div></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh59dISn3ZJQY2XYxe0ZeT-HhwO2Zaq64K9rkYkrTbFcnxGrT_HDQF7fKObUooFhFsccCl8drDscyG0AAWhtwEYBOoWpZR-e8JtUXWgKDsX1_SSN7SJhPMDW33-rFgPDynraNp0d4nuMynCQvyAScfIqvi72mAuC0AM9BNBKOoT868wvTXxKyS-MspnEw/s2960/Screenshot%202023-05-29%20at%2011.09.31%20PM.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1130" data-original-width="2960" height="244" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh59dISn3ZJQY2XYxe0ZeT-HhwO2Zaq64K9rkYkrTbFcnxGrT_HDQF7fKObUooFhFsccCl8drDscyG0AAWhtwEYBOoWpZR-e8JtUXWgKDsX1_SSN7SJhPMDW33-rFgPDynraNp0d4nuMynCQvyAScfIqvi72mAuC0AM9BNBKOoT868wvTXxKyS-MspnEw/w640-h244/Screenshot%202023-05-29%20at%2011.09.31%20PM.png" width="640" /></a></div><br /><div>All documents related to this risk registration, including policies, practices, system designs, incident records, and whatnot is written, gathered, categorized, and submitted to the audit. Depending on the maturity of an organization, the time this duration takes varies. For us, it was a few months. As a part of this practice, we had to define some policies and processes that didn't exist before, like Cryptography Policy and Third-party and Supplier Risk Management Policy. One tends to think of information security as protecting against hackers, or at least I do because I watch too many Hollywood movies growing up. There are indeed elements of defending against attack but nothing too paranoid. For example, leaving your computer open and unattended for hours is not safe but a 5-minute lock screen is good enough, despite one of my team member enthusiasm to explain how much he can do in the time window. ISO chooses to focus more on the complete governance view point rather than dictating the precise implementation. Overall though, we got to observe a good case of convergent innovations. Many of our in-house processes responded well to ISO best practices such as the engineering process from inception to deployment, or the access control mechanism.</div><div><br /></div><div>Upon submission, on-site audits were scheduled. By default, ISO 27001 certification is issued per location. Singapore and Vietnam are places where our product is built and data analyzed so these are the main targets of the audit. Sales offices were skipped as data procession is not a part of their functions.</div><div><br /></div><div>The audit exists essentially as proof: the documents describe robust processes, yet would it crack under pressure and are we doing what we preach? The audit was performed in a series of on-site interviews between the auditor and the people in charge over particular topics like access control, software development process, incident management, and more. These were just the ones in which I played an active role. I noticed that the audit conducted these interviews in two ways. For small isolated topics such as access control, we went through the entire process before he probed with questions and we countered with reasons why the decisions we made fit our circumstances the best. For longer topics, like an end-to-end software engineering process, he went straight to the Q&A dance. That left us mildly disappointed because we even had mock interviews for those long sessions :)</div><div><br /></div><div>Findings during the audit are categorized into major non-conformity, minor non-conformity, and suggestions. In theory, as long as you can address all the non-conformities before the final audit day, you are good. For a multi-site audit, the end-to-end time can be a few weeks. In practice, a major non-conformity is pretty much a no-go. It indicates the lack of a mission-critical process without which information security effectiveness can't be achieved. Such a process takes time to form and even longer to put into practice and become an organization's second nature. In fact, I believe not having enough time in practice is the number one reason a quick attempt to rectify a major non-conformity is rejected. Minor non-conformities are quite easier, they indicate points of improvement in an established process and enjoy a much higher chance of being accepted if you can get them in in time. Suggestions are just what they are, non-consequential pieces of advice from the auditor based on his experience with other organizations.</div><div><br /></div><div>We didn't discover any non-conformity, major or minor, so we are expecting our certificate to come through soon. But that isn't the end of it. Under normal circumstances, the ISO certificate must be renewed annually and only lasts for 3 years before the organization has to go through the full circle again. For ourselves, because ISO 27001:2013 is set to expire in 2025, we will be resetting our certificate with 2022 specifications next year.</div><h3 style="text-align: left;">Conclusion</h3><div style="text-align: left;">Certification being a lucrative business as it is, I do find the whole process to be a positive learning experience.</div><div><ul style="text-align: left;"><li>There are the thought framework approach and the big to-do list approach when it comes to certification.</li><li>Hand waving can take you places but ISO like many certifications is one where the journey is more important than the destination.</li><li>There are many things that we have already done right with the system and the organization around it, the external validation is a great encouragement to our effort.</li><li>Our documentation collection can see more rigorous standards, though everything was produced correctly, it took a bit of time to gather them for the audit proof.</li></ul></div><div><br /></div>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-25878950564448748502022-12-24T08:08:00.003+07:002022-12-24T08:40:04.613+07:002022 Tech downturn<p>Tldr; The market is going through a period of downturn as a chain reaction of the world's economy. Lavish startup life is over, lean time is here. Product-market fit will be tested. But tech is here to stay and top engineers are highly sought after.</p><p>2022 was not a great year for tech. The industry was plagued with widespread layoffs, weak earning calls, plummeting stock prices, and an investment market that shifted from equity to debt. It was another link in a chain reaction of the world's economy: pandemic aftershocks, the war between Russia and Ukraine, oil prices, energy concerns, and weaker buying power. It just happened that this link hit me the closest.</p><div>How exactly did the tech sector get tangled in this whole mess? I think firstly that is what you get from a flat world (not earth), everything is more connected than ever. The very definition of "tech" or "big tech" is ambiguous. It can be anything from social networking companies to EVs and phone makers. Really, the concept of the tech sector is no longer as relevant as it used to be because every company is a tech company to some degree. The whole world economy didn't do well and tech was an integral part of it.</div><div><br /></div><div>Secondly, the industry has operated with a free flow of cheap money for decades. Since the collapse of the housing bubble in 2008, the FED had kept a low interest rate for almost a decade - an unprecedented event in the 60-year-plus history of the organization. Even the increase in 2015 was described at the time as "<a href="https://www.nytimes.com/2015/12/17/business/economy/fed-interest-rates.html" rel="nofollow" target="_blank">a vote of confidence in the American economy</a>". For a long time, the availability of cheap money meant there were strong cases for borrowing money from banks and making profits via investment, including tech ventures. The high ride however ended with the interest rate hike as the FED tried to control the US inflation. I however entered the workforce in 2010, as so did many other startup founders and workers. We thought tech as an infinite source of growth was a norm. We have never experienced a real lean time before. </div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjk7mN4XdMzubyfvxL-JXeoG643ZBZUW8o13ydi-X_GSPpARIi8CXOcCufuCm1oPu5X1lagjDHq-wT9pTHfNKPBccLxaKwflHUywrvu52AzwairNQZVXTU_cTDUcQq8SKRdUdcHb4HsjVkZd1A65QP_8u20okSi5XgfB7VD05uUiY1VFWJ3HUhqaARDhQ/s1784/Screenshot%202022-12-17%20at%201.23.13%20PM.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1176" data-original-width="1784" height="211" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjk7mN4XdMzubyfvxL-JXeoG643ZBZUW8o13ydi-X_GSPpARIi8CXOcCufuCm1oPu5X1lagjDHq-wT9pTHfNKPBccLxaKwflHUywrvu52AzwairNQZVXTU_cTDUcQq8SKRdUdcHb4HsjVkZd1A65QP_8u20okSi5XgfB7VD05uUiY1VFWJ3HUhqaARDhQ/s320/Screenshot%202022-12-17%20at%201.23.13%20PM.png" width="320" /></a></div><br /><div>The third point, the greater force of macroeconomy throws much of future forecast out of the window. Boardgames such as Agricola or Splendor, or computer games such as Factorio are known to be resembling what it is like to run a business. And in all of them, a common theme is that the move you want to do now must be planned a few turns ago. Much of what a business does today relies on what it is expected to deliver in the future. So when the future prediction is skewed, it drags the businesses into the mud.</div><div><br /></div><div>And oh boy are we bad at predicting 2022. I mentioned no one was expecting a war between European countries in the 21st century but there were more. Social isolation throughout the pandemic yielded great returns for tech companies in 2020 and 2021. But then the demand dropped as the world reopened. Turned out, it took more than a pandemic to alter behavior. E-commerce and food delivery offered great convenience and indeed enjoyed a higher adoption rate compared to pre-pandemic but they didn't replace traditional means. Working from home didn't become the dominant working mode. And consuming power was limited to necessities as the world braced for economic difficulties.</div><div><br /></div><div>The combination of cheap money and an over-focus on future delivery means most startups had chosen growth over profit for a long time. It had always been about acquiring the biggest slice of the market pie and only then turning to increase the profit margin. You had to because all your competitors were doing the same. There would only be a little room to grow if you had a tiny slice of the market regardless of how healthy your margin was. The cheap money guaranteed that if slow and steady was your strategy you would end up in a hostile takeover. That also means sometimes companies found themselves running faster than they should have. The widespread layoff we saw was the direct response to a decline in forecast demand and a need to reduce the cash flow till a better time.</div><div><br /></div><div>It was easy to pick up a sage voice and describe the crazy world that was 2022 with much hindsight clarity. Being on the ground, running a team of 100+ head counts, and experiencing the first lean time sucked monkeys balls though.</div><div><br /></div><div>Alright, so that was how we got here. Where do we go from here?</div><div><br /></div><div>The last time tech recessed in the 2000s, it went down in a blast. NASDAQ was tech-heavy and it took 8 years to climb back to where it was previously. Yet there are reasons to believe this downturn wouldn't be as bad. During the dot com bubble, we were in the exploration stage. There was this internet thing that was supposed to be a new era but nobody was quite sure what it could do so everything went. Great ideas mixed with crazy evaluation fueled by FOMO money. Pets.com was arguably the most famous flop. Kozmo and Webvan burned through billions of dollars as their grocery delivery models were not sustainable. Think Tools AG evaluated at CHF 2.5B without having a product. The noise was so bad that after the burst people believed the whole internet thing would just fade away, contributing to the long recovery. </div><div><br /></div><div>The time around, other than the crypto scene which is going down the exact same path, the rest of the market is a lot more mature. Companies stay much closer to reality, solve real problems, and have clear plans for monetization (at least I hope so). Tech is here to stay and people are just preparing to weather the storm.</div><div><br /></div><div>That being said, 2023 is not going to be a great year to raise equity. People who set out for a funding round would be less likely to receive favorable evaluations and terms. The formula used to be that you can plan for a round every 18 - 24 months, one led to another on the basis of momentum, and hope that Tiger or SoftBank will come in with a massive Series C or D and allow you to have some sort of secondary exit and just build the momentum, and maybe a SPAC will you liquidity quickly. Didn't work out quite well for Grab and Sea.</div><div><br /></div><div>At the same time, the general decline in demand across B2B and B2C remains a threat. The product-market fit (pain killer vs vitamin) is once again brought to the forefront. Startups solving more critical problems are more likely to survive. Long-term strategic product programs might be put on hold to make room for initiatives contributing immediately and directly to the bottom line. Some unfortunately will run out of time before things got better. While others are presented with a unique opportunity if the product-market fit is good in the new reality.</div><div><br /></div><div>Interestingly for startups to remain competitive, they need to invest in technology. The performance of a tech team grows slowly, depends much on ad-hoc knowledge management, and is the bottleneck of any innovation. Even though layoffs were spreading, the core of tech teams was protected. Ones that do not spend in the near term will undoubtedly fall behind in the medium term and risk not being around in the long run.</div><div><br /></div><div>The spending pattern might change. Software development in particular has always been one of those fields where the performance gap between top and average performers is tremendous. The term 10x engineer has been saturated to the point now it is more of a meme, but there is a kernel of truth there. In sports, running 1% faster than the next guy results in 10x compensation. But that's it, if a 1% gain costs 10x more, might as well get 10 of the slower ones. Engineering is different. A top performer can provide solutions that bad ones can't come up with, regardless of how many of them, and doesn't cost 10x as much. Then computer automation and scalability come in to multiply this productivity difference many magnitudes more. With cash flow running lower than before, recruitment is no longer a number game. Companies would pay top dollar for a few strategic positions but otherwise not double their team size any time soon.</div><div><br /></div><div>The startup boom has lasted for a decade. It has also faced so many scares, each time more money and power were poured in. But maybe it really is different this time.</div>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-59243340075161790482022-12-18T17:58:00.010+07:002023-12-24T16:42:17.998+07:002022 - Could this year have been better? Yes.<p>It was a lovely afternoon at the beginning of December when I wrote these lines.</p><p>Vy was at her class in the uni and I had the entire apartment just for myself. So it was quiet. Golden sunshine cast on everything, the breeze was pleasing, and the view over the canal was gorgeous. Despite that. I felt uneasy. Felt like there was something I should be doing, that I was losing my time but I couldn't figure out which. And that had been how I felt this whole year - things were dashing by and I was just trying to catch up.</p><p>A little reminder for future people about what kind of year 2022 was. It was the year when Tuan-Anh got married, Soc was born, and the whole world finally came out of Covid-19 after three years of social isolation and economic stagnation hoping for post-pandemic growth. Things perhaps would finally feel normal again. Except Putin said fuck that and evaded Ukraine with his little special military operation. The oil price was all over the place. No country seemed to agree with one another. In Vietnam, there were widespread layoffs because orders in the West were drying up. And in Saigon just last October you literally could not find gas for your motorbike.</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjuL1n7_JRFSHYWoZe8urBm8WwsOYmfjmS8fF2h8cOxtCmK1hINWhNwYwUHpf4hc8eKId0EO6Z0AXUQW9Pcnixaxrm9JVPwnFlDwlDM8LanNx6tuEjuM0jrs9CndNaFUUNEiDuSEugtJG4zeWH9mJ5_vdAphwu0v9jbZHNuqoMuEILxk-jFh9yJEszFg/s698/Screenshot%202022-12-04%20at%204.01.44%20PM.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="680" data-original-width="698" height="312" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjuL1n7_JRFSHYWoZe8urBm8WwsOYmfjmS8fF2h8cOxtCmK1hINWhNwYwUHpf4hc8eKId0EO6Z0AXUQW9Pcnixaxrm9JVPwnFlDwlDM8LanNx6tuEjuM0jrs9CndNaFUUNEiDuSEugtJG4zeWH9mJ5_vdAphwu0v9jbZHNuqoMuEILxk-jFh9yJEszFg/s320/Screenshot%202022-12-04%20at%204.01.44%20PM.png" width="320" /></a></div><h2 style="text-align: left;">The master's degree.</h2><p></p><p>No, it wasn't me. That was why Vy was at the uni on a lovely Sunday afternoon. She got a scholarship at the same place I got my bachelor's.</p><p>I was told that a master's degree is like doping. You use it when your career hits a plateau and you need a little push to go to the next level. And for a really long time, it was one of the most popular pieces of life advice I propelled. Watching Vy though, I learned something new. Once you pass a certain point in your life, you can no longer be schooled. That is, you can no longer entertain someone telling you where to look, what to do, and how to do it. You can no longer fathom the thought of slaving your days and nights away for a thankless assignment solving an imaginary problem just so some old dude can put a no less imaginary score on it. Guess I would not be educatable for a while.</p><p>Vy was crushing it though. She worked hard in her day job, got home, and then put on the second shift for her study, be it preparing for the next day's slides, writing assignments, or working out problematic drama queens in her group. You would not believe how childish some of these master's students could be. Perhaps that constituted their educatability. Anyhow, like any good partner, as Vy slaved away at school, I had been pulling my weight in the kitchen. To be clear, I was not setting up a gym in the kitchen. I looked after the place, prepared meals, and might or might not write a programming assignment recently. Felt like that was the least I could do to support her. She was working so hard, I wish she would be successful. Also, my retirement plan depended on that.</p><p>I got anxious though, it was hard to watch someone working that hard and not do a round of introspection. Was I growing in my career, that I had been better than who I was last year? Was I prepared to continue this path for another decade? Should I be doing something else other than sitting here writing? To be frank, I was a bit scared, like in those recurring dreams where just as you realized the exam was today, you also realized you haven't studied shit. No? Just me? There was a fear of missing out, a fear that something wrong was brewing and by the time you realized it was too late to rectify. I was sure English had a phobia name for that, or German.</p><h2 style="text-align: left;">The apartment</h2>During the pandemic, however, I learned to appreciate the importance of a proper home and how rewarding it could be if you got it right. For a long time, all that I did in terms of housing was alternating between piggybacking in one of my previous companies' apartment, sleeping on the floor at offices, and for the most part, shuttling between a cheap tiny rental unit close to my office during the week and my parents' second home at the weekend. While I thoroughly enjoyed the short commute and my parents never collected rent or embossed any restriction on my freedom, neither place gave me the certainty of ownership. I found it hard to put words into this feeling. Say, the sense of ownership at its core is the difference between working on someone else's code and just wanting to get shit done vs working on your own code, something you know intimately, and wanting not only to do the right thing but also to do it right. For my non-developer friends, err... it's a rental.<div><br /></div><div>The criteria for my "repository" were straightforward: close to my office (and hoped that the office was not moving), bare concrete for minimal waste and maximum customization, and a view that didn't constantly remind me I was living in stacked boxes. For the kind of budget I had, that didn't leave a lot of options, I started looking by Oct 2021 and got the paper done by Nov.</div><div><br /></div><div>Because the place was concrete and nothing else, I went through the whole process of briefing my needs, finding an architect, feedback loops (3 months), and construction (another 3 months). All in all, it was almost a full year from when I bought the place until I moved in. As the nature of construction dictated, the process was waterfall where mistakes in one phase would get really expensive to fix or even unfixable in later phases. With the same attention I used to examine a technical design, I 3D-rendered the apartment in my mind, suggested modifications, and browsed the second page of Google to find the right material. Only desperate people visit the second page of Google. There were plenty of things to think through, like the placement of built-in electrical sockets or the layout of cabinets, as I had neither the budget to redo nor the skills to DIY. </div><div><span style="background-color: white;"><br /></span></div><div><span style="background-color: white;">From start to end, the journey was exciting and fun. Building up a physical space left a tangible spark of joy that was usually missing from my profession - software engineering. But because of how involved it was, for a while, it was imprinted </span><span style="background-color: white;">in my mind that</span> at any point in time, there was something else in the design I had neglected, a modification made, and an article read. I would often find myself on a fine afternoon like today and wonder, should I be doing something else?</div><div><br /></div><div>It had been 2 months since I moved in and the imprinting was fading away. While it lasted though, I realized that, for me, the fear of either leaving something out or being left out that would lead to cumulative damage was real and primal. I can reason to myself that this too shall pass, that in the grand scheme of things it wouldn't matter, that I have done the best that I could but I can't shake off the feeling. There had been several moments this year where I got an acute "tic" that something wrong was happening without me knowing and by the time I knew what it was, it would have been too long.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLWSD8yuLi4BmQQdxtqtToqxYPtO8-axiMKDHzCq5rL9e9OjFQN4mVjMMhKq6XB8b-7K84Tj5vXd0YdmYK3ncOB7LPVQKZeQg00aaVp0ucqZGcZ6aM-OP4ILQRMt6aLECoGM76utBZxiWkp5NpZm22FCpDYzQg0Ohme9Ij0Sne77S8QLoFs05xxesCOg/s2986/Screenshot%202022-12-18%20at%205.57.05%20PM.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1050" data-original-width="2986" height="226" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLWSD8yuLi4BmQQdxtqtToqxYPtO8-axiMKDHzCq5rL9e9OjFQN4mVjMMhKq6XB8b-7K84Tj5vXd0YdmYK3ncOB7LPVQKZeQg00aaVp0ucqZGcZ6aM-OP4ILQRMt6aLECoGM76utBZxiWkp5NpZm22FCpDYzQg0Ohme9Ij0Sne77S8QLoFs05xxesCOg/w640-h226/Screenshot%202022-12-18%20at%205.57.05%20PM.png" width="640" /></a></div><div><br /></div><h2 style="text-align: left;">The tech downturn</h2><div>2022 was not a great year for tech. The industry was plagued with widespread layoffs, weak earning calls, plummeting stock prices, and an investment market that shifted from equity to debt. It was another link in a chain reaction of the world's economy: pandemic aftershocks, the war between Russia and Ukraine, oil prices, energy concerns, and weaker buying power. It just happened that this link hit me the closest.</div><div><br /></div><div>I noted down some of my thoughts on the downturn in <a href="https://blog.khangnguyen.me/2022/12/2022-tech-downturn.html" target="_blank">another post</a> where I said the greater forces of the macroeconomy were making it really hard to go against where the industry was going. Didn't help with the emotions though, did it? No one working in a startup would be happy with just cruising along the "industry average" path. Startup people are competitive and hold on to exceptionalism. Without those traits, who in his right mind would invest years of hard work charging against bigger corporates with deeper pockets and higher headcounts? So this setback was a blow of defeat.</div><div><br /></div><div>Here I had my phantom fear materialized. Had I written better code, spent more time with my colleagues, or initiated better technical investment, could I have soared above the average? I hoped so. I wished I could do better because this feeling that I was not enough, that I was asking for more than I could give was the most tangible sense of helplessness I experienced in a really long time.</div><div><br /></div><div>--</div><div><br /></div><div>By the time I got to these lines, the sun was on the horizon and I had already moved to the shade of a small dock overseeing a peaceful turn of a canal. I know this canal well, I have traversed countless trips on it, and yet its beauty never fails to capture my attention. I always feel attached to bodies of water. The tranquility made me nostalgic. I recalled the time when I was on kayak often, when life was easier, and when the net outcome of action was more direct and calculatable. But life moves on. Tomorrow I would board the first flight to Singapore to discuss how we could best weather the coming storm. 2023 will be challenging but did I say startup people are stupidly competitive?</div><div><br /></div><div><div>Stay strong.</div></div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI9RaG6c-iNpyq08qBxvAo0ITTPITK716KZGkMMLAXASvsgk1Uk6skDdcnk0-TdiBIpPaJaSGpjM_V1Le_Vl2Nga3yhNHLQFfTH6DQnkDWMsyUtym2qrKxG-bkdh2HfqjgYgVroQ0tT_kGf0dObM8ITGEuR79AgVM6X6idKk1g9ZsNUowHUfJQ9j1n49g2/s4080/PXL_20230101_092508796.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="2447" data-original-width="4080" height="384" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI9RaG6c-iNpyq08qBxvAo0ITTPITK716KZGkMMLAXASvsgk1Uk6skDdcnk0-TdiBIpPaJaSGpjM_V1Le_Vl2Nga3yhNHLQFfTH6DQnkDWMsyUtym2qrKxG-bkdh2HfqjgYgVroQ0tT_kGf0dObM8ITGEuR79AgVM6X6idKk1g9ZsNUowHUfJQ9j1n49g2/w640-h384/PXL_20230101_092508796.jpg" width="640" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-66681669853810116452022-06-18T12:59:00.005+07:002022-06-20T13:14:25.502+07:00Prioritizing development decisions<p></p><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjy7-2RpJeiHN8lw8YN6f-yEsRUh0I0FUekR1l4vhpFx6QnUWn1U2kvIbJqYP2xa2m19qfYe3a01uMFLVDaleq54MmwSLSzvceyGd7csfTZ2otOFlha-XIELbc9dIuOYbayS-KLCdf00PtzLsA80vi4XPPsvol5p873p2D-jQwb8rm_sY5cLj37R2ENxw" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="400" data-original-width="464" height="240" src="https://blogger.googleusercontent.com/img/a/AVvXsEjy7-2RpJeiHN8lw8YN6f-yEsRUh0I0FUekR1l4vhpFx6QnUWn1U2kvIbJqYP2xa2m19qfYe3a01uMFLVDaleq54MmwSLSzvceyGd7csfTZ2otOFlha-XIELbc9dIuOYbayS-KLCdf00PtzLsA80vi4XPPsvol5p873p2D-jQwb8rm_sY5cLj37R2ENxw" width="278" /></a></div><p>Great startup stories tend to share the same mold: a great founder dreamed of an equally great vision and followed it fearlessly till the world was conquered. History is written by the victors, of course, but it is relatively common for people to have a rough idea of what they want to build before starting the work. That’s the easy part.</p><p></p><p>But constructing a concrete step-by-step plan to deliver not even the vision but a mere release is hard work. A good plan needs to take advantage of both business and development expertise without letting one overpowers the other. If the business makes all the calls, the development time might be painfully long and the product crashes when traffic starts to peak. If the development decides, we might have a technical wet dream of solving a non-existent problem. That’s where the planning game comes in.</p><p>I can’t decide if a game or a dance is a better metaphor for depicting the collaborative nature of planning. Business and development, each possesses knowledge unavailable to the other and is unable to produce the entire plan. The work can only be done by combining the strength of both sides. In economics, a “game” refers to a situation where players take their own actions but the payoff depends on the actions of all players. Game Theory suddenly sounds less conspiratorially adventurous, doesn’t it? Dance on the other hand isn’t used as much in research literature so I ended up siding with the economists. That’s a sidetrack.</p><p>In an Agile team, a planning game looks like this:</p><p></p><ol style="text-align: left;"><li>The Product Owner decides the scope of the plan. Based on the purposes of the projects, the Product Owner prepares a set of use cases and explains why they are valuable problems to solve and why they should be done first.</li><li>The whole team breaks each use case down into stories. The idea is usually that anything requiring the team to do something other than normal company overhead needs a story.</li><li>The developers “size” the stories. They estimate the time each would take or its complexity. And then group stories that are too small, split ones too big, and decide what to do with stories they can’t estimate.</li><li>The Product Owner prioritizes the stories. Some stories won’t be worth adding, either unimportant or too far in the future.</li></ol><div>There are many things that can be said about the planning game, from Work In Progress should be minimized, the releases should be small and often, to the best answers for “why does it cost so much?”. Those are stories for another time.</div><div><br /></div><div>In this piece, I want to discuss specific friction in step 4 of the game where one story is prioritized over another. The stories laid out in step 2 do not necessarily project the same values to different team members. Some are pretty straightforward, implement feature X to earn Y money, the contract was signed. While others are more tricky such as implementing plug-and-play UI components so that future web pages are built faster. The second category usually comes from the development team who is one layer away from the users and so perceives values differently from the Product Owner. That is the breeding ground for misalignment.</div><div><br /></div><div>Product Owners want to release a solid, usable product. They also have to balance that with the desire to save money and meet market windows. As a result, they sometimes ask developers to skip important technical work. They do so because they aren’t aware of the nuances of development trade-offs in the same way the developers are.</div><div><br /></div><div>Some developers note down all the development options like a shopping list, “outsource” Product Owners to choose, and then roll in agony at the wrong decisions. If such a strategy didn’t work for <a href="https://www.brookings.edu/blog/order-from-chaos/2020/01/14/why-did-the-pentagon-ever-give-trump-the-option-of-killing-soleimani/" target="_blank">the guys at the Pentagon</a>, it wouldn’t work anywhere. Just as Product Owners are the most qualified to decide the product direction, developers are the most qualified to make decisions on development issues. Don’t delegate the decision, take the matter into your own hand. If a development decision isn’t optional then it shouldn’t be prioritized either. Just do it.</div><div><br /></div><div>Instead of:</div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div style="text-align: left;">Our notifications is crucial at informing customers the health of their business. To make the data pipeline behave transactionally, we have several options. Please let me know how should we prioritize them.</div></blockquote><p style="text-align: left;"></p><ul style="text-align: left;"><ul><li>Experiment with Flink’s TwoPhaseCommit, this is new to us so it would take time and be hard to estimate.</li><li>Get Sentry to cover all the projects, this is a passive measure as we passively wait for exceptions.</li><li>Add a check at the end of the pipeline to make sure no duplicated notifications are generated, the check will have to handle its own state.</li><li>Move the final stage of the pipeline to Django, it is a web framework that supports transactional requests by default and we are familiar with it.</li></ul></ul><p></p><div>Try this:</div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div style="text-align: left;">Our notifications is crucial at informing customers the health of their business. The data pipeline is long and consists of multiple nodes, each needs to successfully finish its work to produce a notification. To achieve this notion of exactly-once delivery, we need the pipeline to behave transactionally and every exception to surface swiftly. That is done via Flink’s TwoPhaseCommit and Sentry integration. The work will be done at the beginning of the project as it is easier to handle when the code base is still small. TwoPhaseCommit in particular is new to us so we will have a couple of spike stories to understand the technology.</div></blockquote><p><br /></p><p>When there is a business choice to be made, don’t ask Product Owners to choose between technical options. Instead, interpret the technology and describe the options in terms of business impact. To continue our notification example, before any notification is sent, there is a need to make sure the data we have is the latest. The conversation can go like this:</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;">We are thinking about adding another Kafka queue to request the latest data. We then need to join the request flow with the future trigger with some sort of sliding window, will also need to thinking about out of bound data. Our other option is to set not one but two future triggers so that one can request data and the other handles notifications. Which would you prefer?</p></blockquote><p>Try this instead:</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"></p></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;">We have two choices for ensuring a notification always works on the latest data. We can use a deterministic approach or an empirical approach. The deterministic approach would add a new data request flow right before the notification is sent. The notification is processed after the data request flow so we always sure the latest data is used. But because technically data procession and future notifications are asynchronously independent from each other, it would require several more stories for us to join them together. The empirical approach won’t take any extra work. We observe that it usually takes less than 5 minutes for a data request, so we can set two future triggers instead of one, 10 minutes apart from each other. The first one request data, the second notification. But the margin of error is larger because sometime there can be delay in data request. Which would you prefer?</p></blockquote><div><br /></div><div>And finally, no software engineering discussion would be completed without a talk about code refactoring. In the context of the planning game, it is mostly about justifying the refactoring effort. While it is tempting to do a “spring cleaning” hoping to refactor the whole thing back into shape, the sad truth is halting the development of working software for refactoring is hardly justifiable. Refactoring effort deals with risk (the old code can implode at any time) and potential (the new code is easier to work on). Those values are intangible compared to the usual subjects of a business decision (new features lead to a new set of customers lead to greater revenue).</div><div><br /></div><div>What do we do? Boy scout rule “always leave the campground cleaner than you found it.” Whenever you need to implement a new feature or fix a bug you see if that part needs improvement. Refactoring shouldn’t be a separate phase, it is part of everyday development. Once you nurture this culture of quality, there is nothing to justify.</div><div><br /></div><div>None of the above suggests the easiest way to avoid friction is to keep the business side in the dark while going on waving the engineer's magic wand. Communication remains the key to any successful project. There is more to a project's success than just business decisions, and working out a way to be a (constructive) part of the conversation is more powerful than a baseless delegation.</div><p></p>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com1tag:blogger.com,1999:blog-308191087296040302.post-73588760004663882712022-01-03T17:16:00.003+07:002022-03-10T18:34:14.959+07:00What makes an internship meaningful?<p>I got 2 kick-starts in my career, a part-time job in 2009, and an overseas internship in 2010. I dropped a production database on the first day into the former, and was so ill-prepared in the latter I didn’t look up the place’s climate and local language till I was already on the plane. Both were privileges. They catapulted me into worlds I couldn’t fathom and shaped my career in the following decade. In turn, as I have built startups, I have also been on and off in building internship programs. My number one concern is to replicate the magics I got to experience for the next generation. I am writing this for you who are seeking an internship opportunity.</p><p>What makes an internship meaningful? I ask this a lot, and the most common answer I receive is “<b>a new experience outside of school</b>”. That’s right. Much of what we do in life is to propel ourselves into new experiences. As a species, we are a bit of an adrenaline junkie (and serotonin, caffeine, and cocaine). But it is not particularly good at answering the question.</p><p>A <b>new experience </b>is undoubtedly crucial for an internship, but if it is solely what makes an internship meaningful, wouldn’t picking up something entirely opposite to your education offer the most exposure? I am a software engineer by train, but the course that awed me the most in college was a general elective, macroeconomy. It expanded my horizon, but I wouldn’t have considered an internship in a central bank, it would have ventured too far away from what I wanted to do. An internship should represent a certain level of <b>relevancy</b> to your long-term career.</p><p>An internship is only as meaningful as the <b>mentorship</b> it can provide. The transition to the workforce is the transition from theoretical information into practical knowledge. The rigid structure of the curriculum makes it hard to grasp some key concepts of software engineering. At the end of my degree, I was still not sure if my code was clean - there was no need to reopen last semester’s assignments, my architecture was flexible to changes - no assignment lasted for more than a semester, or my software could be released in small, iterative circles - all assignments were fixed time-box. And it took years for me to get comfortable around these concepts. It was frustrating navigating in uncertainties. I wish someone was there telling me what I did wrong. In the language of the Dunning Kruger effect, my descent from the peak of Mt Stupid to the valley of Despair was rough.</p><p></p><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1du9y6ut2p4DjClzYcy1nORXCsL3bpoIkmfdrcoMEXil7mHGv23ZywVBmBdqwiYzsESExZ2VK78wrQIcAG3PzOYRVCaFxFFoqIeQNkFmqhXmsgAjZkIbUq_dGCSCleQzmzobbJproSgrZ/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="223" data-original-width="512" height="139" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1du9y6ut2p4DjClzYcy1nORXCsL3bpoIkmfdrcoMEXil7mHGv23ZywVBmBdqwiYzsESExZ2VK78wrQIcAG3PzOYRVCaFxFFoqIeQNkFmqhXmsgAjZkIbUq_dGCSCleQzmzobbJproSgrZ/" width="320" /></a></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">A good mentor can ease the journey. Mentorship provides a sense of trusted guidance a personal friend can give with the bonus rule of being the manager. They help you to transit from a student to a professional.</div></div><p></p><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: left;">Lastly, an internship implies getting out of the sandbox. At school, you are learning in isolation, a designed system, a less glamorous version of The Matrix. You spend your days solving imaginary problems and gaining imaginary successes. None of which reliably predicts your success at work, otherwise everyone would be boasting their GPAs now. And if you slip here and there, other than some awkward group assignments, there is hardly any difference in the grand scheme of things.</div></div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: left;"><p>At work, however, there are consequences to your actions. There might be works whose input relies on your delivery. There might be deadlines that are not arbitrary choices dictated by a course’s curriculum but coordinative milestones to align other efforts. And there might be actual end-users of whatever you are delivering. An internship that can’t offer <b>this sense of ownership</b> is nothing but yet another simulation box.</p><p>Compared to the previous two elements, giving someone who has zero experience ownership is considerably more complicated. Many internship programs are decided by management or HR.</p><p></p><ul style="text-align: left;"><li>Management wants to establish a partnership with a university and an internship is a low effort investment.</li><li>Next year recruitment might get harder if none happens this year. Out of sign, out of mind.</li><li>Doing the dean's office a favor. True story.</li></ul><p>While there is nothing wrong with these reasons, too often they are used as excuses to force the internship upon the team adopting you. A forced internship can go wrong in a couple of ways. </p><p>(1) You can be treated as junior developers and go through the same onboarding experience. The thing is, internships are typically short, and by the end of it, you would come back to school to continue your studies, usually, right at the point the onboarding has just been over. The internship is then an overwhelming experience for you and a waste of time for the team.</p><p>(2) You might not be ready to work on production code yet so a common solution is to make up an internship project nobody needs. The general lack of coordination between those who want to host an internship and those who are in charge of it results in uncreative half-assed ideas, such as cloning a working piece of software, building an internal tool whose specifications were a few Slack messages, conducting an experiment that the mentor has not made any progress recently, etc. </p><p>A better internship experience can be built with buy-in from the team. An internship fulfilling the needs of the team can be a constructive experience for everyone involved. It could be to give someone new to management a chance to run his own (intern) team before reaching out for a more substantial responsibility. It could be an isolated project with actual stakeholders who have good incentives to invest in its success. And it could also be an experiment, a research topic, but goals, direction, and companionship should come with it. In short, seek out to align the success of the team and the success of the internship.</p><p>Too many internship hours were wasted on demo code no one reads and pet projects no one uses. I hope yours would be better.</p><p></p></div></div>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-20525340402077988612021-11-06T12:15:00.036+07:002022-12-05T10:30:44.017+07:00Everything seemed normal<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8dv_9mrTNjxvtNMqELa6-a3oTBDgtOGnatXG50IVjXLbVburOfkdxMxfNCkhrTZJ6paZhh85-COf5-pXzSwap7gfdfzrWYpAcer8k-YRopRxX11iklfHRaLcVU7gdrTv7zfHfj2veX6nB/" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img alt="" data-original-height="720" data-original-width="1280" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8dv_9mrTNjxvtNMqELa6-a3oTBDgtOGnatXG50IVjXLbVburOfkdxMxfNCkhrTZJ6paZhh85-COf5-pXzSwap7gfdfzrWYpAcer8k-YRopRxX11iklfHRaLcVU7gdrTv7zfHfj2veX6nB/w640-h360/image.jpeg" width="640" /></a></p><p>The air was warm, humid, and dense. I was panting after a badminton loss. I had never been good at this, or any sport for that matter. But I liked flexing the muscles after months of home confinement. Stepping out of the stadium, a rundown building surrounded by brick walls and metal sheets, the air was a lot more pleasant. The location of the court was nice, right next to a river. An adult was sleeping on a hammock under the shade hovering over the river bank. Half a dozen of kids were swimming in the grayish water. I couldn't help but notice, on the other side of the bank, a slump area where the river was, among other things, part of their toilet and sewage. Right at that moment, it was easy to forget Saigon was a cosmopolitan of ten million residents and the engine of Vietnam’s emerging economic prowess. Under that facade, Saigon was still pretty much a part of South Vietnam - a maze of rivers, canals, and delta farmlands - and the fate of the city and the land is anything but one.</p><p>Everything seemed normal except that it shouldn’t. We just struggled the whole ordeal of a four-month lockdown. Covid was still reigning across the country. Vaccine coverage was around 30%. In the rural area, agricultural produces was left to be spoiled on the field as it would cost more to ship elsewhere. Waves of laborers retreated to their hometowns without the slightest idea of what their future looked like. Meanwhile, in the city, everything cost more and the labor shortage was high. For the first time since the stats were made available in 2000 Vietnam recorded negative economic growth of 6.17%. Real estate, stock, crypto, and even gold markets were reaching their historical records, not because of the belief of a V-shape bounce back, but because of people hiding their assets for the coming inflation once the stimulus hit.</p><p>Everything seemed normal but the substance had changed. The life of the elderly was more fragile. Break-ups and divorces rose. Worst of all, young people, including kids, were robbed of 2 years in the most important period of their lives. Covid was unprecedented. It exposed many of us to our worst enemy, change. Some could brag how embracing change was part of their DNA, but let’s be real, few people turned their life upside down for the kick of it. Changes forced people to face uncertainty, make decisions without a mental model, and live with the consequences. Take an example: kids and schools. Among people my age, the number one reason for self-enforcing isolation at home was the worry that they could bring the virus home to the kids. The local government expressed the same concern, two months after lockdown measures were eased, kindergartens and schools had yet to open. For the entire of 2021, schools had open for around 4 months. While infection risk and its impact on kids were uncertain, it has already been a certainty that the lack of interaction with same-age peers damaged the well-being of kids. Which was more critical, a 0.1% chance of infection, or a 100% chance of development impact? Reports and studies were one thing, but when it came to our lives, the choice was personal, emotional, and far from statistical. Whatever the decision, the scars left on the current generation would be slow to fade.</p><p>A few days ago, Charlie Munger stated <a href="https://www.barrons.com/articles/charlie-munger-markets-overvalued-crazier-than-dot-com-bubble-51638528287" target="_blank">markets were even crazier than the dot-com bubble</a>. He might be right. Statistically, he had been right more often than he was wrong. But the same statement could be found in <a href="https://en.m.wikipedia.org/wiki/United_States_housing_bubble" target="_blank">2006</a>, and then in <a href="https://www.businessinsider.com/this-stock-market-bubble-worse-than-dotcom-boom-2015-8" target="_blank">2015</a>. As the world got better connected, it had been inevitable that we dealt with bigger crises impacting more people. I believe that eventually covid and its crazy variants would be over, that we would treat covid without much discrimination from its influenza cousins, and everything would seem normal again. But make no mistake, the course of the world would never be on the same trajectory had covid not happened, and normalcy is anything but a wet dream.</p><p>In the river, the kids, tired of breaststrokes, had changed their game. The older kids were hopping on a floating traffic marker on the water for big jumps, betting on who could make the biggest splash. The younger ones speculating from the safety of their bright-orange life vests. A few years ago, there would have been neither markers nor life vests. Done mourning my loss, I stepped back to the court for the next game.</p><p></p><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><br /><p></p>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-10320876080801984712021-09-26T14:30:00.006+07:002021-09-26T14:47:47.998+07:00The case of Project Manager<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2ooj0_o9NEwnS-vZENhBqQ7FM9Vo_gyX0CsidfvSDygdh-rfJmTxORhivcdfQKBsrrmwmP3F6JJY9cta3HLiu5aVOAMLZNcVAc4gmz8meY7KhwN79fAQ-w0PuXKLlFL3UT0Or2Suq8eRB/s2271/C3347A20-AEDC-4FDF-946A-40AC938B2185.jpeg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="832" data-original-width="2271" height="234" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2ooj0_o9NEwnS-vZENhBqQ7FM9Vo_gyX0CsidfvSDygdh-rfJmTxORhivcdfQKBsrrmwmP3F6JJY9cta3HLiu5aVOAMLZNcVAc4gmz8meY7KhwN79fAQ-w0PuXKLlFL3UT0Or2Suq8eRB/w640-h234/C3347A20-AEDC-4FDF-946A-40AC938B2185.jpeg" width="640" /></a></div><br /><p>I have never used a clickbait thumbnail for my post before, not even <a href="https://serverheist.com/" target="_blank"><b>when I nuked a production database</b></a>. But admit it, the thumbnail got your interest this time. Let's see if I can deliver.</p><p>The role of a project manager (PM) is somewhat controversial in the software development community. Generalization is bad. There are bad PMs and there are bad engineers. But you don’t quite see the same question posed for product and engineer people, the other two pillars of software development. Google even tried to <a href="https://www.businessinsider.com/larry-page-the-untold-story-2014-4" target="_blank">let go of all of its PMs</a> and had all their engineers reporting to a single VP of Engineering. What makes PM different?</p><p>The opinions about PM are more subjected to bias than others. A lot of PM work happens behind the scene. When work is running smoothly and on schedule, every day is business as usual. At times, it feels thankless and unappreciated. It’s only when a project is plagued with issues that the PM starts getting attention - usually not in a good way. The strategic values of project management tend to take place before shit hits the fan. After that, primitive instincts kick in, engineers keep their heads down and work harder, the process is out of the window, and a PM seems to only stand in the way. A great PM could still shield the engineers, prioritize work so the worst fire is put out first, communicate the impact to stakeholders, and plan for the next step. But usually, a great PM wouldn’t let the shit hit the fan, to begin with. In other words, PMs are judged after things have gone FUBAR, and everyone can afford to be smart in hindsight. The same hindsight you probably had learned from your angry parents after those end-of-year conferences with school teachers. No mom, I wouldn’t have skipped school that day had I known there was a test. </p><p>Every good PM succeeds in his own way, but bad PMs all fail the same: they didn’t do what they are supposed to do. What are they supposed to do? Sadly there is no universal truth. Granted, each company operates in its own way. At some places, SWEs deploy their own code, while in others, SREs reign supreme power over the production system. Some product owners are supposed to help with GUI design (hopefully based on some standard design system) or help with market research. But by and large, the outcome of members in a tech team is bounded to tangible deliveries and the inter-company differences are within standard deviation.</p><p>The PM role? They are supposed to be responsible for the success of the freakin’ project. Now that’s wild, how is that translated into concrete action is anyone’s speculation. That might mean being the link between engineers and customers, making sure both sides get what they want when they want. It could be about being a master of process, fluent in a range of management methodologies, and having an eye on constant improvement. Perhaps it involves understanding the web and mobile architecture, knowing modern technologies, and being quick on the uptake. Or I might have as well just described an Account Manager, Scrum Master, Technical Consultant, and part-time Avenger. PM’s scope of work is ambiguous by design as no two projects are alike and project management is the glue pulling things together, but unclear expectations are the breeding ground of disappointments.</p><p>Another thing that gives the PMs bad rap is the disdain for technology of some of them. Software development is not rocket science but it is not exactly a pure exercise of muscles either. That is to say, the field is somewhat technical. And just as any other technical field, it is full of useless jargon, lame inside jokes, and know-hows that take years to pick up. Non-technical PMs, people who experience difficulties explaining how a website works to a 5-year-old, manage to navigate around this quagmire by materializing all units of work into tickets. And they proceed to treat the tickets as little black boxes. The meaning of the work is watered down into start and end dates, and a set of labels for convenient herding. This makes a complicated system simpler. Some’s navigation skills are good enough that they don’t need to know how the finished work would look like. Work to them is parcels to carriers, you are supposed to ship it, not knowing the dead bodies inside. As I work for Parcel Perform, this sounds great!</p><p>There is a problem though, most people in software development are not in the business of writing code, they are in the business of building software products - hopefully, great ones. The difference is that one is an isolated piece of work with a predefined outcome, the other is a process of figuring out the intersection of market fit and technical prowess. Building non-trivial software requires plenty of arguments, negotiations, and decisions. Non-technical PM can’t call out bullshit. They lack the skills and tools to affect the outcome and all important decisions were handed to engineers and product owners. Without that decision-making capability, PMs turn into administrative assistants, busy themselves with monitoring project status, sending out updates, and keeping track of who does what. It is nice but it doesn't break or make a project.</p><p>Some PMs are shit umbrellas. Some are shit funnels. And the canopy of the said umbrella is technical knowledge. Not the same knowledge that is required for engineers to write code, but the knowledge to see through a project with clarity, know what is important and what is not, and call the shots when needed. It is not unusual for an engineer to pick up a book on Agile to better align his working habits with Scrum’s sprints. The sight of a PM reading Google’s Site Reliability Engineering to come up with a better fire fighting routine is as common as the sight of a <a href="https://en.m.wikipedia.org/wiki/Saola" target="_blank">Saola</a>. Isn’t that having the cake and eat it too?</p><p>Are PMs useless then? The bad ones are. Bad PMs pose negative net morale and productivity for their team. Don’t believe me? Try rewriting the software for the third time because your PM failed to strong-arm a customer to put his shit together. But the whole thing about Google not needing PMs is as much of an urban myth as it is truth. It happened in July 2001, in computer time that was a century ago. It wasn’t the mighty Google where every practice seems to be deliberated, there were around 130 engineers. The layoff didn’t stick. The engineers themselves opposed it. The whole thing lasted for less than a month and was pretty much Larry Page throwing a tantrum against Eric Schmidt’s adult supervision.</p><p>As long as software products are written by humans, the role of project management in coordinating a bunch of professionals toward the common goals is always needed. People who think project management is a good way to get into the tech scene without the background need to get their reality check. PMs are responsible for the success of a project even though they have little control over it - how exactly is that easy? It takes a lot to be a good engineer, and it takes even more to be a good PM. The more people on board with this thought, the better it is for everyone in software development. Don't let a bad apple spoils the bunch, and the best way is to work on being a good one.</p>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-59862352475245383292021-08-15T14:09:00.005+07:002021-08-21T16:56:09.478+07:00The engineer/manager spectrum<p>When I was younger, I was given many forms of the same advice: at some point, I would be better off to double down on either being an engineer or a manager if I wanted to progress in my career. I was booted from my first job while pretending to be good at everything, naturally, I thought this was a good idea.</p><p>Life looked at my choice with as much consideration as my parents when I told them I wanted to join the infantry in kindergarten - which was not at all. I joined a startup, and subsequently another startup. I never got to choose. Working at an early-stage startup is like conducting an old locomotive on a half-done railway. You juggle between keeping the engine running, because just like your code it breaks a lot, and building the railway with whatever you find along the way, you are selling a product before it is fully ready. The game is over when either stops.</p><p>Along that journey, I realize that career choices are not complicated, they are complex. Complication is when a system consists of multiple parts with intricate interactions. As overwhelming as the parts and interactions could be, they follow some universal laws that eventually make the system knowable. Complexity is when the parts and interactions are not fully knowable and the best we can do is some reasonable prediction. A car is complicated, the traffic is complex. I think people who gave me advice earlier understood this. But in an attempt to simplify a complex subject, half of the truth was lost.</p><p>I think when people talk about the engineer/manager choice, they probably mean that at the point you have to weigh your options as being an engineer or a manager, you have collected all the low-hanging fruits in your career. To proceed, you need a level of focus you haven’t experienced earlier. And as it is hard to focus on two hard things <i>at the same time</i>, you would be better off focusing on only one. Which is also true for my parents’ expectations of me.</p><p>While that is true, to perform at the peak level you cannot spread your effort all over the place, there is another half of the truth, that is the choice is neither binary nor permanent. You are probably overthinking if you view the management as a critical, life-changing decision. The move to management is not a promotion, it is a change in scope of work in which you drop existing responsibilities to pick up others. And if you hate the new responsibilities, you can always go back to the engineering track. Most companies have had sufficient understanding on this matter and will be glad to provide some rotation options for their employees.</p><p>The higher you advance in your career, the stronger the element of leadership exists in your scope of work. People in the technical track get promoted for working on hard and impactful problems. You wouldn’t get to work on the hard problems unless the easier ones were effectively delegated to someone else. The most impactful problems tend to be ones that once solved provide sustainable advantages for the business. Such problems tend to involve people of different background and function. Someone would need to rally them and align their interests to the shared goal. And yet delegation, communication, and goal alignment aren’t common practices among engineers. Although technical leadership isn’t exactly the same as people leadership, I think you will navigate better if you have an opportunity to learn about how people think, how larger projects are prioritized , and how organizations are run.</p><p>From my own experience, alternating between the two roles makes me better at both. In a hard project that challenges my technical prowess, I am more of an engineer. In a more relaxing project, my manager role is stronger so I can oversee things go to the right places. I am a better manager because I understand the morale friction caused by a poorly planned project, and I am a better engineer because I know the red flags of a poor project and when to fire alarms.</p><p>The final caveat is that engineer and manager roles exist in a spectrum and it curbs your professional and personal development by forcefully align your options along with the extremes. Not only do most people find themselves somewhere in the middle, but they also move back and forth as their careers progress. People who can turn a technical advantage into a business advantage usually have a blend of engineer/manager in them. You can advance without spending time in management. But if you want to give management a try, you should. Do not let your identity be simply defined by a title. </p><p></p><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://cdn.quotesgram.com/small/6/27/1526246614-duties-226.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="347" data-original-width="226" height="347" src="https://cdn.quotesgram.com/small/6/27/1526246614-duties-226.jpg" width="226" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div></div><p></p>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-84952855827051784342021-05-01T12:50:00.006+07:002021-05-02T08:36:12.503+07:00Lessons from a promotion<p>My tech team is organized into squads - cross-functional teams owning end-to-end feature development. The squad leadership is a joint collaboration of a product owner, a project manager, and a tech lead. As the business expands, more squads are needed and it falls to me to fulfill these new tech lead vacancies. To facilitate professional growth, internal promotion is favored over external hires. Along the way, I learned a few lessons.</p><h4 style="text-align: left;"><span style="background-color: white;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCwCY7Zm0FymYK4vaz-J1l2UXDcKgMDzQA2kyhzMiyyAj2K19zLHuy5gzTizXuCvnCjzv_Z9FGnNhdXt3i-_tdk9JSmboIgqBGdnJlr1pUWUpumwYfy3_94NOr9SekBoRvXn5dNRmhQm2L/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1222" data-original-width="1576" height="310" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCwCY7Zm0FymYK4vaz-J1l2UXDcKgMDzQA2kyhzMiyyAj2K19zLHuy5gzTizXuCvnCjzv_Z9FGnNhdXt3i-_tdk9JSmboIgqBGdnJlr1pUWUpumwYfy3_94NOr9SekBoRvXn5dNRmhQm2L/w400-h310/image.jpeg" width="400" /></a></div></span></h4><h4 style="text-align: left;"><span style="background-color: white;">It starts with a job description</span></h4><p>The one thing that makes or breaks the promotion is the job description. A vertical promotion from junior to senior involves performing harder and grander tasks with tighter deadlines, in other words, being more proficient a what you have already been doing. A promotion to a leadership position is more “horizontal” in that sense. Pretty much like the time you left high-school, I don’t think any amount of prior experience can truly prepare one for what comes next.</p><p>Our tech leads work with people across all principles to provide a cohesive technical vision for the squad, contribute to the product strategy, and coach their members. In that role, many activities are new to them. They will be working with people whose functions they haven’t fully comprehended, like a tech lead with FE background working with DevOps for a deployment plan. They are asked for estimations while given far less details than what they received pre-promotion. They are exposed to HR matters around the well-being of their crew, not all of which make everyone happy. And just sometimes they have the trauma of having their handcrafted solution taken out of context for an entirely different thing and 3 days to deliver. Given the drastic change in scope of work (and the PTSD), it is understandable that post-promotion, some feel like a fish out of water. Unfortunately if there is a structural approach to eliminate this sense of disorientation, I haven’t found it yet.</p><p>While it is tempting to propose a five-page long job description listing out all little details one is supposed to perform and hence solve the challenge once for all, the managerial wet dream is nothing more than a motivational debt. Software development rewards people for their creative prowess and that in turn attracts great problem solvers to the craft. Practically spelling out what one needs to do is the opposite of that. The job description should enable the person to picture the boundaries of her authority and the impact she has on the team without resorting to dictating the specific activities. Everyone will have different responses to “make tactical moves to ensure successful deliveries”, or “look after the career development of team members”, and that’s part of the growth. Take that, Tiger Mom!</p><h4 style="text-align: left;">Strength in diversity</h4><p>In the previous year, the squad model had some initial successes. The first two squads jelled and performed well, relatively uneventfully. Structurally. both were the mirroring image of each other: BE-heavy, big data focus, led by old-timers. So when it came to the next new squad, there was a strong urge to copy the earlier success: same leadership profile, same structure, and same kind of work. That should be easy, the management knows what to do, the promoted people have existing role models to follow, and things probably fall into the right place like they had done before. That was as close to a squad printer as I could think of.</p><p>In reality, my third squad was FE-heavy, had a strong interest in UI/UX topics, and had a product owner stationed away from the main body of the team. It couldn’t be any more different from the former two. I am glad that this happened.</p><p>A parthenogenetic offspring of a squad would have been an easy choice down a slippery slope.</p><p>I didn’t realize at the time, but collectively the technical discussion had already leaned towards the server side of thing more than it should. It is normal that individually each of us turns our face towards what we know and against what we don’t. But it gets dangerous when we all turn towards the same thing, we get ignorant of our faults and prejudices. In OOP, that is known as closed for modification, and <a href="https://en.m.wikipedia.org/wiki/Open%E2%80%93closed_principle" target="_blank">closed for extension</a> too.</p><p>The identical leadership profile would also send the wrong kind of signal, that one has be X and work in Y to get promoted. Everyone with a different profile probably feels unappreciated like a 40-year-old on Snapchat and take their chance elsewhere.</p><p>With the birth of the third squad, I got to learn the importance of a design system, the vast untapped advancement of browser technologies, and the bias in BE-FE collaboration. All these are areas of improvement that wouldn’t have surfaced if we had gone down the easy path and promoted yet another BE engineer. It itches me to sound like a social justice warrior, but we did find new powers in diversity.</p><h4 style="text-align: left;">The support structure</h4><p>No, the new squad was not released to the wild to fend for itself. That would have been bogus.</p><p>In fact, the support structure was the one area received the most attention back in the squad formation. We defined the 3-prong structure where product, technology, and agenda support each other. The right people were hand picked for the backbone and the remaining vacancies received the highest recruitment priority. Meeting plan was laid out so everyone had multiple outlets to discuss their opinions. </p><p>The support structure was least of my concern, till something hit me in the face, something technical yet also... sociological.</p><p>The new squad got its people and work split from the two existing ones, like a cell division. Hence found itself co-contributing a number of code repos with the others. That led to some confusions where its realm of existence started and ended. The team operated with a constant fear of stepping on someone else’s foot. The organizational structure was changed and had not been reflected in the software interface. Wham! It was such a classic case of the Conway’s law that I was awed to observe it first hand, yet hurt for not seeing it coming earlier. The law was one of my favorite engineering observations, right up there with Murphy’s.</p><p>The following rectification was relatively straightforward. We educated people about the boundaries of squads, brought in service contract to strengthen the interface between them, and proceeded to splitting shared services into smaller ones where it made sense.</p><h4 style="text-align: left;">A personalized journey</h4><p>Accompanying the tech leads on their way through aforementioned obstacles was a rewarding experience but easy it was not. There are many questions yet few definite answers, how many tech debts are too many, when an internal tool should be made. Much variation in preference, some are more than happy to deal with abstraction where others are keen on a transparent view. And much uncertainty ahead for that no plan can account, how one accounts for spending a month waiting for an engineer to onboard just to have the guy quit a day before his start.</p><p>It is probably apparent now that I haven’t done enough of this to actually know what I am doing. But I am experienced enough in software development to deal with uncertainties. And I am invested in getting it work for my team.</p><p>The Agile Manifesto has it that</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div style="text-align: left;">Individuals and interactions over processes and tools</div><div style="text-align: left;">Working software over comprehensive documentation</div><div style="text-align: left;">Customer collaboration over contract negotiation</div><div style="text-align: left;">Responding to change over following a plan</div></blockquote><p><span style="text-align: -webkit-center;">They are good guidelines to drive our weekly catch-ups. We keep a very experimental approach at what we are doing, and maintain a close feedback loop. What work are replicated elsewhere, what don’t are studied. But most importantly, I always try to be a thought partner through out this journey. Interactively growing a </span><span style="text-align: -webkit-center;">team where people are collaborative and open about their problems today is more important than having it down to a science with a rigid plan tomorrow.</span></p><p>If one day I have a toddler, I ain’t no need for books.</p><p>Onwards.</p><p><br /></p>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-76886767284499453102020-12-13T08:37:00.053+07:002022-06-27T21:36:36.900+07:002020 - A year unlike any others<p style="text-align: left;"><span style="font-size: small;">—</span></p><p style="text-align: left;"><span style="font-size: x-small;">The post is dedicated to Parcel Perform’s engineering journey to profit through the pandemic - also known affectionately as the longest Tet ever by the locals. Other teams fought hard too and deserved their own stories. Opinions are my own and do not express that of my employer.</span></p><p style="text-align: left;"><span style="font-size: small;">—</span></p><h2 style="text-align: left;">Patient zero</h2><p>It was a day in March that Covid stopped being a passing disease in some corners of the world like SARS, Zika, or Ebola and started to be a reality to me. Vietnam got its first case in January. Sequoia’s letter to its founders and CEOs was making its way around the Internet. And WHO declared Covid a pandemic. </p><p>I am not going to lie, despite the unease of entering a pandemic period, part of me was looking forward to it, fondly. I was ten when the dot-com boom took place. Vietnam was some remote corner of the world, known only for the war. The Internet was a foreign word. Mom brought home rice, canned food, and fish sauce in a basket to prepare for Y2K. Everything I know about the dot-com bubble, its magic and devastation, and how Internet giants emerged from its crumble was told to me. More than just an economic downturn, it was imprinted on me that not just surviving, but striving through a hard time is the ultimate test for me as an engineer, and the system I have built. All were thoughts and stories until that day in March. The old farts could move aside with their bubble. My generation had Covid.</p><p>The management acted swiftly and decisively. Changes were introduced to protect individuals and the company's sustainability. The flexible WFH policy had always been there. Teams were split to come to the office on alternative days. Meetings were moved online. Monthly townhalls were broadcast online with details of a break-even plan by end of the year. With everyone in lockdown, e-commerce activities - our main source of revenue - raised significantly. Vietnam also took aggressive measures to minimize the spread of the virus. The new changes in life both inside and outside of work were new and exciting. For a while, Covid had seemed like a challenge we could face with a sense of hope.</p><p>It was then the grim reality set in. As more customers worked online, our platform became the only way for them to stay on top of their logistics situation. The eyes were on us. The demand for service availability, something previously had not been as desirable as it could be, skyrocketed. In parallel, the virus spread at a speed that made the Mongolian hordes look like amateurs. People started to cite stories about the infamous Spanish Flu. Recovery estimation was changed from months to years. The virus was not the only thing in the air, fear also was. The investment market contracted, dried up, and imploded on its former optimistic self. No one wanted to bet on the uncertainty of what eventually became the biggest threat to humanity in this century (so far). We had to make changes to keep the remaining runway as long as they could be and planned for an unattractive funding round that we did not even know could happen.</p><p>By April, we found ourselves fighting bigger challenges, with a smaller and less effective workforce. Basically a sadistic role-play of the US health care system.</p><h2 style="text-align: left;">Is less more?</h2><p>In software development, there is a sense of elitism of doing more with less (people). Whatsapp was sold with 55 employees, serving billions of messages every single day. Instagram with 13 employees, including the two founders. Markus Frind built Plenty of Fish solo for 6 years. Imagine what else these people could have done if they had had Asian parents! </p><p>That wasn't exactly our story though. </p><p>The break-even plan was broken down into monthly targets. We scored the first month, then got into a dry spell effectively the whole Q2. The B2B sales cycle was notoriously long, we had strong leads that took months to realize. The general anxiety about the future did not make anything better. The off-track plan placed a hiring freeze on us. Meanwhile, the unprecedented quarantine surge traffic and sales effort sent more work our way. We typically won customers by going the extra mile to make custom features and integrations. The work was not hard, but irritatingly time-consuming as most normal customers were not fluent in the programming if-this-then-what riddle.</p><p></p><ul style="text-align: left;"><li>The time on custom work kept us from working on the core system.</li><li>The under-invested core could not handle the surge in traffic, so here and there engineers got pulled out to help fire-fighting.</li><li>The looming committed deadlines were peril dishes for easy implementation whose architectural simplicity was an afterthought. The code became brittle and less welcomed for an extension, making custom work and fire fighting even more time-consuming.</li></ul><p></p><p>It was a vicious circle that could very quickly deplete the team morale and leave everyone burnt out.</p><p>We sook to find the balance between feature work, and core system. We designated 90% of the development time to be split between making new features, adding customizations, and paying tech-debt, and the remaining 10% on whatever the team thought would be a future issue if left unchecked. That went exactly as good as Donald Trump’s plan for the pandemic: utter chaos stems from reality detachment.</p><p>The reality was that we had more on our plate than we could chew. We struggled to keep up with the delivery schedule before, we would not be able to with 10% less. And though the time invested on tech-debt would help us in the long run, investment took time, the time we could not afford. In the deadline frenzy, the 10% budget was a forbidden fruit, and development time was given to whatever made the biggest noise at the time. Sometimes it was the core system because nothing spoke louder than a system outage. But for the rest of the time, it was a customer-first policy. We needed that revenue stream to pull through the hard time. We were the homeless of software development trying to make a saving account.</p><p>Less is more does not come solely from the engineering side of things, it only thrives where it aligns with the whole business as a cohesive unit. Whatsapp, Instagram, and Plenty of Fish are consumer mobile apps that demand very little customization compared to the world of B2B that we are in. SAP has thousands of developers - not Techcrunch headline material - yet denying so is to deny the laws of physics.</p><p>There were some interesting leads that we kept hearing about, but otherwise, Q2 ended on an eerily uneventful note. Little did we know, this was the night before the storm.</p><h2 style="text-align: left;">Growing pains</h2><p>Then came Q3 in a way we could not have expected. The traffic that has not slowed down in previous quarters then gained even more momentum. Malls in the US and EU were shutting down and likely remained so over the holiday season due to concerns about spreading the virus (it indeed happened). People turned their compulsive buying online. Good times if your business is centered around tracking and analyzing e-commerce shipments. There was only one little convenience. The data stream flooded through us with all the force of the mighty Mekong before people built all those dams over her.</p><p>Having invested in a horizontally-scale application layer, our journey to scalability was a walk in the park. Except for trees, the park had carnivorous ents, pedestrian Nazguls, and birds fell beasts. The squirrels were cool, they probably just had rabies. The walkthrough Mordor taught me about growing pains more than all my teenage years combined.</p><div>For months we wrestled with performance issues, always shoved into our hands at the most inconvenient moment. The embodiment of Murphy’s law, solidified by postmortems piled higher and deeper, stem from a series of adequacy:</div><div><ul style="text-align: left;"><li>Lack of imagination. As much as the growth was welcomed, we did not successfully foresee the full impact of such growth on the system.</li><li>Lack of experience and expertise. When incidents happened, we did not know what to do, not immediately, and not fluently executed. Various types of database lock, Kafka data corruption, and Flink zombie jobs all happened for the first time to all of us.</li><li>Lack of infrastructure investment. The list of tech-debt was ever-growing, and the development of internal tooling came too little and too late. </li></ul></div><div>There has always been a tug of war between building a solid system with imaginative traffic vs a house of cards with desirable features waiting to collapse on its own weight. Both compete on finite resources that are time and effort. A seasoned entrepreneur would say the latter is preferable as it indicates we have found a problem worth solving and people are willing to pay for the pain killer. But such knowledge offered little comfort as you were staring at the screen at two in the morning, with the weight of the entire system on the chest, feeling the cold sweeps in the limbs subduing other sensory leaving only numbness, getting angry with yourself and everyone and everything.</div><div><br /></div><div>We had three system meltdowns for the three months of Q3. Each came in a magnitude that threatened the existence of Parcel Perform and made me question the decisions of my life. And when you do that three times in a row, you start to question your sanity too. But there has always been light at the end of a tunnel, no matter how long. With sheer efforts and a lot of hours staring at the screen and self-doubt - because hey we were lack of everything else - we pulled through. I wrote <a href="https://blog.khangnguyen.me/2020/08/ung-bo-cuoc.html" target="_blank">this piece</a> to remind myself the fight is only over when I give up.</div><div><br /></div><div>Performance issues suck really hard. Going through them is painful. And I wish them to never be a part of my life. But the growth which comes with the effort to overcome each and every performance challenge is undeniable. Every time we solve a problem, the system gets stronger and stands higher. <a href="https://blog.khangnguyen.me/2020/04/building-postmortem-culture.html" target="_blank">We acquire knowledge we would not fathom before.</a> Our process is taking steps to mature so we can fight together effectively and reserve the hard-earned knowledge, though we are a long way from the finish. The insurmountable mountains are less intimidating. Much of what we consider valuable in our world arises out of these kinds of challenges because the act of facing overwhelming odds produces greatness and beauty. Such is growth and pain at their worst and best combinations. </div><div><br /></div><div>We ended Q3, bleeding from the mouth, but confident to start hiring again.</div><div><br /></div><h2 style="text-align: left;">The aftermath</h2><div>2020 was a bizarre experience. The world was forced to accept a new reality for better or worse. We were forced to mature. Decisions in the back of our minds that we knew we would get there somehow were put in the first-row seat as the pandemic arrived.</div><div><ul style="text-align: left;"><li>We transformed the functional engineering team into cross-functional squads just days before the country entered lockdown.</li><li>We now have a CS team that collectively covers 24 hours of a day and a stronger presence in the EU while the common situation of the industry is to contract inwards.</li><li>Our SLA went from best-attempt to actual tangible values. A move that increased the excitement of the sales team ten folds, the exact amount it decreased from my engineering team. A striking example of conservation of happiness.</li><li>We ended up overshooting the break-even plan 2x.</li></ul><div>But on the other hand, there is no denying that wherever we look we see improvement needed. We are trying to keep a system that has grown 4x since the beginning of the year stable. New components like pg_bouncer and setups like splitting the Flink cluster into single jobs were irreplaceably crucial. Yet the biggest pain points have shared the same pattern: the application logic we implemented a year ago can no longer handle the traffic we have today. We haven’t fully escaped the hideous Sisyphus circle. Patches of various degrees of permanence were introduced; still usually before we could fully complete the implementation and internalize the knowledge, another thing would happen. The little tech debts we eagerly took are coming back with compound interests.</div><div><br /></div><div>And we are doing all these jugglings in the aftermath of a 6-month hiring freeze. We had had the same set of people working together from the beginning till the end of the pandemic (Vietnam calendar, we have been lucky). We worked intimately with all the services and developed an acute sense of troubleshooting. While that experience was pivotal in maintaining the system, our investment in the onboarding process, documentation, and internal tooling was insidiously slacked off. A person unfamiliar with the system would find what we have been working on in the last 6 months overwhelming. The participation of the new engineers post-hiring freeze was a waking call.</div><div><br /></div><div>An exploding system and a high overhead of adding new members were a stressful combination. At times, we were probably just steps away from the death of a thousand cuts. Despite all that, there is a mutual feeling that the worst Covid has to offer was in the past and thing only gets better from here on. We have a strong team of young people who every day fight above their weight class. The debts we have are solvable with the right amount of time. We have a product that fits the market and has heaps of space to develop. When you manage to grow so much despite the turbulent time of 2020, fewer things can frighten you. We are looking at 2021 with much anticipation.</div></div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7dJjG93Jl6AJSOplf-5pyo7aADhogPWyCMEpHO3X3rCHA1gPpj0cnhyLZY2-_7nrEbqfpB4WcUf8-0KH4lTPXnoFCV3ysdnWKjVgaoWJ3wgIGHy9RlQwpBy-ENjkxHQ6XcZHH8sKlgfYG/s5184/IMG_3189.jpeg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="3456" data-original-width="5184" height="266" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7dJjG93Jl6AJSOplf-5pyo7aADhogPWyCMEpHO3X3rCHA1gPpj0cnhyLZY2-_7nrEbqfpB4WcUf8-0KH4lTPXnoFCV3ysdnWKjVgaoWJ3wgIGHy9RlQwpBy-ENjkxHQ6XcZHH8sKlgfYG/w400-h266/IMG_3189.jpeg" width="400" /></a></div><div><br /></div><div><br /></div>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com1tag:blogger.com,1999:blog-308191087296040302.post-65085155298957834052020-08-15T17:52:00.011+07:002020-11-21T21:22:28.415+07:00Đừng bỏ cuộc<p>Gần hai giờ sáng, phía ngoài văn phòng, một cặp vẫn đang tâm sự. Dù ánh đèn 7-11 hắt ra, mắt vẫn díu lại, chẳng nhìn được rõ mặt. Mấy ngày trước, vài khách hàng lớn bắt đầu sử dụng dịch vụ, lượng thông tin tăng mạnh. Hệ thống như nhà cấp 4 chặn đường bão cấp 8, dột vô số chỗ. Hy vọng đây là đêm cuối. Mọi người đã giải quyết được nhiều vấn đề. Giấc ngủ sẽ trở lại sau những ngày thấp thỏm trông con mọn.</p><p>Tôi làm việc ở một startup đã vài năm. Làm việc ở đây cảm giác như lái tàu hoả từ thời Liên Xô trên đường ray chưa tồn tại. Tàu vừa chạy vừa xây đường, bằng bất cứ gì có được xung quanh. Xây lúc nhanh lúc chậm. Đường lúc lên lúc xuống. Nhưng quan trọng nhất là tàu vẫn phải chạy.</p><p>Thiếu người, bắt đầu sau, và ít tiền, nhưng vẫn làm tốt hơn những công ty cạnh tranh, không có cách khẳng định "tôi giỏi" nào đơn giản mà mạnh mẽ hơn vậy. Những ngày đó, bạn đi trên mây và thế giới là của riêng mình bạn. Và cũng có những ngày như hôm nay, người gồng, đầu cúi, mắt nhìn không qua được mặt bàn. Công việc là một chuỗi dài những sai lầm ngu xuẩn.</p><p>Lúc nhỏ, ba mẹ hay nói lớn lên sẽ làm được cái này cái kia. Đầu lớp một đạp được xe. Lên cấp ba làm được trại 26/3. Vào đại học tự lập. Như thể bên trong có những cái công tắc màu nhiệm, đủ tuổi thì công tắt bật, sẽ hiểu được những hệ thống bự đùng, thấu sự đời, và đạt niết bàn. Theo đúng thứ tự như vậy.</p><p>Có điều, sau hai startups thất bại, vẫn chưa cái công tắc nào được bật. Chỉ có công việc là khó hơn. Nhiều khi sợ hãi, như người bơi xa sợ đuối nước, chỉ muốn quay đầu, mọi áp lực này sẽ biến mất. Không còn những cuộc gọi lúc nửa đêm. Không còn những đêm dài một mình trước màn hình, nghe dưới da nhịp tim tăng dần. Không còn vò đầu bứt tóc, bất lực trước những câu hỏi tại sao. Nhưng làm startup nhiều hạn chế. Không có lưới bảo hiểm. Giờ mà buông bỏ, khó quá không làm, thì sau lưng cũng không còn ai làm cả.</p><p>Từ dòng code đầu tiên, chật vật mới xử lý hết 20k requests trên con máy ảo bé tí, đến giờ mỗi ngày vài "Tê" đi ra đi vào, hệ thống và mọi người xung quanh nó đã dậy thì biết bao nhiêu lần, có cả chết đi sống lại, đều là nhờ không bỏ cuộc mà tìm được lối ra.</p><p>Không có một bí kíp luôn đúng cho các vấn đề của một hệ thống phức tạp. Quan trọng là kiên nhẫn và đừng quá khó khăn với bản thân. Nhìn được chuỗi sai lầm ngu xuẩn là đi được bước đầu tiên rồi. Giải quyết một vấn đề, tàu chạy được một ngày. Giải quyết một vấn đề nữa, chạy thêm một ngày. Rồi vấn đề thứ ba, thứ tư, thứ năm. Đến cuối cùng của chuỗi ngu xuẩn, là đến đích rồi. Hoặc là thế, hoặc là thất bại và có được một bài blog ngon lành trên con đường chống dốt. An toàn hạnh phúc với những dự án bé bé xinh xinh, rồi sao chịu được sóng to gió lớn?</p><p>Có lẽ, đó là cái công tắc cuối cùng, đã được bật từ lâu.</p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgz0CKT_fZXz_IJjQjBP2PzTnDfsMUNB4LBE1M4diZhQURp18QtIPgO90CLPneKowWd5n5OugFaiYiuNH5nzfzVpcrsNDXy6zt1HoXJDpa6EVW5SV5mCLyhKV4eIhqfUMBO6L7dO_8LLGj4/s2048/IMG_20190615_183310.jpg" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1536" data-original-width="2048" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgz0CKT_fZXz_IJjQjBP2PzTnDfsMUNB4LBE1M4diZhQURp18QtIPgO90CLPneKowWd5n5OugFaiYiuNH5nzfzVpcrsNDXy6zt1HoXJDpa6EVW5SV5mCLyhKV4eIhqfUMBO6L7dO_8LLGj4/w640-h480/IMG_20190615_183310.jpg" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><br /></td></tr></tbody></table><p></p><blockquote><blockquote><blockquote><p style="text-align: center;">Sleep is for the weak.</p><p style="text-align: center;">I am weak.</p></blockquote></blockquote></blockquote><p><br /></p><p>P/s: Sau khi lên nháp ý tưởng bài blog này, hệ thống của tôi bị sập mất Kafka - lần đầu sau gần 5 năm. Tốn thêm bốn tiếng căng thẳng mới giải quyết được vấn đề. Một minh chứng về việc một hệ thống IT chỉ tồn tại giữa những lần bị sập, và không có lần sập cuối cùng.</p><p style="text-align: center;"></p>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com4tag:blogger.com,1999:blog-308191087296040302.post-63713299379147452032020-04-25T22:24:00.001+07:002020-12-09T17:10:26.298+07:00Building a postmortem culture<blockquote class="tr_bq">
<h2>
<span style="font-size: large;">post·mor·tem</span></h2>
<i><span style="color: #073763;">\ ˌpōs(t)-ˈmȯr-təm \</span></i> </blockquote>
<blockquote class="tr_bq">
noun</blockquote>
<blockquote class="tr_bq">
1. Autopsy</blockquote>
<blockquote class="tr_bq">
<span style="color: #666666;"> A postmortem showed that the man had been poisoned.</span></blockquote>
<blockquote class="tr_bq">
2. An analysis or discussion of an event after it is over</blockquote>
<blockquote>
<span style="color: #666666;"> The blameful postmortem culture shuts down the exploration of the problem because no one wants to be seen as stupid, even if it's ignoring the clear truth.</span></blockquote>
<br />
A postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring. The postmortem concept is well known in the technology industry.<br />
<br />
I picked up the concept of postmortem from my previous job at Silicon Straits Saigon. The idea that we could study an incident was there, but the guidelines and culture enforcement was weak. So though I was sold that postmortem was a powerful practice and with proper enforcements made a system become more robust over time, I didn't exactly know how to start a culture around it. The most concrete guidelines I received was from <a href="https://landing.google.com/sre/books/">Site Reliability Engineering</a>. Wherever the Google practices seemed too extreme or impractical in my context, there was the Internet. The knowledge was powerful and enlightening, and I appreciated the journey in the last 6 months to transform it into operational HOWTOs.<br />
<br />
From the very beginning, I was aware that a postmortem culture needed to be a joined effort of the entire organization for it to be effective. And I was never interested in being a secretary. But like many other initiatives that involve other people, you can't just make an announcement and expect things to happen, magically. I tried. A few times. So in the beginning it was just me recording the incidents that I was a part of either the solution or the problem. Most of the time both. And that gave me the time and experience I needed to make calibrations to the plan before it was presented to everyone.<br />
<br />
<h3>
Work to take blame out of the process</h3>
<div>
Blame, both the act of blaming and the fear of being blamed is the enemy of a productive postmortem culture. If a culture of finger-pointing and shaming individuals or teams for doing the "wrong" thing prevails, people will not bring issues to light for fear of punishment, or stop investigations prematurely as soon as a "culprit" is identified. Such halts the development of preventive methods for the same situation in the future. The force to blame is formidable, we as human beings are wired for it. Dr. Brené Brown, in a <a href="https://www.ted.com/talks/brene_brown_the_power_of_vulnerability?language=en">TED talk</a>, explained blame existence as "a way to discharge pain and discomfort". The fact that whenever you want to trace back whose code caused your miserable wake up at two in the morning for a system outage, the command says `git blame` certainly doesn't help.</div>
<div>
<br /></div>
<div>
This is where being blameless gets popular in postmortem literature. And if there is anything subjective to an objective piece of work that is a postmortem, this is it. I find being completely blameless hard to implement. On the one hand, a postmortem is simply not a place to vent frustration. On the other hand, at times, it feels like tip-toeing around people, so worried about triggering their fragile souls that you miss out on chances to call out where and how services can be improved. This is where I come to an agreement with <a href="https://techbeacon.com/app-dev-testing/blameless-postmortems-dont-work-heres-what-does">J. Paul Reed</a> that it is important to acknowledge the human tendency to blame, allow a productive form of its expression, and yet constantly refocus to go beyond it.</div>
<div>
<br /></div>
<div>
Here are some examples. The examples might or might not involve me in might or might not actual scenario.</div>
<div>
<br /></div>
<div>
<div>
<b>Blameful:</b></div>
<blockquote class="tr_bq">
Someone pushed bad code to production via emergency pipeline. The tests in CICD could have caught this, but someone thought he knew better. Seriously, if you aren't sure what you are doing, you shouldn't act so recklessly. Rolling back in the middle of the night is a waste of time.</blockquote>
<blockquote class="tr_bq">
Action items:<br />
<ul>
<li>Think before you edit someone's code.</li>
</ul>
</blockquote>
<div>
<br /></div>
<div>
<b>Completely blameless:</b></div>
<blockquote class="tr_bq">
Last night, an unauthorized code was pushed to production. CICD was skipped because CICD takes 30min and it was fire fighting situation. The fix was not compatible with a recent refactor. </blockquote>
<blockquote class="tr_bq">
Action items:<br />
<ul>
<li>Improve CICD speed</li>
</ul>
</blockquote>
<div>
<br /></div>
<div>
<b>Blame-aware:</b></div>
<blockquote class="tr_bq">
Last night, an unauthorized code was pushed to production. CICD was skipped because CICD takes 30min and it was fire fighting situation. The fix was not compatible with a recent refactor. </blockquote>
<blockquote class="tr_bq">
Action items:<br />
<ul>
<li>Improve CICD speed</li>
<li>Infrequent contributors should use the safety net of CICD</li>
<li>Issues in a service need escalating to maintainer of the repo for code review</li>
<li>Rollback mechanism needs to be available to developers on pager duty.</li>
</ul>
</blockquote>
</div>
<div>
<div>
I felt like in the examples above, without accepting that pushing code to an unfamiliar service in the middle of the night was a reckless action, we would miss the chance to put in preventive measures. But again it is subjective, perhaps my blame-aware version fits perfectly into a blameless version of another. I hope you get the point.<br />
<br /></div>
<h3>
Work on some guidelines</h3>
<div>
It is useful to be as specific as possible about when a postmortem is expected, who should write it, what should be written, and what the goals of the record are. Not only it provides a level of consistency across your organization, but it also prevents the task of writing a postmortem to be seen as a whimsical assignment from some higher-level authority and someone is being picked on as a punishment for doing the "wrong" thing.</div>
<div>
<br /></div>
<div>
Some of my personal notes of the matter:</div>
<div>
<ul>
<li>Different teams might have different sets of postmortem triggers. The more critical your function is, the more detailed the triggers should be.</li>
<li>People who caused the incident might or might not be the ones to write the postmortem. The choice should be based on the level of contribution the person has to offer, both in terms of context and knowledge, not because of his previous actions.</li>
<li>Be patient. The people you are working with are professionals in software development, but the ability to write good software does not transmit into the ability to write a good document. Quality of root cause analysis prevails eloquence. Save the latter for your blog.</li>
</ul>
<div>
<br /></div>
</div>
<h3>
</h3>
<h3>
Work on the impact to your audience</h3>
<div>
in the beginning, the incidents I was working on were about a database migration to Aurora (and a hasty fallback), so I was assuming my audience would be my fellow developers. Possibly extend to project managers, project managers love knowing why you are stealing time from their team. And a reasonable consequence was to write the postmortem in markdown and store them in the same code repo with the affected services. There were a few issues with that.</div>
<div>
<br /></div>
<div>
Firstly, in a technology startup, the scope of tech choice is always bigger than "just the tech team". In my case, Customer Success people need to know what the impacts on our customers were and are, Product people want to know if the choice comes with new possibilities, and Sale people want to sell those possibilities. As much as I love such integration between the developers and the rest of the company, the idea of granting universal access to code repos to view markdown postmortems terrifies my boss, and therefore subsequently me, obviously.</div>
</div>
<div>
<br /></div>
<div>
<div>
Secondly, as familiar as markdown is, it is not a very productive option if you want to include media in it. And we want charts of various system metrics during the time of the incident to be included in the postmortem.</div>
<div>
<br /></div>
<div>
Lastly, writing a postmortem is gradually becoming a collaboration effort, and git repo, though supports collaboration, does not do it in real-time.</div>
<div>
<br /></div>
<div>
Considering all the options, we finally settled with a shared Google Drive in the company account. It is neither techie nor fancy. But it allows very flexible accessibility, tracks versions, natively supports embedded media, and lets multiple people collaborate in real-time. We share our postmortems in a company-wide channel, and sometimes hold an additional presentation for particularly interesting ones.<br />
<br /></div>
<h3>
Let it grow</h3>
<div>
When you have done your homework, built a foundation of trust and safety, laid out the guidelines and constantly improved it, and integrated the postmortems with your larger audience, it is probably time to take a step back and let the culture take a spin on its own. My company's postmortem culture won't be the same as Google's no matter how many Google books do I read. And as long as it works for us, it doesn't matter.</div>
<div>
<br /></div>
<div>
With some <i>gentle</i> nudges, my colleagues are picking up postmortem on their own. We have seen contributions from Product Owners and Project Managers, besides the traditional developers and DevOps contributors. The findings are anticipated by a large audience across the company. </div>
<div>
<br /></div>
<div>
And in the latest incident, which involved the degradation of performance in a few key features of our SaaS offering over the course of a week, we identified another usage of a postmortem: a postmortem updated regularly with the latest incident reports, findings, and potential impacts in a near real-time fashion is a powerful communication tool across the organization, both ensure the flow of information to people who need it (CS to answer questions, PM to change project plans for urgent hotfixes, etc) and allow developers to focus on their critical work without frequent interruptions.</div>
</div>
<div>
<br /></div>
<div>
As we grow and our system gets more sophisticated, hopefully, the constructive postmortem culture would turn out to be a solid building block.</div>
Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-29410238828190822292020-01-16T11:57:00.001+07:002021-01-08T19:17:15.435+07:00Run, Forrest, Run!Tl;dr: I ran my first marathon, and whined about it. Move on.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8nBIatr0jPhMYGXKMNB0enHwcb0DP30yqST81PNnukqbekOc6KFWwbz2JXOu0R2IwQYqtUVzmU1T5EqwbRN-1CtI0pcO4d09qDdGLig0SH4xbPXwc8DECbjKFQt2UQPBaAGQeB524SQ6O/s1600/IMG_20200105_035729.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="756" data-original-width="1600" height="302" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8nBIatr0jPhMYGXKMNB0enHwcb0DP30yqST81PNnukqbekOc6KFWwbz2JXOu0R2IwQYqtUVzmU1T5EqwbRN-1CtI0pcO4d09qDdGLig0SH4xbPXwc8DECbjKFQt2UQPBaAGQeB524SQ6O/s640/IMG_20200105_035729.jpg" width="640" /></a></div>
<br />
4 years after finishing <a href="https://blog.khangnguyen.me/2016/01/run-like-you-stole-it.html">my first half marathon</a>, I finally did my first full marathon, 42k of sweat and pain. 2019 was horrible for me, through all ups and downs, the marathon plan is one of a few that keep me together. The cut off time was 7 hours. I wanted to do a sub-5 (complete the run under 5 hours) but ended up with a sub-6. I was squarely in the bottom quarter of my age group. So it wasn't all glory and stuff, but I am so glad I did it.<br />
<br />
I must have started the training back in March or something, and didn't follow the training plan through and through, obviously. I got sick, which paused the plan by a week every time it happened. I got injuries that eventually put me out of action for a whole month. And when I was back, following the original training plan just gave me too much stress and guilt, which I certainly didn't need - my life was really low, so I forwent it and just ran whatever the fuck I wanted. That was probably 2 months ago.<br />
<br />
The injuries were actually a blessing in disguise. They forced me to rethink my running form. I picked up a book on running (that is not Murakami's autobiography) and tried to avoid "common sense" misconceptions, like most notably, landing on your whole foot. I finished 42k without any injuries. Yay!<br />
<br />
The day I got the bib, it came with a shock. I was put into the 30-39 age group. Technically, it is not my birthday yet. And despite all the talks, I was not mentally prepared for this. Ouch! Oh and I also got interviewed.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgib8O1xYMNzNvXnHSxqB0QZzKvS6sO44TC6Vnrco11BHHI20hdXvRhAGKOmeEZSJnKRaqovNkN8gUq6K7kh5T3GWaSDs8kCbpJALYgN-GRymMRoZf73-ppK1KOvwnwbX5T47vVDlg9zbM2/s1600/IMG-20200116-WA0000.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1280" data-original-width="960" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgib8O1xYMNzNvXnHSxqB0QZzKvS6sO44TC6Vnrco11BHHI20hdXvRhAGKOmeEZSJnKRaqovNkN8gUq6K7kh5T3GWaSDs8kCbpJALYgN-GRymMRoZf73-ppK1KOvwnwbX5T47vVDlg9zbM2/s400/IMG-20200116-WA0000.jpg" width="300" /></a></div>
<br />
I had never run the full distance prior to the run and in retrospect, wasn't a great idea. I now believe that the body would prepare for an extra few km on top of the maximum distance you have covered but not by a long shot. And it makes sense, why would my body be ready for 100k if I have never run 50k? The longest I had done was 30k and that explained why from 34k I got cramps so bad.<br />
<br />
It was also the first run that I got proper sleep the night before. And I wasn't hungry. I sure stuffed myself with loads of carb, so full that on the night before the run, I thought it was stupid, I couldn't possibly run with such a stomach. but above all, shout out to the organizers, the route had more than sufficient water, electrolyte, and banana.<br />
<br />
One last thing, the Nike app has improved a lot between then and now. It is no longer off by 30% and comes with cooler features. Well done Nike.<br />
<br />
---<br />
<br />
If that hasn't bored you out of your skull yet, you might want to see how my run broke down. "How did you remember all of this?" - I knew I gonna write one of these post-event, so it wasn't that hard. And I made up all the bit I didn't remember, including that I ran at all. Bwahaha.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEjDGTh3XFjjOKJRSKo_CYpmBoHcuTgfQmpyQd2cRKdEU9hNPeU2v-FZloGqg4mj3cE8lKpe2KpUXIKykWZO14Fe3jz5gMII2S6v5KaCG3UaRj01f9hiqs5hlG1ooB7x2YVqbhIQHMbmew/s1600/Screenshot_20200113-222149.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="758" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEjDGTh3XFjjOKJRSKo_CYpmBoHcuTgfQmpyQd2cRKdEU9hNPeU2v-FZloGqg4mj3cE8lKpe2KpUXIKykWZO14Fe3jz5gMII2S6v5KaCG3UaRj01f9hiqs5hlG1ooB7x2YVqbhIQHMbmew/s640/Screenshot_20200113-222149.jpg" width="302" /></a></div>
<br />
<br />
<b>Starting line:</b> That's right, 42k is the first wave, the first-class citizen of a marathon. With all the volunteers standing around and looking, the limelight feels good. Wait, hang on. It's already 4. Why aren't we starting? Technical issue? Great, I am trying to get some work-life balance and here I am, with bugs.<br />
<br />
<b>0km:</b> 10 minutes in, here we go guys!!! Let me just start my run on the Nike mobile app. Fuck fuck fuck. I dropped an energy bar while shoveling the phone back to the running belt. Screw it, I am not fighting against a wave of runners for a stupid bar. What a start.<br />
<br />
<b>1km:</b> An old man with a Vietnam flag on his back is making a crude joke that a bunch of fit men, leaving their horny wives and young children home, to run on the street at 4 in the morning must all have mental issues. It could have been a good joke, it could have. But why did you have to be so fucking disgusting in your choice of words old man? Urgg why are you even carrying our flag?<br />
<br />
<b>2km:</b> Some already making pit stops at the trees by the sides of the road. Shit looking at them gives me the urge too. Nah. If I sweat enough, the exceeding liquid will just be repurposed in time. Probably. The 42k 4:45 pacers are here, but they seem slow (1) and have loud music on. Better keep some distance.<br />
<br />
<b>3km:</b> Here it is, the first major water station. Thanks to the Starting Line Incident, I am down to 4 bars now. I should have more bananas. Double portion, please! There are Waldo, Doraemon, and Ao Dai right in front of me. Cute, but I am not falling behind casual cosplayers. Onwards!<br />
<br />
<b>5km:</b> We are joined by a group of 21k, they seem to have a shorter route. I no longer hear the music of the 4:45 pacers. I also don't want to have my pace mess up by 21k runners. Time to speed up a bit.<br />
<br />
<b>7km:</b> Just gulped down the first energy bar. Entering the beast - Phu My bridge. Still have a vivid memory of how it wore me out in my first 21k. Some 21k runners keep passing me. Well, at least they aren't 42k.<br />
<br />
<b>9km:</b> The easier quarter of the bridge was easy. Neat, there is a water station before the hardest quarter. Go in for a shower. Feel so good. Kimochi!!!<br />
<br />
<b>10km:</b> Wow that's the highest point of the ascending half already? That was quick. I'm feeling great. The training works!<br />
<br />
<b>12km:</b> "Coming through!" I didn't yell but it was certainly loud as I ran past a few runners. I'm sprinting! Not supposed to put stress on my feet no? But I am on a runner's high. Gotta take advantage of this slope then.<br />
<br />
<b>14km:</b> Keeping up a good speed. Ketchup guy wait for me! Well, he is a 42k runner in costume which, for the lack of visual detail, only makes me think of a bottle of ketchup. I might be running too fast. There is no downhill gravity to play with. Slowing down.<br />
<br />
<b>15km:</b> Crossroad. Am I turning or keeping straight? Oh, there is a volunteer, neat. I asked you twice for the direction and the best you can fathom is "Huh?". You, sir, are truly an idiot.<br />
<br />
<b>16km: </b>"That fires we don't put out, will bigger burn". And that's exactly why I am standing here right next to a tree, minding my own business. Here comes the same water station on the 3rd KM. Banana! I am joined by a bunch of 21ks. This group is with pacers of 2:20. Guess I'm not doing too badly myself (2). But they are loud. I am putting in some distance.<br />
<br />
<b>18km:</b> This is proceeding nicely. I'm bored. Time for some music. After all, what is the point of having the pinnacle of technology in my belt? And lost a bloody energy bar.<br />
<br />
<b>20km:</b> I am rejoined by the 2:20 pacers. This time the topic is on the color of the underwear the pacers are wearing. I should now add that the pacers in this group are women. "You're wearing nothing!" Someone screamed top of his lungs. Look like he is having a really good time. No, he isn't carrying a Vietnam flag. I looked. I'd love to add some distance again, but I am getting slower.<br />
<br />
<b>21km:</b> Canada International School eh? Funny. I'd be here again later this afternoon to watch a game of Saigon Heat. This is a massive waste of energy. Doraemon is behind me. I'm not running behind a cosplayer, not a blue fat cat with comically short legs (and balls for hands). Just a bit faster. Entering the differentiator turn, this is the part of the route that 21ks don't join. This stretch of the route seems to last forever (3).<br />
<br />
<b>25km:</b> The sun is already high. I can't possibly head to a tree this time, can I? (4) Embracing myself for a stinky toilet. Wow, it's actually clean. This is awesome. The toilet, not me pissing.<br />
<br />
<b>27km:</b> Doraemon is behind me again, but I can't possibly run any faster than this. I tried. Ran ahead of him for tens of meters and I would fall back to a normal pace and he would pass. Not just Doraemon though. I am losing count of how many have passed me.<br />
<br />
<b>28km:</b> Good morning milady, can you help me with some of that muscle spray please, on both legs? Wow, that was refreshing! Thank you very much.<br />
<br />
<b>29km:</b> God damn, some of that spray got onto my crotch. My balls are freezing. I sure hope they don't fall off.<br />
<br />
<b>30km:</b> Got tension on the thighs. I got this. I got this. I trained for this. The app announces I have 12km left. Took me a while to calculate that I have run for 30km. Math is super hard.<br />
<br />
<b>32km:</b> I have never run this far in one go. From here on, it's uncharted territory. Squats, I need to do a few squats, it stretches my thighs a bit so they are functional again.<br />
<br />
<b>33km: </b>The tensions have turned into cramps. Squat. Run for a couple of hundred meters. Cramp. Squat. Rinse and repeat. It hurts so much.<br />
<br />
<b>34km:</b> Arrggg I fought, but I can't run anymore. My thighs got cramps. My ankles hurt from all this stomping. And the soles of my feet too, for pretty much the same reason. Worst of all, my brain seems to go blank, this is stupid, what I am even doing. I have to walk now.<br />
<br />
<b>35km:</b> I have run a few short dashes, a couple of hundred meters each. One of the attempts locked my legs, almost landed my face on asphalt. The cramps are still going strong. Someone just handed me a big chunk of ice. It freezes my hands. Dude, what I am supposed to do with this? My balls are gone and that's bad enough. Here, tree, your daily ration in a solid form.<br />
<br />
<b>36km:</b> Here is the plan, I gonna run between one crossroad to the next, then walk till the next crossroad. I still get cramps whenever the 2 crossroads a bit farther apart, but at least my mind has come back. The sunlight is roasting me. I miss you, sunshine.<br />
<br />
<b>41km:</b> Fuck no! My legs gave up on me. Completely. I get cramps just from walking. No amount of squat seems to help. I can hear the crowd from here, so fucking close.<br />
<br />
<b>42km: </b>My legs are at the stage where any excessive movement would give me cramps. The last 200 meters, the finish line is finally here. Here goes nothing. The legs don't seem to be mine, I move them like two sticks. I run. I cross the line. I get a high five. A girl put this medal around my neck. Heck, I can' even recall what she looks like. But she was wearing a Bà Ba, that was a nice touch. Under normal conditions, I would have appreciated the outfit, but right now I am having a strong urge to vomit my guts out.<br />
<br />
---<br />
(1) This is probably the first sign that I didn't manage my energy level well. Too cocky. But again, I aimed for sub-5, so...<br />
(2) I conveniently forgot the fact that 42k started 30 min in advance. But we also had a longer route since the beginning. All else being equal, I was running the first half between 2:10-2:20.<br />
(3) It didn't. It lasted for 3.5km. Running on a familiar route made me feel like it was shorter.<br />
(4) I talked to a friend about this. Pro-tip is to just pee on yourself. In a race, you probably consume enough water that your pee is transparent anyway. My shorts were white, so it didn't help much with the level of confidence. Best to do this at a water station where they usually put a big bucket of water for a quick shower.Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-44374614045070232662019-10-26T21:10:00.001+07:002019-10-26T22:19:05.483+07:00What you want to do in your lifeQuite some time ago, I wrote "<a href="https://blog.khangnguyen.me/2015/04/good-grades-good-opportunities.html"><span id="goog_156575249"></span>Good grades, good opportunities<span id="goog_156575250"></span></a>". Basically what I meant was that working hard in school is one way to propel in life, but it isn't necessarily the best way. By focusing too much on a part, one risks missing the whole spectrum of possibilities and opportunities, like a sad puppy looking at a rainbow. Yet supposed you have graduated and what you want to do with your life is anything but a fogged window, how would you get yourself into a position with better visibility?<br>
<div>
<br></div>
<div>
My generation was fed that we should be following our passion. That implies somewhere in your mind there is a hardwired plan waiting to unfold, and that you know what you like - more so than any others. Well, if that was the case, we wouldn't be sitting here, would we? The people who said that also didn't really mean it. They probably meant that though the journey to discover what to do in your life is more banging your head against a brick wall than smooth sailing, don't let it demoralized you. I agree. You shouldn't underestimate your potential, and succumb to the thought you can't do what other people can.</div>
<div>
<br></div>
<div>
The first step towards a way out is to ask whether you must have a job. Hint: if you can afford the question, you probably don't. Besides providing a means to get by, being in a job is also a useful experience. It is certain that each job is unique, but given a bit of abstraction, they have a fair share of similarity. For example, most engineering jobs are project-based and focus on time management, communication, and problem solving. These skills are interchangeable regardless of industry. Through the job, you also get exposed to an industry, where opportunities barely visible from outsiders. No, the world doesn't have a conspiracy against people like you and me, nor some people have it easier than others. It just changes fast and because innovation rarely follows a pattern, being in the right space is crucial.</div>
<div>
<br></div>
<div>
Remember this is your day job. Working at a day job doesn't mean doing it half-heartedly. It means you shouldn't let your identity be defined by this job, just as an aspiring writer with a day job as a cram school teacher doesn't think of himself as a teacher. I usually tell fresh grads that who they are in the next 5 years is <a href="https://blog.khangnguyen.me/2016/12/dont-work-after-7pm.html" target="_blank">defined by what they do after 7PM</a>. Because that is when the day job ends and real work begins. If you are one of the fortunate individuals who can afford not having a job, you get to do real work full time.</div>
<div>
<br></div>
<div>
What real work? To discover what you like. This is actually harder than what it sounds like. There are a lot of jobs, ones that you don't know and ones whose name hasn't existed yet. And the dramatic way the media has described professionals only makes an accurate image harder to find. Most lawyers go through their careers without a single murder case. Literally no one hacks into a system by typing in a terminal like a maniac. And my favorite series House MD is about the exact opposite what doctors do.<br>
<br>
I am probably the worst one to whom you can talk about the definition of "like". But I know that working on a job where you only execute instructions is not going to be very fulfilling. It is better to work in areas where you are interested to understand how things work, can exercise your freedom of thought, and get into the zone easily (where time flies). In other words, work on stuff about which you are curious. You don't need a full time job in an industry to experience it. Better yet, don't even bother with the concept of industry, it puts boundaries to your thoughts. Just pick a project that seems interesting: volunteer at a professional event, research the answer to a hard question, build a bottle rocket on your roof top. Choose a project that only takes a few weeks. Make sure it is something you can finish without any blocker. It should be a bit challenging, but not so hard that you feel overwhelmed. Online courses and books are both very helpful in this journey, though you should avoid acquiring too much data without a project to put it to use. Rinse and repeat with another project, and another, and yet another.<br>
<br>
These self-directed activities can be a bit overwhelming at first, especially if you have got used to receiving assignments from school. But you will be able to adapt and get better. It is a part of being human, and being young.You need some initial discipline to get started, but once you do, given the project is an object of your interesting, curiosity will turn work into play and discipline is no longer needed.<br>
<br>
If you continue down this path, you probably find something that stands out from the rest, enough that you want to come back for more. If you are lucky, you would find more than one. Depends on how good you are at multi-tasking, you might need to choose. Choose the option that leads to more options afterwards. There is a paradox of choice, but advancement in life is measured in possibilities. You go from one stage of education to another because the higher stage give you more options. There are simply more things a guy with a college degree can do compared to one graduated from high school. Building a software product gives your more options than doing business analysis. Once you have built a product, you can get into business analysis later if you prefer the narrow focus. But analyzing business is not going to make you a product manager.</div>
<div>
<br></div>
<div>
Paulo Coelho's renowned The Alchemist had it that when you really want something, omens is the way the universe points you to the right direction. I have picked up many things in my life, and given up most of them. The ones that stick are usually ones I experienced some sense of beginner's luck. Perhaps it is my omen. Yours might be the same. Or it could be the urge to do something you weren't allowed to do in your younger years. Or the thrill of an audacious plan even people who have known you for years couldn't think you would pull it off (actually, the more people know you, the more likely they are to subject to stereotypical thinking). </div><div><br></div><div>You should believe you have it within, that you will sense the click when the omen happens. Now go give you some more exposure of what the world has to offer.</div>
Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-59140461315201474542019-08-31T10:54:00.000+07:002019-08-31T10:55:52.666+07:00Postgres does not use index on gigantic numeric valueNothing beats a Saturday morning when you wake up fresh and excited, ready for the second sleep, and realize that your Postgres database was harassed over the night and accumulated a number of downtime minutes that is too embarrassing to state here.<br />
<br />
My database load looked like this during the outage.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyyx4I8zkOg301xZDyflE3u9ZPsp5dnK52YLsbhb1qKjgAJRmARu3x-OM1iGpAQ4L_mVg2cJ5rCuH4Lsfwr1Kn9umW1_P0_DJT_Bk0-KdCdLnmJvgtEYlbkBkSuT0QzhoII1WOTYokosCz/s1600/Screen+Shot+2019-08-31+at+10.30.36+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="279" data-original-width="825" height="216" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyyx4I8zkOg301xZDyflE3u9ZPsp5dnK52YLsbhb1qKjgAJRmARu3x-OM1iGpAQ4L_mVg2cJ5rCuH4Lsfwr1Kn9umW1_P0_DJT_Bk0-KdCdLnmJvgtEYlbkBkSuT0QzhoII1WOTYokosCz/s640/Screen+Shot+2019-08-31+at+10.30.36+AM.png" width="640" /></a></div>
<br />
<span style="font-family: "courier new" , "courier" , monospace;">buffer_mapping</span> looks new. Postgres official documentation on this matter seems to be written by Captain Obvious: <span style="background-color: white; color: #336791; font-family: "open sans" , sans-serif; font-size: 14.4px; font-weight: 900;">Waiting to associate a data block with a buffer in the buffer pool.</span> Thanks for nothing. Basically <span style="font-family: "courier new" , "courier" , monospace;">buffer_mapping</span> is a lightweight lock in read operations, my processes were fighting to reserve buffers in which to read data pages.<br />
<br />
I have a read problem.<br />
<br />
This query accommodates the highest number of locks:<br />
<br />
<pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; color: #1d1c1d; font-family: Monaco, Menlo, Consolas, "Courier New", monospace !important; font-size: 12px; font-variant-ligatures: none; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; white-space: pre-wrap; word-break: normal;">pp_pqsql_prod=> explain select * FROM "big_table" WHERE "big_table"."id" = 9200190224054041915721;
<wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> QUERY PLAN
------------------------------------------------------------------------------------------------
Gather <wbr style="box-sizing: inherit;"></wbr>(cost=1000.00..4568847.85 rows=535675 width=1673)
<wbr style="box-sizing: inherit;"></wbr> Workers Planned: 2
<wbr style="box-sizing: inherit;"></wbr> -> <wbr style="box-sizing: inherit;"></wbr>Parallel Seq Scan on big_table <wbr style="box-sizing: inherit;"></wbr>(cost=0.00..4514280.35 rows=223198 width=1673)
<wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> Filter: ((id)::numeric = '9200190224054041915721'::numeric)
(4 rows)</pre>
<br />
It is a sequence scan, on a 100M-row table, so it is obvious why it caused all the ruckus. What's less obviously is why Postgres performed a sequence scan on a primary key column.<br />
<br />
With the help of a colleague, it appeared that given a smaller numeric value, the index kicks in just fine.<br />
<br />
<pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; color: #1d1c1d; font-family: Monaco, Menlo, Consolas, "Courier New", monospace !important; font-size: 12px; font-variant-ligatures: none; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; white-space: pre-wrap; word-break: normal;">pp_pqsql_prod=> explain select * FROM "big_table" WHERE "big_table"."id" = 9200190;
<wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr> <wbr style="box-sizing: inherit;"></wbr>QUERY PLAN
-------------------------------------------------------------------------------------------------------
Index Scan using big_table_pkey on big_table <wbr style="box-sizing: inherit;"></wbr>(cost=0.57..8.59 rows=1 width=1673)
<wbr style="box-sizing: inherit;"></wbr> Index Cond: (id = 9200190)
(2 rows)</pre>
<br />
The problem is the size of the queried value. Eventually I stumbled upon this <a href="https://dba.stackexchange.com/questions/146294/why-does-postgresql-perform-a-seq-scan-when-comparing-a-numeric-value-with-a-big" target="_blank">stackexchange Q&A</a>. It can be seen in the first <span style="font-family: "courier new" , "courier" , monospace;">explain</span> that because 9200190224054041915721 was too big, it had to be casted into <span style="font-family: "courier new" , "courier" , monospace;">numeric</span> data type. My primary key was not that big, its data type was <span style="font-family: "courier new" , "courier" , monospace;">bigint</span>. So it had to be casted too, because apple can't be compared to orange. What I have now is a <span style="font-family: "courier new" , "courier" , monospace;">numeric</span> to <span style="font-family: "courier new" , "courier" , monospace;">numeric</span> comparison, and a <span style="font-family: "courier new" , "courier" , monospace;">bigint</span> index can't serve that.<br />
<br />
Problem be gone and so was my morning.Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-89568188406253723132019-08-21T22:52:00.000+07:002019-08-21T23:05:19.707+07:00On organizing Barcamp <div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIecKMQQGv6FmYAMsurr7y2c0PiUIpPzNct74FT4Hc8OD1G2t3YTRd64l0MBCr7jc9nwa63TFntonkJ5nKWrZDUv18N9ndhhDmPHHM8mQy8WJQHisTzAKjkKPHze88gDhqN9cDI-VR34Ns/s1600/67362257_10157642123082754_224630070935814144_o.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1600" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIecKMQQGv6FmYAMsurr7y2c0PiUIpPzNct74FT4Hc8OD1G2t3YTRd64l0MBCr7jc9nwa63TFntonkJ5nKWrZDUv18N9ndhhDmPHHM8mQy8WJQHisTzAKjkKPHze88gDhqN9cDI-VR34Ns/s200/67362257_10157642123082754_224630070935814144_o.png" width="200" /></a></div>
If you haven't heard of Barcamp Saigon, then putting it shortly, Barcamp is an open space, also known as an unconference. As organizers, we prepare the facilities to make presentations happen, but both the content and agenda are determined by participants on the event day. Topics are submitted online a couple of months before the event to somewhat ease the hassle. But the voting is strictly limited to the event day, to ensure only the opinions of those who come matter. Topics are then allocated to rooms and time slots accordingly to their popularity. It's that simple.<br />
<br />
I have been involving in the Barcamp scene of Vietnam for a few years. There was a 2-year hiatus when it was a make-it-or-break-it period for my employment. Yet eventually, when the dust settled, I found myself, once more, organizing one.<br />
<br />
There are more or less a thousand participants in each Barcamp Saigon event. And though we typically only hold one a year, it is still a time sucker, with some hard work, and being a volunteer community project, one pretty much plays the hand that's dealt. Organizing Barcamp is one of the things in my life I consider unusual. Contacting seemingly random people asking for money, dealing with finance and paperwork, and coordinating other people, for an introvert, sound dreadful. Making and answering phone call push my wrong buttons.<br />
<br />
Yet I hold on to organizing Barcamp Saigon like my dear life depends on it. I guess because as a twenty-something, I understand how hard it is to get heard in this exhilarating hyperactive city. You kinda have this neat little idea in your mind, but can't seem to find a thought partner to bounce it back and forth, nor an audience to whom the thought is relevant. Heck, even getting invited into a conference (as a speaker) is hard. You are nobody, and the conference you need doesn't exist yet. After a while, you just lose the excitement, the idea is now collecting dust in some closet, in a far corner of your clustered mind. An open space like Barcamp provides all people with an idea to share a platform to do exactly that. You don't need invitation, and the voting board of Barcamp merely shows the topic name, everyone is nobody, and content is king. In a way, an open space is the serverless of knowledge exchange. You get right into the exciting bits.<br />
<br />
Barcamp audience is more diverse than it is specialized, more of a melting pot than a silo. If you want to argue the better approach to natural language processing, go to an AI conference; get started with conversational UI (aka chatbot), an UX conference. If you want to make the reservation experience at your restaurant more fluid, well, I guess you are out of luck, there isn't such a space in Saigon yet. But Barcamp is probably the only place you can find people with these three expertise mingling. It reminds me fondly of university time. There was these general elective courses where I got to pick non-IT ones. In those courses, everyone knew something I didn't, and the other way around. And the clash of knowledge, mindsets, and life styles was refreshing. As time goes by, my "tribe" gets smaller and more focused, usually on topics I am very much familiar with. The magic moment when something you have known since forever is basically witchcraft to me get fewer and farther in between. So I learned to appreciate Barcamp for pulling me out of my cave.<br />
<br />
To be at Barcamp, in the middle of all communities is, for lack of a better word, a happening feeling. I get to see communities as living creatures, breathing, morphing into shape and form, and reincarnating. That doesn't necessarily mean a good thing. I have been to some Barcamps, at different places. Some were mind blowing. Some sucked though. One was particularly bad that I decided I wanted to do something about it. In a way, Barcamp is a health indicator of its local communities. But unlike GDP, which indicates wealth (1), AQI for pollution, or HPI for happiness (2), this indicator is one where individual actions can make huge difference. I am doing my part.<br />
<br />
Open space concept doesn't have to be limited to Barcamp. In fact it shouldn't be. It should be another option in every community builder's toolkit for growing their group of like-minded people. It reduces the stress to find speakers for the next meetup, and give new participants chances to shine. And by the time open space is adopted as a second nature, perhaps Barcamp can retired for it has accomplished what it means to :)<br />
<br />
---<br />
<br />
(1) Whether GDP can indicate economy growth is a controversial topic that I am not jumping into.<br />
(2) In Zen And The Art Of Motorcycle Maintenance, Pirsig argued that quality cannot be defined. When one defines characteristics of quality, those characteristics cease to be quality and can't describe the whole, which is greater than the sum of its parts. Is happiness quality?<br />
<br />
<br />
<br />Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com2tag:blogger.com,1999:blog-308191087296040302.post-79988623405879457872019-06-30T01:17:00.000+07:002019-07-05T18:53:17.075+07:00Random thoughts on a job fair<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="margin-left: 1em; margin-right: 1em;">
</div>
<br />
<br />
Random thoughts from a tech expo that I attended just now, not in any particular order. And some thoughts might have nothing to do with a tech expo, I am not apologizing.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://s3.amazonaws.com/media.eremedia.com/wp-content/uploads/2018/04/18080614/careerfair-700x467.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="Image result for job fair" border="0" height="266" src="https://s3.amazonaws.com/media.eremedia.com/wp-content/uploads/2018/04/18080614/careerfair-700x467.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">src: tlnt.com</td></tr>
</tbody></table>
<br />
<ul>
<li>Today I learned that there is career fair, and then there is job fair, each with a <a href="https://smartaxe.wordpress.com/2014/01/12/a-career-fair-vs-a-job-fair/" target="_blank">slight distinction</a> from the other. The one I went to was probably more of the latter. But really, it was mostly about mascot costumes and freebies. So just.. fair?</li>
<li>If you are a company, and spend anywhere between some and shit lot of money for a booth, make sure your representative can represent something.</li>
<ul>
<li>Recruiters should know about their company business nature, organization structure, and for what their job titles actually stand. The use of buzz words like partnership, innovation, or opportunities is bad communication and can't be used to replace solid understanding on business model, job description, or career path respectively.</li>
<li>Developers coming to support recruitment are great helps. But they should know that while being there, they are effectively salesmen. If they can't articulate, with a hint of passion, what their job is like in a day, a month, and a quarter, they are just wasting oxygen and space.</li>
<li>If you think that's a lot to ask for, you are right. It should be. Hiring is the most impactful task one can make to his organization. Bring your highest caliber executives in, because otherwise TinyPulse is bringing their Director of Technology and awe your potential hires with insightful thoughts and sincerity.</li>
</ul>
<li>If you are a recruiter, you should have a plan to deal with people who have zero interest in your vacancies. The very best candidates are seldom between jobs, they are more likely to be employed and well-paid, you need to poach them to stand any chance. They might not be hooked in the positions, but you can still get them interested in the company. Getting one's contact info doesn't mean shit, people are just polite. Being accountable for answering queries on business model, offering career development path, or simply representing your culture are all significantly more important. And as far as I can tell, nobody is keeping tabs. Be different.</li>
<li>If you are an attendee, don't get annoyed if people shove flyers into your hands. They most likely didn't mean any harm. They are just trying to do a good job within their capacity, given the little or zero training they received before hand. Be kind, take flyers, and enjoy a small conversation. You might learn something new while making someone's day.</li>
<li>If you are looking for a job, reading between the lines of job titles and their descriptions is a great way to learn about a company's modus operandi, tech stack, and drawbacks. For example:</li>
<ul>
<li>If one is hiring multiple data analyst positions, it has a data-driven culture.</li>
<li>Tech stack is revealed through job description and requirement is self-explanatory, but also pay attention to what is not written. Overwhelming mentions of batch processing indicates a lack of stream processing.</li>
<li>A company's drawbacks might get reflected in job requirements precisely because issues happen and the company is working on resolving them with fresh <strike>meat</strike> minds.</li>
</ul>
<li>Similarly, if you want to follow a particular career path, however vague, look at a company's vacancies whose jobs rely on yours. The number of vacancies shows how mature the company is in your chosen field and how much support you gonna receive. For example, big data engineers can look for vacancies of analytics and ML; front-end developers, product and design; product managers, any sign of customer interaction. There should be a healthy combination of both business and technical vacancies for a good network effect. Nothing is worse than spending your time one something with zero impact on anyone.</li>
<li>People in outsourcing industry should get panic if they aren't working on hard problems. That you can code the same CRUD operation in 5 different languages is a waste of resource rather than something to be proud of. Local powerhouses like Ahamove or Tiki are solving hard problems that impact the life of millions of people. And it doesn't matter if they can do it in only one language. They are doing it.</li>
</ul>
<div>
<br /></div>
<div>
PS: apparently, I wasn't the only one who thought JDs is a great medium to learn about a company. Paul Graham used job listings to learn and scale his potential competitors too. <a href="http://www.paulgraham.com/avg.html">http://www.paulgraham.com/avg.html</a></div>
Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-50579400000630534242019-03-31T23:17:00.001+07:002019-06-20T11:42:45.865+07:00Do young developers have it easier?<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="margin-left: 1em; margin-right: 1em;">
<img alt="Image result for dilbert intern" height="199" src="https://i.pinimg.com/originals/b2/1a/92/b21a921f9625719a3e536736c334f8c2.gif" width="640" /></div>
<br />
<br />
The world of computer programming is changing, fast. In the short 10 years of my career, I have had chances to witness industry standards emerged, demolished, and resurrected. Serverless, the latest abstraction upon abstractions that mankind call programming, has had its glorious days and is now probably on the peak of Gartner Hype Cycle. The time I spent on XML and its abomination of a sibling XSLT is one I would never get back, though the problems have not necessarily gone away. And everyone is deep learning left, right and center, after decades of hiatus.<br />
<br />
As we get better at reinventing programming paradigm, streamlining development process, and specializing careers, new crops of developers come into the work force, with perspectives drastically different from their previous generations. But does growing up with an iPhone in their pockets necessarily mean these developers have it easier?<br />
<br />
Like youngsters of any species, young developers, despite their non-existential social skills and blissful ignorance, are known for being fast, energetic, and tenacious. And I think this is where a lot of misperceptions come from. Where I give credit to young developers is their stamina. Programming is neither easy nor relaxing. It is a demanding job requiring intense concentration. Young people have the energy, especially when combined with pizza and coffee, to stick to her computers at 2 in the morning, working on intermittent quirks of an API she released earlier, and yet be delight for the new knowledge and that her work matters. Such work is harder when one gets older. He has a life he needs to keep up with. Actually, he is probably pissed that the API doesn't just work the way it should be.<br />
<br />
That, however, does not mean young developers get shit done faster. They really don't, otherwise you would see me preparing my retire plan really soon. If anything, they tend to screw thing up faster than The Flash on red bull. I have had almost every interns accidentally drop a database, or delete something they shouldn't have, in their 3-month time. Neither do they learn faster. I am in the camp of believing as one makes progress in his career, he learns faster. Programming knowledge is cumulative. "New" ideas usually have popped up some time before in a slightly different form, in some other language or situation. Young developers, outside of school work, do not spend time tinkering assembly, distributed model, or the 7 layers of the Internet. They rip the benefits of other developers who worked out the details and packaged them nicely so that they are more approachable by others.<br />
<br />
Does the fact that young developers rely on more layers of abstraction a bad thing? Of course not. I am glad that I can just sit here writing my post in plain English and not HTML nor CSS. That's exactly how we progress as a species, relying on abstractions built by generations before. In that direction, current mature technology favors quick iterations and shorter time to market. But wait, isn't that good old Agile, what's the big deal? Agile is about building one thing that is small yet works well, then a little more. Yet in an ecosystem with 600k apps in Play Store alone, builders aren't even sure if the one thing they are building gonna be a hit, or miss, till it is out. There are so many high quality reusable building blocks: authentication, real-time database, infinitely scalable database, etc. Writing a new software is now less about constructing and more about figuring out the right combination of gluing things together. The rise of function as a service means young developers can just build it. Work? No? Blow it up. Do it again. All before lunch starts. Older developers have been burned, they don't want to get burned again, they dread the concept of redoing till figuring out.<br />
<br />
Though blessed with a beginner's mind, I think young developers still have a long way to go. I don't tend to find old developers that often. Part because IT is a new profession in where I am living. Part because many by their middle age move on to management level, or move out. The guys who remain hand-on throughout their career is really a force to be reckoned with. They are matured as a developer and have learned the art of leadership. They delegate appropriate work to junior people, balance between net contribution and learning something new, and thus saving both theirs and project time. People who have been around a while also see the bigger picture more clearly, both in terms of product scope and project life cycle, which problem can be delayed, and which shouldn't.<br />
<br />
Then comes scaling. A new application gaining traction, at some point, would hit the scalability wall. Hard. Like Miley Cyrus on a wrecking ball hard. Scaling is a lot harder than throwing together a bunch of code one did not write *cough* StackOverflow *cough*, and duct-taping till it works. A scalable system takes real dedication, expertise, and a calling to learn more and better ways to be a great developer. And these don't come over night.<br />
<br />
Young developers are intelligent, hard working, and excellent at solving the problems they are given.<br />
Old developers are intelligent, lazy, and excellent at predicting problems in 6 months time and building the right foundation today to make it easy when time comes.Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-8608519307729378242019-02-22T12:37:00.005+07:002019-02-22T17:43:24.198+07:00Simple example on why LIMIT is not always the best thing for SQL DatabaseSince our production database grew to the size where querying data without indices became a suicide mission, we have learned that LIMIT does not always translate into faster query time and less work on the machine. Today at work, we ran into a beautifully simple example to illustrate this point, which was perceived as counter-intuitive by more junior teammates.<br />
<br />
Here we have a query over a 8M-row table, the condition fits squarely into an index.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">EXPLAIN SELECT "record".*</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">FROM "record" WHERE ("record"."status" = 'not_found' AND "record"."updated_date" < '2018-12-23 15:48:23.008320+00:00');</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> QUERY PLAN</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">------------------------------------------------------------------------------------------------------------------------------------------</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Bitmap Heap Scan on record (cost=256890.59..3475874.26 rows=8448978 width=1740)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Recheck Cond: (((status)::text = 'not_found'::text) AND (updated_date < '2018-12-23 15:48:23.00832+00'::timestamp with time zone))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -> <b>Bitmap Index Scan</b> on <b>not_found_records_idx</b> (cost=0.00..254778.35 rows=8448978 width=0)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> <b>Index Cond</b>: (((status)::text = 'not_found'::text) AND (updated_date < '2018-12-23 15:48:23.00832+00'::timestamp with time zone))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">(4 rows)</span><br />
<br />
<br />
That was still lot of records, so we were hoping a LIMIT clause would shorten the execution time.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">EXPLAIN SELECT "record".*</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">FROM "record" WHERE ("record"."status" = 'not_found' AND "record"."updated_date" < '2018-12-23 15:48:23.008320+00:00') <b>LIMIT 1</b>;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> QUERY PLAN</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">--------------------------------------------------------------------------------------------------------------------------------------</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Limit (cost=0.00..0.46 rows=1 width=1740)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -> <b>Seq Scan</b> on record (cost=0.00..3923250.74 rows=8448978 width=1740)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Filter: ((updated_date < '2018-12-23 15:48:23.00832+00'::timestamp with time zone) AND ((status)::text = 'not_found'::text))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">(3 rows)</span><br />
<br />
<br />
Only that it didn't. Adding the LIMIT clause changed the query plan from Bitmap Index Scan to a Sequence Scan, which is about the worst thing we want to do on a 8M-row table. It is a typical "optimization" of the planner, who thought that the query with LIMIT clause was too small.<br />
<br />
Operation wise, an index scan is more expensive than a sequence scan. An index scan would require to read the index pages first and then read the data pages for relevant rows. The sequence scan only deal with data pages. And so when the LIMIT is small, combined with the table statistics, the planner is tempted to believe there are enough (random) rows that match the filter condition, which makes a sequence scan cheaper.<br />
<br />
In fact, somehow this table statistics suggests that every LIMIT is too small till it is half of the table!<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">EXPLAIN SELECT "record".*</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">FROM "record" WHERE ("record"."status" = 'not_found' AND "record"."updated_date" < '2018-12-23 15:48:23.008320+00:00') <b>LIMIT 3000000</b>;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> QUERY PLAN</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">--------------------------------------------------------------------------------------------------------------------------------------</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Limit (cost=0.00..1393038.57 rows=3000000 width=1740)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -> <b>Seq Scan</b> on record (cost=0.00..3923250.74 rows=8448978 width=1740)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Filter: ((updated_date < '2018-12-23 15:48:23.00832+00'::timestamp with time zone) AND ((status)::text = 'not_found'::text))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">(3 rows)</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">EXPLAIN SELECT "record".*</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">FROM "record" WHERE ("record"."status" = 'not_found' AND "record"."updated_date" < '2018-12-23 15:48:23.008320+00:00') <b>LIMIT 4000000</b>;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> QUERY PLAN</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">------------------------------------------------------------------------------------------------------------------------------------------------</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Limit (cost=257234.59..1781198.16 rows=4000000 width=1740)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -> Bitmap Heap Scan on record (cost=257234.59..3476218.26 rows=8448978 width=1740)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Recheck Cond: (((status)::text = 'not_found'::text) AND (updated_date < '2018-12-23 15:48:23.00832+00'::timestamp with time zone))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -> <b>Bitmap Index Scan</b> on <b>not_found_records_idx</b> (cost=0.00..255122.35 rows=8448978 width=0)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> <b>Index Cond</b>: (((status)::text = 'not_found'::text) AND (updated_date < '2018-12-23 15:48:23.00832+00'::timestamp with time zone))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">(5 rows)</span><br />
<br />
<br />
Adding an ORDER BY clause into the query brings structure to the search and guide the planner to use the index<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">EXPLAIN SELECT "record".*</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">FROM "record" WHERE ("record"."status" = 'not_found' AND "record"."updated_date" < '2018-12-23 15:48:23.008320+00:00') <b>ORDER BY "record"."imported_date" LIMIT 1</b>;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> QUERY PLAN</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">------------------------------------------------------------------------------------------------------------------------------------------------------</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Limit (cost=3518127.15..3518127.15 rows=1 width=1740)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -> Sort (cost=3518127.15..3539249.59 rows=8448978 width=1740)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Sort Key: imported_date</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -> Bitmap Heap Scan on record (cost=256898.59..3475882.26 rows=8448978 width=1740)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Recheck Cond: (((status)::text = 'not_found'::text) AND (updated_date < '2018-12-23 15:48:23.00832+00'::timestamp with time zone))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -> <b>Bitmap Index Scan</b> on <b>not_found_records_idx</b> (cost=0.00..254786.35 rows=8448978 width=0)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> <b>Index Cond</b>: (((status)::text = 'not_found'::text) AND (updated_date < '2018-12-23 15:48:23.00832+00'::timestamp with time zone))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">(7 rows)</span><br />
<br />
Actually, any random ORDER BY would do<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">EXPLAIN SELECT "record".*</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">FROM "record" WHERE ("record"."status" = 'not_found' AND "record"."updated_date" < '2018-12-23 15:48:23.008320+00:00') <b>ORDER BY random() LIMIT 1</b>;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> QUERY PLAN</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">------------------------------------------------------------------------------------------------------------------------------------------------------</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Limit (cost=3539253.59..3539253.60 rows=1 width=1748)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -> Sort (cost=3539253.59..3560376.04 rows=8448978 width=1748)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Sort Key: (random())</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -> Bitmap Heap Scan on record (cost=256902.59..3497008.70 rows=8448978 width=1748)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Recheck Cond: (((status)::text = 'not_found'::text) AND (updated_date < '2018-12-23 15:48:23.00832+00'::timestamp with time zone))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -> <b>Bitmap Index Scan</b> on <b>not_found_records_idx</b> (cost=0.00..254790.35 rows=8448978 width=0)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> <b>Index Cond</b>: (((status)::text = 'not_found'::text) AND (updated_date < '2018-12-23 15:48:23.00832+00'::timestamp with time zone))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">(7 rows)</span><br />
<br />
<br />
However, that is still far from optimal. In order to return the first row, the database first needs to fetch all rows matching the index condition, and only after that, sorts. This creates strains on the machine memory and maybe even disk swap.<br />
<br />
Index is actually ordered structure (BTree). The most optimal query is the one using one of the indexed columns for sorting.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">EXPLAIN SELECT "record".*</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">FROM "record" WHERE ("record"."status" = 'not_found' AND "record"."updated_date" < '2018-12-23 15:48:23.008320+00:00') <b>ORDER BY "record"."updated_date" LIMIT 1</b>;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> QUERY PLAN</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">------------------------------------------------------------------------------------------------------------------------------------------</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Limit (cost=0.56..1.90 rows=1 width=1740)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -> Index Scan using <b>not_found_records_idx</b> on record (cost=0.56..11280384.47 rows=8448978 width=1740)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> <b>Index Cond</b>: (((status)::text = 'not_found'::text) AND (updated_date < '2018-12-23 15:48:23.00832+00'::timestamp with time zone))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">(3 rows)</span><br />
<br />
This, in fact, is even more effective that the original no-order query, with or without LIMIT.Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com3tag:blogger.com,1999:blog-308191087296040302.post-32015717178076350262019-02-01T23:58:00.001+07:002019-02-02T07:44:59.154+07:002018 - Shit am I getting old?Roughly 4 years ago I put together <a href="https://blog.khangnguyen.me/2015/02/the-transforming-2014.html" target="_blank">The Transforming 2014</a>. In retrospective, it was a great year. It felt exactly what youth should feel like, feel like I can afford to fail and bounce back, that opportunities are there to take, and that I am the master of my life. 2018 feels very different. The year followed a trajectory I have carved out over the last few years. It is not particularly eventful, the changes are there, but not exactly tangible. For the lack of eloquence, I would put it similar to when one cries, all the sorrows and memories and grudges build up inside, but till the instant everything bursts out in the form of tear drops, the entire thing is internal and invisible. If a tree falls in a forest and no one is around to hear it, does it make a sound?<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBbP8crZhLbVJt68qZC4JNOCGqrTBmiUjT8-dJKtvNcrMC33jvDo-2Ib1H-OkUt287_IzXb-sol1fo8Yr476o-lwHo71Ra3FVSKOlovGPiMyOVrIMLj5ozMpQXwMAW28W9ayZSOTc3yQX-/s1600/IMG_0563.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1067" data-original-width="1600" height="266" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBbP8crZhLbVJt68qZC4JNOCGqrTBmiUjT8-dJKtvNcrMC33jvDo-2Ib1H-OkUt287_IzXb-sol1fo8Yr476o-lwHo71Ra3FVSKOlovGPiMyOVrIMLj5ozMpQXwMAW28W9ayZSOTc3yQX-/s400/IMG_0563.jpg" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<h3>
Size and Duration matter</h3>
Before I got accused for having lewd thought, this is a note about Parcel Perform. That's what I do. I am a software engineer here. For the fourth year. I have never worked on a single project that long, and I definitely have been staying in Vietnam longer than my intention.<br />
<br />
We build a product around the belief that logistics optimization is the next frontier of e-commerce war. And in that business, size matters. Specifically, dataset size, which grows by dozens of GB every week. At that speed, what worked a month ago, wouldn't work for the next. The job description is as pure as it can get, find a way to process incoming data most efficiently before the size squashes you like an insignificant insect in tropical storm, all while adding increasingly advanced features to deliver values. Or to build a train from the first material you find, all while keeping it running on a century-old rails, hoping you would make it to the next station, if you prefer to speak metaphors.<br />
<br />
I can no longer just add a column with a default value to my database, because that would update hundreds of millions of rows and grind everything to a halt in the process. Everything I thought I knew about software is put into tests. At times, I feel like an imposter, a fraud. I feel alive every time anything breaks. Which is about every day.<br />
<br />
This is also my third time being employee #0 and building a tech team from the ground up. I have a knack doing this. I know which minimal unit of work provides most impact. And I know what developers need to level up their career. But I hope I won't have to repeat the same thing in my next job. The first year of every startup is to build glorified excel sheets. The first engineer's day is always a constant juggle between development, test, and expectation management. It gets old.<br />
<br />
I have hung around with Parcel Perform long enough to pass that stage. We get to work on interesting challenges unique to our business. Our tech stack moulded around us. We have a support network so that the weight of the entire system does not have to rest on a single shoulder. It takes time for things to come together, and so, duration matters.<br />
<br />
<h3>
Creativity Winter</h3>
As much as I enjoy seeing my technical skills grow in the last couple of years, the same can't be said about my creativity outside of work.<br />
<ul>
<li>Between 2016 and 2018, I wrote 13 posts. I wrote 19 posts in 2015 alone. </li>
<li>At some points, my photo app only shows content on white board I took post-meeting.</li>
<li>The last Barcamp in Saigon was in 2016.</li>
</ul>
<div>
None of these was an act of consciousness. Like, I never decided I didn't need this in my life any more. I noticed that my most creative period was when my life happens between airports and on MRT rides. Somehow the constant scenery change kept my mind alert and fresh. Though I successfully prevented my career from repeating itself, the life outside of work got stuck in an <a href="https://en.wikipedia.org/wiki/Multi-armed_bandit" target="_blank">explore and exploit paradox</a>. On the one hand, there must still be stuff out there to give me an adrenaline rush. On the other hand, comfort feels good. I long for a kick start to get out of this stage of limbo.<br />
<br /></div>
<div>
Not all hope is lost. I am writing this, albeit sluggish. A kid at work got into photography recently, perhaps I can get a partner in crime. And Barcamp is back in 2019. (Surprised? Me too!)<br />
<br />
<h3>
Pretty sure we are the only company throwing morning kayak sessions in the city</h3>
I was in a HackerX event in 2018. Basically speed dating for developer employment. And that was literally how I started my part of flirting duty. Getting a tandem kayak was pretty much an impulsive purchase. But it was a great one. I wondered why I didn't think of that earlier. In Vietnam, I get to pretty much drag the boat and pop it in any body of water I feel like. That my office is a few hundreds meters away from a pier also helps. The same can't be said for any other countries I have lived, due to either regularity or location. Be it a morning exercise before work, or a more adventurous weekend, it's good fun and brings friends together. One at a time. There are only two seats on a tandem kayak, damn it.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEje6ZqFR-AIM8KgPklV8m23LPGhpjYMUZG9kN2lkY6vzXkoJQS-9CEdJ7DTcp4dafjPEgHS1gtseB-Mm7lIJ186R8dZYaPOJc-WW6SPINJdJWMTwK2v-gLJSizwTgXZ5_GBum19DTK5BkBf/s1600/Untitled+collage.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1002" data-original-width="1600" height="250" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEje6ZqFR-AIM8KgPklV8m23LPGhpjYMUZG9kN2lkY6vzXkoJQS-9CEdJ7DTcp4dafjPEgHS1gtseB-Mm7lIJ186R8dZYaPOJc-WW6SPINJdJWMTwK2v-gLJSizwTgXZ5_GBum19DTK5BkBf/s400/Untitled+collage.jpg" width="400" /></a></div>
<br />
I wish 2018 was a grandiose. Instead, I just got lot of doubts. I had thing figured out, or my life just got boring? The progress I made in my career was solid, or I was making excuse and succumbing to familiar comfort? I've got the right balance, or work was slowly gulping my life? I still identify myself more as a 20 year-old kid scared shitless in his first job, rather than a 30 year-old I would be when this year ends. Well, I guess that's it right? Till this year ends, I have another year to live in my 20s and do stupid stuff. 30s uncertainty can wait, its time will come later :)</div>
Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-82961718320663776992019-01-01T01:34:00.000+07:002019-01-21T00:07:41.747+07:00Parcel Perform - Another (hack)day on timelapse<div style="text-align: center;">
<div style="text-align: left;">
It was another good year at Parcel Perform. I hope this video turn out to be a better version than last year's CCTV on soundtrack. Great to see the side-by-side <a href="https://blog.khangnguyen.me/2017/12/parcel-perform-day-on-timelapse.html" target="_blank">comparison</a> of how things changed in a year anyway.<br />
<br />
(Yes, I wore the same T-shirt, that was on purpose. No, I don't own only 1 T-shirt in my wardrobe) </div>
</div>
<br />
<div style="text-align: center;">
<iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/ECW3CbpVaeU" width="560"></iframe>
</div>
Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0tag:blogger.com,1999:blog-308191087296040302.post-16072864866402658492018-10-07T21:48:00.010+07:002021-11-12T15:11:39.140+07:00Safe Operations For High Volume PostgreSQL<h4 style="text-align: left;"><b>Add a new column (safe)</b></h4>
In general, adding a new column is a safe operation that doesn't lock the table. However, there are cases where certain options of the command lock the table. These cases are quite common when working on an evolving product.<div><br />
<h4 style="text-align: left;"><b>Add a new column with a non-nullable constraint / default value (unsafe)</b></h4>
This requires each row of the table to be updated so the default column value can be stored. The entire table is locked during this operation, depending on the size of the table, it can take a while and everything else comes to a halt. The solution is to add the new nullable column without a default value, backfill the desired value in batches, then add the non-nullable constraint/default value with an alter statement.<br />
<br />One big good news is that from PostgreSQL 11, adding a new column with default value will be painless, as it should have been :)<br />
<h4 style="text-align: left;"><b>Add a default value to an existing column. (safe)</b></h4><div>If the column is nullable previously, the default value won't alter those existing null rows. If the column is non-nullable, all rows are guaranteed to store some value. Either way, the operation does not block the table and is safe to execute.</div><div>
<br />
<h4 style="text-align: left;"><b>Change the type of a column. (unsafe)</b></h4></div><div>Strictly speaking, this operation locks the table and is therefore unsafe. If the underlining datatype is not changed, like increasing the length of varchar, then the table isn't locked. But if the change requires a rewrite/re-cast, each row of the table is to be updated and the table is locked for the same reason as adding a new column with a default value. The solution is to add a new column, backfill the new column, and get rid of the old one.</div><div>
<br />
<h4 style="text-align: left;">
Add a Foreign Key (unsafe)</h4>
To protect data integrity, an AccessExclusiveLock is placed on <b>both tables</b>, grinds every read and write to a halt (PostgreSQL 9.6+ reportedly allows read). The solution is to take advantage of Postgres' ability to introduce invalid constraints. The procedure is to first create an invalid Foreign Key constraint by specifying <span style="font-family: "courier new" , "courier" , monospace;">NOT VALID</span> in the <span style="font-family: "courier new" , "courier" , monospace;">ALTER TABLE</span> statement, and then validate it in a separate <span style="font-family: "courier new" , "courier" , monospace;">VALIDATE CONSTRAINT</span> statement. The second validation only requires a <span style="font-family: "courier new" , "courier" , monospace;">RowShareLock</span> that doesn't block reads nor writes. Do note that if there is a reference to non-existing rows, the operation won't be completed and you have to take care of integrity on your own.<br />
<br />
<h4 style="text-align: left;"><b>Add an index (unsafe)</b></h4>
By nature, the table being indexed is locked against writes and the entire index is built in a single scan of the table. Read transactions can still be done meanwhile.<br />
<br />
For a production environment, it is always better to build the index without locking writes. <span style="font-family: "courier new" , "courier" , monospace;">CREATE INDEX</span> comes with the <span style="font-family: "courier new" , "courier" , monospace;">CONCURRENTLY</span> option for this purpose. <b>Though building index is still extra work for the database and this is reflected on extra CPU and I/O load.</b> This might still slow other operations, we notice an increased queue depth when we add index concurrently on one of the biggest tables in the system. Because our infrastructure is in the cloud, with a minimal budget, we can upsize the database a couple of sizes larger than normal. The extra power makes adding indices (still with the <span style="font-family: "courier new" , "courier" , monospace;">CONCURRENTLY</span> option) a lot more comfortable and productive.<br />
<br />
<h4 style="text-align: left;"><b>Add a column with a unique constraint (unsafe)</b></h4>
This operation will lock the table because it requires a scan for uniqueness. The solution is to add the column, add a unique index concurrently, and then add the constraint onto the table (<span style="font-family: "courier new" , "courier" , monospace;">ADD CONSTRAINT UNIQUE USING INDEX</span>).<br />
<br />
<h4 style="text-align: left;"><b>Drop a column / a constraint (safe)</b></h4>
This is safe and quick. If the operation appears to take time, the cause is more likely because the table/index was in use rather than the operation time itself. A quick check on <span style="font-family: "courier new" , "courier" , monospace;">pg_stat_activity</span> should confirm that.<br />
<br />
<h4 style="text-align: left;">
Rename an entity (safe)</h4>
The entity can be a table (or an index, sequence, or view) or a column in a table. This operation has no effect on stored data and therefore also isn't affected by the size of the data.<div><br /></div><h4 style="text-align: left;">Turn on MultiAZ (unsafe)</h4><p style="text-align: left;">This is particular to AWS RDS. AWS RDS MultiAZ setup is different from a typical read replica in nature, one is synchronous replication, the other is asynchronous. As such, naively turning on MultiAZ on a (high volume) write-intensive DB will lead to every single write operation being synchronously replicated to the standby replica. It would severely impact the performance of the master because initially the data on the slave machine would not satisfy many integrity constraints and result in either locks or scans on the master for missing data.</p><p style="text-align: left;">It is recommended to take advantage of the asynchronous nature of read replication to get around:</p><p style="text-align: left;"></p><p></p><ol style="text-align: left;"><li>Create a Multi-AZ Read Replica of your Single AZ DB Instance and enable automated backups.</li><li>Once the read replica is synchronized, promote the read replica to be a standalone DB Instance.</li><li>Redirect your application to the newly promoted read replica.</li><li>Rename the source and the new instance to preserve the endpoint (optional).</li></ol><p></p><p></p></div></div>Khang Nguyenhttp://www.blogger.com/profile/06938982405397153584noreply@blogger.com0