Saturday, April 25, 2020

Building a postmortem culture


\ ˌpōs(t)-ˈmȯr-təm \ 
1. Autopsy
    A postmortem showed that the man had been poisoned.
2. An analysis or discussion of an event after it is over
    The blameful postmortem culture shuts down the exploration of the problem because no one wants to be seen as stupid, even if it's ignoring the clear truth.

A postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring. The postmortem concept is well known in the technology industry.

I picked up the concept of postmortem from my previous job at Silicon Straits Saigon. The idea that we could study an incident was there, but the guidelines and culture enforcement were weak. So though I was sold that postmortem was a powerful practice and with proper enforcements made a system become more robust over time, I didn't exactly know how to start a culture around it. The most concrete guidelines I received was from Site Reliability Engineering. Wherever the Google practices seemed too extreme or impractical in my context, there was the Internet. The knowledge was powerful and enlightening, and I appreciated the journey in the last 6 months to transform it into operational HOWTOs.

Since the very beginning, I was aware that a postmortem culture needed to be a joined effort of the entire organization for it to be effective. And I was never interested in being a secretary. But like many other initiatives that involve other people, you can't just make an announcement and expect things to happen, magically. I tried. A few times. So in the beginning it was just me recording the incidents that I was a part of either the solution, or the problem. Most of the time both. And that gave me the time and experience I needed to make calibrations to the plan before it was presented to everyone.

Work to take blame out of the process

Blame, both the act of blaming and the fear of being blamed, is the enemy of a productive postmortem culture. If a culture of finger pointing and shaming individuals or teams for doing the "wrong" thing prevails, people will not bring issues to light for fear of punishment, or stop investigations prematurely as soon as a "culprit" is identified. Such halts the development of preventive methods for the same situation in the future. The force to blame is formidable, we as human beings are wired for it. Dr. Brené Brown, in a TED talk, explained blame existence as "a way to discharge pain and discomfort". The fact that whenever you want to trace back whose code caused your miserable wake up at two in the morning for a system outage, the command says `git blame` certainly doesn't help.

This is where being blameless gets popular in postmortem literature. And if there is anything subjective to an objective piece of work that is a postmortem, this is it. I find being completely blameless hard to implement. On the one hand, a postmortem is simply not a place to vent frustration. On the other hand, at times, it feels like tip toeing around people, so worried about triggering their fragile souls that you miss out chances to call out where and how services can be improved. This is where I come to agreement with J. Paul Reed that it is important to acknowledge the human tendency to blame, allow a productive form of its expression, and yet constantly refocus to go beyond it.

Here are some examples. The examples might or might not involve me in a might or might not actual scenario.

Someone pushed bad code to production via emergency pipeline. The tests in CICD could have caught this, but someone thought he knew better. Seriously, if you aren't sure what you are doing, you shouldn't act so recklessly. Rolling back in the middle of the night is a waste of time.
Action items:
  • Think before you edit someone's code.

Completely blameless:
Last night, an unauthorized code was pushed to production. CICD was skipped because CICD takes 30min and it was fire fighting situation. The fix was not compatible with a recent refactor. 
Action items:
  • Improve CICD speed

Last night, an unauthorized code was pushed to production. CICD was skipped because CICD takes 30min and it was fire fighting situation. The fix was not compatible with a recent refactor. 
Action items:
  • Improve CICD speed
  • Infrequent contributors should use the safety net of CICD
  • Issues in a service need escalating to maintainer of the repo for code review
  • Rollback mechanism needs to be available to developers on pager duty.
I felt like in the examples above, without accepting that pushing code to an unfamiliar service in the middle of the night was a reckless action, we would miss the chance to put in preventive measures. But again it is subjective, perhaps my blame-aware version fits perfectly into a blameless version of another. I hope you get the point.

Work on some guidelines

It is useful to be as specific as possible about when a postmortem is expected, who should write it, what should be written, and what the goals of the record are. Not only it provides a level of consistency across your organization, it also prevents the task of writing a postmortem to be seen as a whimsical assignment from some higher level authority and someone is being picked on as a punishment for doing the "wrong" thing.

Some of my personal notes of the matter:
  • Different teams might have different sets of postmortem triggers. The more critical your function is, the more detailed the triggers should be.
  • People who caused the incident might or might not be ones to write the postmortem. The choice should be based on the level of contribution the person has to offer, both in term of context and knowledge, not because of his previous actions.
  • Be patient. The people you are working with are professional in software development, but the ability to write a good software does not transmit into the ability to write a good document. Quality of root cause analysis prevails eloquence. Save the latter for your blog.

Work on the impact to your audience

in the beginning, the incidents I was working on were about a database migration to Aurora (and a hasty fallback), so I was assuming my audience would be my fellow developers. Possibly extend to project managers, project managers love knowing why you are stealing time from their team. And a reasonable consequence was to write the postmortem in markdown and store them in the same code repo with the affected services. There were a few issues with that.

Firstly, in a technology startup, the scope of tech choice is always bigger than "just the tech team". In my case, Customer Success people need to know what the impacts on our customers were and are, Product people want to know if the choice comes with new possibilities, and Sale people want to sale those possibilities. As much as I love such integration between the developers and the rest of the company, the idea of granting universal access to code repos to view markdown postmortems terrifies my boss, and therefore subsequently me, obviously.

Secondly, as familiar as markdown is, it is not a very productive option if you want to include media in it. And we want charts of various system metrics during the time of incident to be included in the postmortem.

Lastly, writing a postmortem is gradually becoming a collaboration effort, and git repo, though supports collaboration, does not do it in real time.

Considering all the options, we finally settled with a shared Google Drive in the company account. It is neither techie nor fancy. But it allows very flexible accessibility, tracks versions, natively supports embedded media, and lets multiple people collaborate in real time. We share our postmortems in a company-wide channel, and sometimes hold an additional presentation for particularly interesting ones.

Let it grow

When you have done your homework, built a foundation of trust and safety, laid out the guidelines and constantly improved it, and integrated the postmortems with your larger audience, it is probably time to take a step back and let the culture take a spin on its own. My company's postmortem culture won't be the same as Google's no matter how many Google books do I read. And as long as it works for us, it doesn't matter.

With some gentle nudges, my colleagues are picking up postmortem on their own. We have seen contribution from Product Owners and Project Managers, beside the traditional developers and devops contributors. The findings are anticipated by a large audience across the company. 

And in the latest incident, which involved the degradation of performance in a few key features of our SaaS offering over the course of a week, we identified another usage of postmortem: a postmortem updated regularly with latest incident reports, findings, and potential impacts in a near real-time fashion is a powerful communication tool across the organization, both ensures flow of information to people who need it (CS to answer questions, PM to change project plans for urgent hotfixes, etc) and allows developers to focus on their critical work without frequent interruptions.

As we grow and our system gets more sophisticated, hopefully the constructive postmortem culture would turn out to be a solid building block.

Thursday, January 16, 2020

Run, Forrest, Run!

Tl;dr: I ran my first marathon, and whined about it. Move on.

4 years after finishing my first half marathon, I finally did my first full marathon, 42k of sweat and pain. 2019 was horrible for me, through all ups and downs, the marathon plan is one of a few that keep me together. The cut off time was 7 hours. I wanted to do a sub-5 (complete the run under 5 hours), but ended up with a sub-6. I was squarely in the bottom quarter of my age group. So it wasn't all glory and stuff, but I am so glad I did it.

I must have started the training back in March or something, and didn't follow the training plan through and through, obviously. I got sick, which paused the plan by a week every time it happened. I got injuries which eventually put me out of action for a whole month. And when I was back, following the original training plan just gave me too much stress and guilt, which I certainly didn't need - my life was really low, so I forwent it and just ran whatever the fuck I wanted. That was probably 2 months ago.

The injuries were actually blessing in disguise. They forced me to rethink my running form. I picked up a book on running (that is not Murakami's autobiography) and tried to avoid "common sense" misconceptions, like most notably, landing on your whole foot. I finished 42k without any injuries. Yay!

The day I got the bib, it came with a shock. I was put into the 30-39 age group. Technically, it is not my birthday yet. And despite all the talks, I was not mentally prepared for this. Ouch! Oh and I also got interviewed.

I had never run the full distance prior to the run and in retrospective, wasn't a great idea. I now believe that the body would prepare for an extra few kms on top of the maximum distance you have covered but not by a long shot. And it makes sense, why would my body be ready for 100k if I have never run 50k? The longest I had done was 30k and that explained why from 34k I got cramps so bad.

It was also the first run that I got proper sleep the night before. And I wasn't hungry. I sure stuffed myself with loads of carb, so full that on the night before the run, I thought it was stupid, I couldn't possibly run with such a stomach. but above all, shout out to the organizers, the route had more than sufficient water, electrolyte, and banana.

One last thing, the Nike app has improved a lot between then and now. It is no longer off by 30%, and comes with cooler features. Well done Nike.


If that hasn't bored you out of your skull yet, you might want to see how my run broke down. "How did you remember all of this?" - I knew I gonna write one of this post-event, so it wasn't that hard. And I made up all the bit I didn't remember, including that I ran at all. Bwahaha.

Starting line: That's right, 42k is the first wave, the first class citizen of a marathon. With all the volunteers standing around and looking, the limelight feels good. Wait, hang on. It's already 4. Why aren't we starting? Technical issue? Great, I am trying to get some work life balance and here I am, with bugs.

0km: 10 minutes in, here we go guys!!! Let me just start my run on the Nike mobile app. Fuck fuck fuck. I dropped an energy bar while shoveling the phone back to the running belt. Screw it, I am not fighting against a wave of runners for a stupid bar. What a start.

1km: An old man with a Vietnam flag on his back is making crude joke that a bunch of fit men, leaving their horny wives and young children home, to run on the street at 4 in the morning must all have mental issues. It could have been a good joke, it could have. But why did you have to be so fucking disgusting in your choice of words old man? Urgg why are you even carrying our flag?

2km: Some already making pit stops at the trees by the sides of the road. Shit looking at them gives me the urge too. Nah. If I sweat enough, the exceeding liquid will just be repurposed in time. Probably. The 42k 4:45 pacers are here, but they seem slow (1) and have loud music on. Better keep some distance.

3km: Here it is, the first major water station. Thanks to the Starting Line Incident, I am down to 4 bars now. I should have more banana. Double portion please! There are Waldo, Doraemon, and Ao Dai right in front of me. Cute, but I am not falling behind casual cosplayers. Onwards!

5km: We are joined by a group of 21k, they seem to have a shorter route. I no longer hear the music of the 4:45 pacers. I also don't want to have my pace mess up by 21k runners. Time to speed up a bit.

7km: Just gulped down the first energy bar. Entering the beast - Phu My bridge. Still have vivid memory how it wore me out in my first 21k. Some 21k runners keep passing me. Well at least they aren't 42k.

9km: The easier quarter of the bridge was easy. Neat, there is a water station before the hardest quarter. Go in for a shower. Feel so good. Kimochi!!!

10km: Wow that's the highest point of the ascending half already? That was quick. I'm feeling great. The training works!

12km: "Coming through!" I didn't yell but it was certainly loud as I ran pass a few runners. I'm sprinting! Not supposed to put stress on my feet no? But I am on a runner's high. Gotta take advantage of this slope then.

14km: Keeping up good speed. Ketchup guy wait for me! Well, he is a 42k runner in costume which, for the lack of visual detail, only makes me think of a bottle of ketchup. I might be running too fast. There is no down hill gravity to play with. Slowing down.

15km: Crossroad. Am I turning or keeping straight? Oh there is a volunteer, neat. I asked you twice for the direction and the best you can fathom is "Huh?". You, sir, are truly an idiot.

16km: "That fires we don't put out, will bigger burn". And that's exactly why I am standing here right next to a tree, minding my own business. Here comes the same water station at the 3rd KM. Banana!  I am joined by a bunch of 21ks. This group is with pacers of 2:20. Guess I'm not doing to badly myself (2). But they are loud. I am putting in some distance.

18km: This is proceeding nicely. I'm bored. Time for some music. After all what is the point of having the pinnacle of technology in my belt. And lost a bloody energy bar.

20km: I am rejoined by the 2:20 pacers. This time the topic is on the color of the underwear the pacers are wearing. I should now add that the pacers in this group are women. "You're wearing nothing!" Someone screamed top of his lungs. Look like he is having a really good time. No, he isn't carrying a Vietnam flag. I looked. I'd love to add some distance again, but I am getting slower.

21km: Canada International School eh? Funny. I'd be here again later this afternoon to watch a game of Saigon Heat. This is a massive waste of energy. Doraemon is behind me. I'm not running behind a cosplayer, not a blue fat cat with comically short legs (and balls for hands). Just a bit faster. Entering the differentiator turn, this is the part of the route that 21ks don't join. This stretch of the route seems to last forever (3).

25km: The sun is already high. I can't possibly head to a tree this time, can I? (4) Embracing myself for a stinky toilet. Wow it's actually clean. This is awesome. The toilet, not me pissing.

27km: Doraemon is behind me again, but I can't possibly run any faster than this. I tried. Ran ahead of him for tens of meters and I would fall back to normal pace and he would pass. Not just Doraemon though. I am losing count of how many have passed me.

28km: Good morning milady, can you help me with some of that muscle spray please, on both legs? Wow that was refreshing! Thank you very much.

29km: God damn, some of that spray got onto my crotch. My balls are freezing. I sure hope they don't fall off.

30km: Got tension on the thighs. I got this. I got this. I trained for this. The app announces I have 12km left. Took me a while to calculate that I have run for 30km. Math is super hard.

32km: I have never run this far in one go. From here on, it's uncharged territory. Squats, I need to do a few squats, it stretches my thighs a bit so they are functional again.

33km: The tensions have turned into cramps. Squat. Run for a couple of hundred meters. Cramp. Squat. Rinse and repeat. It hurts so much.

34km: Arrggg I fought, but I can't run any more. My thighs got cramps. My ankles hurt from all these stomping. And the soles of my feet too, for pretty much the same reason. Worst of all, my brain seems to go blank, this is stupid, what I am even doing. I have to walk now.

35km: I have run a few short dashes, a couple of hundred meters each. One of the attempts locked my legs, almost landed my face on asphalt. The cramps are still going strong. Someone just handed me a big chunk of ice. It freezes my hands. Dude, what I am supposed to do with this? My balls are gone and that's bad enough. Here, tree, your daily ration in a solid form.

36km: Here is the plan, I gonna run between one crossroad to the next, then walk till the next crossroad. I still get cramps whenever the 2 crossroads a bit farther apart, but at least my mind has come back. The sunlight is roasting me. I miss you, sunshine.

41km: Fuck no! My legs gave up on me. Completely. I get cramps just from walking. No amount of squat seems to help. I can hear the crowd from here, so fucking close.

42km: My legs are at the stage where any excessive movement would give my cramps. The last 200 meters, the finish line is finally here. Here goes nothing. The legs don't seem to be mine, I move them like two sticks. I run. I cross the line. I get a high five. A girl put this medal around my neck. Heck, I can' even recall what she looks like. But she was wearing a Bà Ba, that was a nice touch. Under normal condition, I would have appreciated the outfit, but right now I am having a strong urge to vomit my guts out.

(1) This is probably the first sign that I didn't manage my energy level well. Too cocky. But again, I aimed for sub-5, so...
(2) I conveniently forgot the fact that 42k started 30 min in advance. But we also had a longer route since the beginning. All else being equal, I was running the first half between 2:10-2:20.
(3) It didn't. It lasted for 3.5km. Running on familiar route made me feel like it was shorter.
(4) I talked to a friend about this. Pro tip is to just pee on yourself. In a race, you probably consume enough water that your pee is transparent anyway. My shorts were white, so it didn't help much with the level of confidence. Best to do this at a water station where they usually put big bucket of water for quick shower.

Saturday, October 26, 2019

What you want to do in your life

Quite some time ago, I wrote "Good grades, good opportunities". Basically what I meant was that working hard in school is one way to propel in life, but it isn't necessarily the best way. By focusing too much on a part, one risks missing the whole spectrum of possibilities and opportunities, like a sad puppy looking at a rainbow. Yet supposed you have graduated and what you want to do with your life is anything but a fogged window, how would you get yourself into a position with better visibility?

My generation was fed that we should be following our passion. That implies somewhere in your mind there is a hardwired plan waiting to unfold, and that you know what you like - more so than any others. Well, if that was the case, we wouldn't be sitting here, would we? The people who said that also didn't really mean it. They probably meant that though the journey to discover what to do in your life is more banging your head against a brick wall than smooth sailing, don't let it demoralized you. I agree. You shouldn't underestimate your potential, and succumb to the thought you can't do what other people can.

The first step towards a way out is to ask whether you must have a job. Hint: if you can afford the question, you probably don't. Besides providing a means to get by, being in a job is also a useful experience. It is certain that each job is unique, but given a bit of abstraction, they have a fair share of similarity. For example, most engineering jobs are project-based and focus on time management, communication, and problem solving. These skills are interchangeable regardless of industry. Through the job, you also get exposed to an industry, where opportunities barely visible from outsiders. No, the world doesn't have a conspiracy against people like you and me, nor some people have it easier than others. It just changes fast and because innovation rarely follows a pattern, being in the right space is crucial.

Remember this is your day job. Working at a day job doesn't mean doing it half-heartedly. It means you shouldn't let your identity be defined by this job, just as an aspiring writer with a day job as a cram school teacher doesn't think of himself as a teacher. I usually tell fresh grads that who they are in the next 5 years is defined by what they do after 7PM. Because that is when the day job ends and real work begins. If you are one of the fortunate individuals who can afford not having a job, you get to do real work full time.

What real work? To discover what you like. This is actually harder than what it sounds like. There are a lot of jobs, ones that you don't know and ones whose name hasn't existed yet. And the dramatic way the media has described professionals only makes an accurate image harder to find. Most lawyers go through their careers without a single murder case. Literally no one hacks into a system by typing in a terminal like a maniac. And my favorite series House MD is about the exact opposite what doctors do.

I am probably the worst one to whom you can talk about the definition of "like". But I know that working on a job where you only execute instructions is not going to be very fulfilling. It is better to work in areas where you are interested to understand how things work, can exercise your freedom of thought, and get into the zone easily (where time flies). In other words, work on stuff about which you are curious. You don't need a full time job in an industry to experience it. Better yet, don't even bother with the concept of industry, it puts boundaries to your thoughts. Just pick a project that seems interesting: volunteer at a professional event, research the answer to a hard question, build a bottle rocket on your roof top. Choose a project that only takes a few weeks. Make sure it is something you can finish without any blocker. It should be a bit challenging, but not so hard that you feel overwhelmed. Online courses and books are both very helpful in this journey, though you should avoid acquiring too much data without a project to put it to use. Rinse and repeat with another project, and another, and yet another.

These self-directed activities can be a bit overwhelming at first, especially if you have got used to receiving assignments from school. But you will be able to adapt and get better. It is a part of being human, and being young.You need some initial discipline to get started, but once you do, given the project is an object of your interesting, curiosity will turn work into play and discipline is no longer needed.

If you continue down this path, you probably find something that stands out from the rest, enough that you want to come back for more. If you are lucky, you would find more than one. Depends on how good you are at multi-tasking, you might need to choose. Choose the option that leads to more options afterwards. There is a paradox of choice, but advancement in life is measured in possibilities. You go from one stage of education to another because the higher stage give you more options. There are simply more things a guy with a college degree can do compared to one graduated from high school. Building a software product gives your more options than doing business analysis. Once you have built a product, you can get into business analysis later if you prefer the narrow focus. But analyzing business is not going to make you a product manager.

Paulo Coelho's renowned The Alchemist had it that when you really want something, omens is the way the universe points you to the right direction. I have picked up many things in my life, and given up most of them. The ones that stick are usually ones I experienced some sense of beginner's luck. Perhaps it is my omen. Yours might be the same. Or it could be the urge to do something you weren't allowed to do in your younger years. Or the thrill of an audacious plan even people who have known you for years couldn't think you would pull it off (actually, the more people know you, the more likely they are to subject to stereotypical thinking). 

You should believe you have it within, that you will sense the click when the omen happens. Now go give you some more exposure of what the world has to offer.

Saturday, August 31, 2019

Postgres does not use index on gigantic numeric value

Nothing beats a Saturday morning when you wake up fresh and excited, ready for the second sleep, and realize that your Postgres database was harassed over the night and accumulated a number of downtime minutes that is too embarrassing to state here.

My database load looked like this during the outage.

buffer_mapping looks new. Postgres official documentation on this matter seems to be written by Captain Obvious: Waiting to associate a data block with a buffer in the buffer pool. Thanks for nothing. Basically buffer_mapping is a lightweight lock in read operations, my processes were fighting to reserve buffers in which to read data pages.

I have a read problem.

This query accommodates the highest number of locks:

pp_pqsql_prod=> explain select * FROM "big_table" WHERE "big_table"."id" = 9200190224054041915721;
                                           QUERY PLAN
 Gather  (cost=1000.00..4568847.85 rows=535675 width=1673)
   Workers Planned: 2
   ->  Parallel Seq Scan on big_table  (cost=0.00..4514280.35 rows=223198 width=1673)
         Filter: ((id)::numeric = '9200190224054041915721'::numeric)
(4 rows)

It is a sequence scan, on a 100M-row table, so it is obvious why it caused all the ruckus. What's less obviously is why Postgres performed a sequence scan on a primary key column.

With the help of a colleague, it appeared that given a smaller numeric value, the index kicks in just fine.

pp_pqsql_prod=> explain select * FROM "big_table" WHERE "big_table"."id" = 9200190;
                                              QUERY PLAN
 Index Scan using big_table_pkey on big_table  (cost=0.57..8.59 rows=1 width=1673)
   Index Cond: (id = 9200190)
(2 rows)

The problem is the size of the queried value. Eventually I stumbled upon this stackexchange Q&A. It can be seen in the first explain that because 9200190224054041915721 was too big, it had to be casted into numeric data type. My primary key was not that big, its data type was bigint. So it had to be casted too, because apple can't be compared to orange. What I have now is a numeric to numeric comparison, and a bigint index can't serve that.

Problem be gone and so was my morning.

Wednesday, August 21, 2019

On organizing Barcamp

If you haven't heard of Barcamp Saigon, then putting it shortly, Barcamp is an open space, also known as an unconference. As organizers, we prepare the facilities to make presentations happen, but both the content and agenda are determined by participants on the event day. Topics are submitted online a couple of months before the event to somewhat ease the hassle. But the voting is strictly limited to the event day, to ensure only the opinions of those who come matter. Topics are then allocated to rooms and time slots accordingly to their popularity. It's that simple.

I have been involving in the Barcamp scene of Vietnam for a few years. There was a 2-year hiatus when it was a make-it-or-break-it period for my employment. Yet eventually, when the dust settled, I found myself, once more, organizing one.

There are more or less a thousand participants in each Barcamp Saigon event. And though we typically only hold one a year, it is still a time sucker, with some hard work, and being a volunteer community project, one pretty much plays the hand that's dealt. Organizing Barcamp is one of the things in my life I consider unusual. Contacting seemingly random people asking for money, dealing with finance and paperwork, and coordinating other people, for an introvert, sound dreadful. Making and answering phone call push my wrong buttons.

Yet I hold on to organizing Barcamp Saigon like my dear life depends on it. I guess because as a twenty-something, I understand how hard it is to get heard in this exhilarating hyperactive city. You kinda have this neat little idea in your mind, but can't seem to find a thought partner to bounce it back and forth, nor an audience to whom the thought is relevant. Heck, even getting invited into a conference (as a speaker) is hard. You are nobody, and the conference you need doesn't exist yet. After a while, you just lose the excitement, the idea is now collecting dust in some closet, in a far corner of your clustered mind. An open space like Barcamp provides all people with an idea to share a platform to do exactly that. You don't need invitation, and the voting board of Barcamp merely shows the topic name, everyone is nobody, and content is king. In a way, an open space is the serverless of knowledge exchange. You get right into the exciting bits.

Barcamp audience is more diverse than it is specialized, more of a melting pot than a silo. If you want to argue the better approach to natural language processing, go to an AI conference; get started with conversational UI (aka chatbot), an UX conference. If you want to make the reservation experience at your restaurant more fluid, well, I guess you are out of luck, there isn't such a space in Saigon yet. But Barcamp is probably the only place you can find people with these three expertise mingling. It reminds me fondly of university time. There was these general elective courses where I got to pick non-IT ones. In those courses, everyone knew something I didn't, and the other way around. And the clash of knowledge, mindsets, and life styles was refreshing. As time goes by, my "tribe" gets smaller and more focused, usually on topics I am very much familiar with. The magic moment when something you have known since forever is basically witchcraft to me get fewer and farther in between. So I learned to appreciate Barcamp for pulling me out of my cave.

To be at Barcamp, in the middle of all communities is, for lack of a better word, a happening feeling. I get to see communities as living creatures, breathing, morphing into shape and form, and reincarnating. That doesn't necessarily mean a good thing. I have been to some Barcamps, at different places. Some were mind blowing. Some sucked though. One was particularly bad that I decided I wanted to do something about it. In a way, Barcamp is a health indicator of its local communities. But unlike GDP, which indicates wealth (1), AQI for pollution, or HPI for happiness (2), this indicator is one where individual actions can make huge difference. I am doing my part.

Open space concept doesn't have to be limited to Barcamp. In fact it shouldn't be. It should be another option in every community builder's toolkit for growing their group of like-minded people. It reduces the stress to find speakers for the next meetup, and give new participants chances to shine. And by the time open space is adopted as a second nature, perhaps Barcamp can retired for it has accomplished what it means to :)


(1) Whether GDP can indicate economy growth is a controversial topic that I am not jumping into.
(2) In Zen And The Art Of Motorcycle Maintenance, Pirsig argued that quality cannot be defined. When one defines characteristics of quality, those characteristics cease to be quality and can't describe the whole, which is greater than the sum of its parts. Is happiness quality?

Sunday, June 30, 2019

Random thoughts on a job fair

Random thoughts from a tech expo that I attended just now, not in any particular order. And some thoughts might have nothing to do with a tech expo, I am not apologizing.

Image result for job fair

  • Today I learned that there is career fair, and then there is job fair, each with a slight distinction from the other. The one I went to was probably more of the latter. But really, it was mostly about mascot costumes and freebies. So just.. fair?
  • If you are a company, and spend anywhere between some and shit lot of money for a booth, make sure your representative can represent something.
    • Recruiters should know about their company business nature, organization structure, and for what their job titles actually stand. The use of buzz words like partnership, innovation, or opportunities is bad communication and can't be used to replace solid understanding on business model, job description, or career path respectively.
    • Developers coming to support recruitment are great helps. But they should know that while being there, they are effectively salesmen. If they can't articulate, with a hint of passion, what their job is like in a day, a month, and a quarter, they are just wasting oxygen and space.
    • If you think that's a lot to ask for, you are right. It should be. Hiring is the most impactful task one can make to his organization. Bring your highest caliber executives in, because otherwise TinyPulse is bringing their Director of Technology and awe your potential hires with insightful thoughts and sincerity.
  • If you are a recruiter, you should have a plan to deal with people who have zero interest in your vacancies. The very best candidates are seldom between jobs, they are more likely to be employed and well-paid, you need to poach them to stand any chance. They might not be hooked in the positions, but you can still get them interested in the company. Getting one's contact info doesn't mean shit, people are just polite. Being accountable for answering queries on business model, offering career development path, or simply representing your culture are all significantly more important. And as far as I can tell, nobody is keeping tabs. Be different.
  • If you are an attendee, don't get annoyed if people shove flyers into your hands. They most likely didn't mean any harm. They are just trying to do a good job within their capacity, given the little or zero training they received before hand. Be kind, take flyers, and enjoy a small conversation. You might learn something new while making someone's day.
  • If you are looking for a job, reading between the lines of job titles and their descriptions is a great way to learn about a company's modus operandi, tech stack, and drawbacks. For example:
    • If one is hiring multiple data analyst positions, it has a data-driven culture.
    • Tech stack is revealed through job description and requirement is self-explanatory, but also pay attention to what is not written. Overwhelming mentions of batch processing indicates a lack of stream processing.
    • A company's drawbacks might get reflected in job requirements precisely because issues happen and the company is working on resolving them with fresh meat minds.
  • Similarly, if you want to follow a particular career path, however vague, look at a company's vacancies whose jobs rely on yours. The number of vacancies shows how mature the company is in your chosen field and how much support you gonna receive. For example, big data engineers can look for vacancies of analytics and ML; front-end developers, product and design; product managers, any sign of customer interaction. There should be a healthy combination of both business and technical vacancies for a good network effect. Nothing is worse than spending your time one something with zero impact on anyone.
  • People in outsourcing industry should get panic if they aren't working on hard problems. That you can code the same CRUD operation in 5 different languages is a waste of resource rather than something to be proud of. Local powerhouses like Ahamove or Tiki are solving hard problems that impact the life of millions of people. And it doesn't matter if they can do it in only one language. They are doing it.

PS: apparently, I wasn't the only one who thought JDs is a great medium to learn about a company. Paul Graham used job listings to learn and scale his potential competitors too.

Sunday, March 31, 2019

Do young developers have it easier?

Image result for dilbert intern

The world of computer programming is changing, fast. In the short 10 years of my career, I have had chances to witness industry standards emerged, demolished, and resurrected. Serverless, the latest abstraction upon abstractions that mankind call programming, has had its glorious days and is now probably on the peak of Gartner Hype Cycle. The time I spent on XML and its abomination of a sibling XSLT is one I would never get back, though the problems have not necessarily gone away. And everyone is deep learning left, right and center, after decades of hiatus.

As we get better at reinventing programming paradigm, streamlining development process, and specializing careers, new crops of developers come into the work force, with perspectives drastically different from their previous generations. But does growing up with an iPhone in their pockets necessarily mean these developers have it easier?

Like youngsters of any species, young developers, despite their non-existential social skills and blissful ignorance, are known for being fast, energetic, and tenacious. And I think this is where a lot of misperceptions come from. Where I give credit to young developers is their stamina. Programming is neither easy nor relaxing. It is a demanding job requiring intense concentration. Young people have the energy, especially when combined with pizza and coffee, to stick to her computers at 2 in the morning, working on intermittent quirks of an API she released earlier, and yet be delight for the new knowledge and that her work matters. Such work is harder when one gets older. He has a life he needs to keep up with. Actually, he is probably pissed that the API doesn't just work the way it should be.

That, however, does not mean young developers get shit done faster. They really don't, otherwise you would see me preparing my retire plan really soon. If anything, they tend to screw thing up faster than The Flash on red bull. I have had almost every interns accidentally drop a database, or delete something they shouldn't have, in their 3-month time. Neither do they learn faster. I am in the camp of believing as one makes progress in his career, he learns faster. Programming knowledge is cumulative. "New" ideas usually have popped up some time before in a slightly different form, in some other language or situation. Young developers, outside of school work, do not spend time tinkering assembly, distributed model, or the 7 layers of the Internet. They rip the benefits of other developers who worked out the details and packaged them nicely so that they are more approachable by others.

Does the fact that young developers rely on more layers of abstraction a bad thing? Of course not. I am glad that I can just sit here writing my post in plain English and not HTML nor CSS. That's exactly how we progress as a species, relying on abstractions built by generations before. In that direction, current mature technology favors quick iterations and shorter time to market. But wait, isn't that good old Agile, what's the big deal? Agile is about building one thing that is small yet works well, then a little more. Yet in an ecosystem with 600k apps in Play Store alone, builders aren't even sure if the one thing they are building gonna be a hit, or miss, till it is out. There are so many high quality reusable building blocks: authentication, real-time database, infinitely scalable database, etc. Writing a new software is now less about constructing and more about figuring out the right combination of gluing things together. The rise of function as a service means young developers can just build it. Work? No? Blow it up. Do it again. All before lunch starts. Older developers have been burned, they don't want to get burned again, they dread the concept of redoing till figuring out.

Though blessed with a beginner's mind, I think young developers still have a long way to go. I don't tend to find old developers that often. Part because IT is a new profession in where I am living. Part because many by their middle age move on to management level, or move out. The guys who remain hand-on throughout their career is really a force to be reckoned with. They are matured as a developer and have learned the art of leadership. They delegate appropriate work to junior people, balance between net contribution and learning something new, and thus saving both theirs and project time. People who have been around a while also see the bigger picture more clearly, both in terms of product scope and project life cycle, which problem can be delayed, and which shouldn't.

Then comes scaling. A new application gaining traction, at some point, would hit the scalability wall. Hard. Like Miley Cyrus on a wrecking ball hard. Scaling is a lot harder than throwing together a bunch of code one did not write *cough* StackOverflow *cough*, and duct-taping till it works. A scalable system takes real dedication, expertise, and a calling to learn more and better ways to be a great developer. And these don't come over night.

Young developers are intelligent, hard working, and excellent at solving the problems they are given.
Old developers are intelligent, lazy, and excellent at predicting problems in 6 months time and building the right foundation today to make it easy when time comes.