Forem: Yiming Chen

How to Partially Restore a PostgreSQL Database?

Yiming Chen — Tue, 14 Jan 2020 00:00:00 +0000

Recently, I need to restore a PostgreSQL Database from production to staging.
Unfortunately, our staging server is not as powerful as our production server.
Specifically, our staging only has ~800GB of disk storage.
But the total spaces needed for production database is slightly over 800GB.
And due to some technical constraints, we can't add more spaces to staging.
So we choose to do a partial restore, i.e. only restoring data from tables we needed, excluding data we didn't need.
This task turns out to be harder than I expected.
I've made many mistakes until I find the correct way to do it.
Hope you can learn from my mistakes and restore your data correctly.

TL;DR

Don't use pg_restore -t for a clean restore! Don't use pg_restore -t for a clean restore! Don't use pg_restore -t for a clean restore!

Use pg_restore -l to dump a table of contents of the archive.
Comment out unneeded DATA from the ToC
Use pg_restore -L to restore the whole database except DATA that were excluded in 2

Problems with `pg_restore -t`

I searched on StackOverflow. The first solution I found was pg_restore -t. So I dropped the staging db, ran the pg_restore -t command immediately.

But I ran into three annoying issues. Only after I'd fixed all of them, did I realize I shouldn't have used pg_restore -t at the first place.

Indices were missing

The first thing I noticed was the staging app became so slow after the restore. Then I checked the indices in staging database and found nothing. This was the moment I realized that the -t option is doing its job too well.

(Apparently, another developer has already run into the same issue before: postgresql - pg_restore on a single table not restoring indexes - Server Fault)
Constraints were missing

The next issue happened when I started using the Rails app. Whenever I requested a record, I got the same error: Unknown primary key for table

This was because all the primary key CONSTRAINTS were missing (not restored) from the database. When these constraints were missing, ActiveRecord didn't know id was the primary key, which triggered the error.

After primary key constraints were restored, this issue was fixed.
Sequences/Defaults were missing

Finally, I got another issue about id. Whenever I saved a record, I got this error: ActiveRecord::StatementInvalid: PG::Error: ERROR: null value in column “id” violates not-null constraint

This was because all the SEQUENCES and DEFAULTS for the id columns were missing (not restored) from the database. ActiveRecord uses SEQUENCES and DEFAULTS to make primary keys automatically increment itself when a new record is created in the db. When these two were missing from the database, ActiveRecord could only assign primary key to nil, which violated the constraint.

More interestingly, I only restored the SEQUENCES for id columns. But the issue still persisted. So I also tried to reset the SEQUENCES (which didn't work because DEFAULTS were still missing):
```
ActiveRecord::Base.connection.tables.each do |t|
  ActiveRecord::Base.connection.reset_pk_sequence!(t)
end
```
After I restored DEFAULTS, the issue was fixed.

After these three annoying issues, I knew I did something wrong. Fortunately, it was only our staging server, so I only wasted my own time and energy.

Reading the documentation for pg_restore -t, I finally realized it wasn't the right tool for the job.

-t table
--table=table
    Restore definition and/or data of only the named table. For this purpose,
    "table" includes views, materialized views, sequences, and foreign tables.
    Multiple tables can be selected by writing multiple -t switches. This
    option can be combined with the -n option to specify table(s) in a
    particular schema.

        Note
        When -t is specified, pg_restore makes no attempt to restore any other
        database objects that the selected table(s) might depend upon.
        Therefore, there is no guarantee that a specific-table restore into a
        clean database will succeed.

As you may have notices from the Note, pg_restore makes no attempt to restore any other database objects that the selected table(s) might depend upon. (I didn't read this Note because I thought StackOverflow was trustworthy.)

Besides all the missing indices, constraints, sequences, defaults, and more, pg_restore -t won't create any tables that were not specified with the -t option. So any db migration related to the missing tables would fail on staging in the future. Which means this solution is absolutely unacceptable in this use case.

What else shall we do?

Introducing `pg_restore -l` and `pg_restore -L`

Turns out the correct answer is right below the first one in the same StackOverflow page.

pg_restore -l psql_backup.dump > db.list

Passing -l option to pg_restore would produce a table of content for the dump file. This ToC includes all the data and their locations included in the dump file:

;
; Archive created at Mon Sep 14 13:55:39 2009
; dbname: DBDEMOS
; TOC Entries: 81
; Compression: -1
; Dump Version: 1.13-0
; Format: CUSTOM
; Integer: 4 bytes
; Offset: 8 bytes
; Dumped from database version: 10.10
; Dumped by pg_dump version: 10.10
;
;
; Selected TOC Entries:
;
28920 70946 DATABASE - DBDEMOS pasha
55141; 15509 237 SCHEMA - public pasha
50798; 80741 51835 COMMENT - SCHEMA public pasha
66197; 75700 30831 ACL - public pasha
79603; 43218 86982 TYPE public composite pasha
59759; 64565 4792 EXTENSION - pg_trgm
95301; 10755 17786 COMMENT - EXTENSION pg_trgm
31637; 99705 30851 TABLE public users DBDEMOS
18171; 24739 26392 SEQUENCE public users_id_seq DBDEMOS
81822; 19526 48192 SEQUENCE OWNED BY public users_id_seq DBDEMOS
59215; 11301 6736 DEFAULT public users id DBDEMOS
2227; 53943 37511 TABLE DATA public users DBDEMOS
37029; 1384 1099 SEQUENCE SET public users_id_seq DBDEMOS
14500; 53947 96995 CONSTRAINT public users users_pkey DBDEMOS
26691; 92878 55511 INDEX public index_users_on_email DBDEMOS
85474; 88765 68415 TRIGGER public users username_update DBDEMOS

Notice that this file contains all types of data in the dump: databases, schemas, comments, access control lists (ACL), extensions, tables, sequences, defaults, table data, constraints, indices, triggers, and so on.

Comment out data we do not need

We can change this ToC by commenting out or deleting the part we don't want to restore. In this case, we don't want to restore DATA from users table, so we delete it.

;
; Archive created at Mon Sep 14 13:55:39 2009
; dbname: DBDEMOS
; TOC Entries: 81
; Compression: -1
; Dump Version: 1.13-0
; Format: CUSTOM
; Integer: 4 bytes
; Offset: 8 bytes
; Dumped from database version: 10.10
; Dumped by pg_dump version: 10.10
;
;
; Selected TOC Entries:
;
28920 70946 DATABASE - DBDEMOS pasha
55141; 15509 237 SCHEMA - public pasha
50798; 80741 51835 COMMENT - SCHEMA public pasha
66197; 75700 30831 ACL - public pasha
79603; 43218 86982 TYPE public composite pasha
59759; 64565 4792 EXTENSION - pg_trgm
95301; 10755 17786 COMMENT - EXTENSION pg_trgm
31637; 99705 30851 TABLE public users DBDEMOS
18171; 24739 26392 SEQUENCE public users_id_seq DBDEMOS
81822; 19526 48192 SEQUENCE OWNED BY public users_id_seq DBDEMOS
59215; 11301 6736 DEFAULT public users id DBDEMOS
; 2227; 53943 37511 TABLE DATA public users DBDEMOS
37029; 1384 1099 SEQUENCE SET public users_id_seq DBDEMOS
14500; 53947 96995 CONSTRAINT public users users_pkey DBDEMOS
26691; 92878 55511 INDEX public index_users_on_email DBDEMOS
85474; 88765 68415 TRIGGER public users username_update DBDEMOS

pg_restore -L db.list

-L option would tell pg_restore to only restore the data specified in the Table of Content file.

After restoring with this solution, our staging behaves the same as production. No weird issues like Unknown primary key for table anymore!

Lessons Learned

The biggest lesson from this experience for me is to always RTFW (read the f**king manual). Don't trust StackOverflow or blog posts online too much. They can be misleading more often than not. Always read the documentation to fully understand the meaning of the script, command, option I'm going to use.

It's lucky that these issues only happened to our staging server, so they didn't impact our users. But I need to train myself to be more careful on staging, and even on my local. Only after good habits are cultivated, can I perform these actions (or automate them) on production safely.

How to Write Elixiry Ruby - Result Object

Yiming Chen — Wed, 08 Jan 2020 00:00:00 +0000

Why do we need Result Objects?
- Returning nil: The Billion Dollar Mistake
- Raising exceptions: The Million Dollar Mistake
Tagged Tuple: Modeling Errors in Elixir
Result Objects: Introducing an extra level of indirection
- OpenStruct: Tagged Tuple in Ruby
- Struct: Don't Repeat Yourself
- Result Object: Adding More Behaviours to a Struct
Result Struct in Elixir?
Summary: Modeling Business Error as Data/Objects

"A language that doesn't affect the way you think about programming is not worth knowing." - Alan J. Perlis

Elixir/Erlang certainly affected the way I think about programming. Specifically, how to handle inevitable failure/error cases in my program. In Elixir/Erlang, error cases can be easily modeled as tagged tuples: {:error, reason}. But in a more Object-Oriented language like Ruby, it's hard to model error as a lightweight data structure like Elixir tuples. After some trial-and-error, I've found a great way to model errors in OO languages: Result Object.

Why do we need Result Objects?

Before explaining what is Result Object and how to write it, it's necessary to know why do we need it. Error handling is inevitable in every programming language. Take parsing CSV for example:

The file might be missing.
The header might be invalid.
The row might not match the header.

Our program cannot always run in a happy path. We have to handle errors one way or another.

Two methods I used to handle errors in Ruby are:

returning nil or false.
raising an exception.

Let's see why these two methods are not enough for us.

Returning `nil`: The Billion Dollar Mistake

Before learning Elixir, I used to return nil or false to indicate some error happens in Ruby:

def parse_csv(path)
  if file_exists?(path)
    file = read(path)
    parse(file)
  else
    nil
  end
end

# client
if results = parse_csv
  # do something
else
  # assuming file read failed
end

It worked okay when the requirements were simple. But it quickly broke down when another error needs to be handled.

def parse_csv(path)
  if file_exists?(path)
    file = read(path)
    rows = parse(file)
    if valid_header?(rows)
      # return results
    else
      # what to return here?
      # nil is already used
    end
  else
    nil
  end
end

When there are multiple error cases, a simple nil or false cannot tell the client what really goes wrong.

More importantly, if the client doesn't handle nil and use nil as a normal return value, then a NoMethodError would be raised and the whole application would crash. Countless bugs are produced due to this reason. That's why null (nil) references is called The Billion Dollar Mistake. We need a better way to handle error cases than nil!

Raising exceptions: The Million Dollar Mistake

If nil is not enough to tell what error happened, we can raise different exceptions, right?

def parse_csv(path)
  file = read!(path)
  parse!(file)
end

private

def read!(path)
  raise NoFileError unless file_exists?(path)
  # ...
end

def parse!(file)
  raise InvalidHeadersError unless valid_headers?
  # ...
end

It worked okay at first glance. Happy path is separated from sad paths. The code is much simpler and straight-forward now!

But if we dig deeper, using exceptions has tremendous costs:

Raising/Catching exceptions have a performance penalty.
Method API is now unpredictable.

We cannot see all the possible returns from a method.
- "Are there any exceptions being raised beside the normal return value?"
- "Are there any other exceptions being raised in these private methods?"
- "Are there any other exceptions being raised in other object methods that are used in this method?"
Adding behaviours to exception class is hard.

Exception classes are inherited from StandardError. So they have some default methods: backtrace, cause, etc. If we add more domain-related methods to it (e.g. reason), is it still an StandardError class?

All the costs above are fixable. We can optimize the exception raising/catching performance so it won't be an issue. We can declare all the exceptions raised by a method (like Java). We can add a base class (inherited from StandardError) and inherit this base class to reuse behaviours.

But the fundamental flaw of using exceptions to handle sad paths is unsolvable: using exceptions mixes fatal exceptions with domain errors.

Every program needs to handle fatal exceptions: syntax error, divided by zero, and so on. But the errors we are handling here are domain errors: missing files, invalid inputs, and so on.

They need to be handled differently. For fatal exceptions, the program may crash and notify a developer to fix the bug that leads to the exception. For domain errors, the program may need to recover from the error and notify the user for a better user experience.

Raising exception like StandardError is the only solution for fatal exceptions for now. That's why almost every programming language has this feature. If we use exceptions for business errors, it's polluting the purity of fatal exceptions. And both developers and operators would be confused when a non-fatal exception is raised on production.

So we need our own way to handle business errors. And different languages have different specific ways to do it.

Tagged Tuple: Modeling Errors in Elixir

Let's first see how errors are modeled in Elixir. And we'll see how it inspired me to find Result Object. In Elixir, a function would indicate its success/failure by returning a tagged tuple:

def parse_csv(path) do
  if {:ok, file} = read_file(path) do
    # ...
    {:ok, results}
  else
    {:error, reason}
  end
end

# client
case pase_csv() do
  {:ok, results} ->
    # ...
    nil

  {:error, reason} ->
    nil
    # ...
end

{:ok, results} means the operation succeeded, callee can safely assume the results is correct.
{:error, reason} means the operation failed, and callee can know what caused the error based on the value of reason.

So now, when I work in a Ruby project, I want to write similar code to handle errors more gracefully. But simply returning an array won't work elegantly as Elixir does:

def parse_csv(path)
  if read(path)
    [:ok, results]
  else
    [:error, reason]
  end
end

# client
case parse_csv(path)
when [:ok, results] # results won't be bind to the return value
  # ...
when [:error, reason]
  # ...
end

Before Ruby 2.7, we don't have pattern matching to write Elixiry case conditions. What shall we do instead? Is there a Object-Oriented way to modeling errors?

Result Objects: Introducing an extra level of indirection

We can solve any problem by introducing an extra level of indirection. - Fundamental theorem of software engineering

To get the similar result as tagged tuple in Elixir, I start with a lightweight approach. Then I gradually add more behavior and refactor it. Finally I landed with a more complicated design pattern: Result Object.

`OpenStruct`: Tagged Tuple in Ruby

My first try is to wrap the return value in an OpenStruct:

def parse_csv(path)
  if happy_path
    OpenStruct.new({valid: true, data: results})
  else
    # sad path
    OpenStruct.new({valid: false, reason: :file_missing})
  end
end

# client
result = parse_csv(path)
if result.valid
  p result.data
else
  case result.reason
  when :file_missing
    # ...
  end
end

As you can see above, OpenStruct works just like tuples in Elixir. We can create them and use them in a cheap and easy way. And all the possible return values of a method is clearer than raising exceptions.

`Struct`: Don't Repeat Yourself

As more and more OpenStruct objects are created, they start to attract behaviors. The validation logic is a natural fit to be put in these objects, so I don't need to set :valid to true/false manually.

Result = Struct.new(:path) do
  attr_reader :data, :reason

  def valid?
    if happy_path
      @data = ...
    else
      @reason = ...
    end
  end
end

# client
result = Result.new(path)
if result.valid?
  p result.data
else
  case result.reason
  when :file_missing
    # ...
  end
end

Defining Struct classes like this helps us group OpenStruct objects. So the happy result and the sad result for the same method are always bundled together. And different methods' result objects won't be mixed (since they are all OpenStruct before).

Result Object: Adding More Behaviours to a `Struct`

Finally, when these Struct classes become larger, they can have their own classes. And if needed, they can have their own validation DSL, similar to what ActiveModel::Validations does.

class Result
  validate_file_existence :path

  def initialize(path)
    # ...
    # validate here
  end

  def valid?
    # run_validations!
  end
end

Most importantly, we can discover more domain concepts just by following the lead of Result Objects. When I wrote the code to parse a CSV file and transform it into an ActiveRecord object, I discovered these classes along the way:

CSVWrapper as a wrapper around Ruby's builtin CSV library. It provides an API (headers, each_row) that's more suitable to our use cases.
CSVParser to transform a CSV file to a enumerator that returns valid domain objects.
RowParser to transform a CSV row to a valid domain objects.

To quickly summarize, business errors are cases we need to handle as normal happy paths. And just like normal business logic, we handle them by modeling them as data structures like tagged tuples or objects/classes like Result Objects.

Result Struct in Elixir?

Following the path from OpenStruct to Result Object, I'm wondering if we need to define Result Struct in Elixir.

defmodule Result do
  defstruct [:valid, :data, :reason]

  def new(data) do
    # ...
    # initialize data or reason based on the data passed in
  end
end

Result Struct seems to be an overkill in Elixir/Erlang.

Elixir already has a great pattern matching system that working with tagged tuples is simple yet powerful.
Tagged tuple is a more common way for handling errors in Elixir/Erlang.

But I think if the business logic gets more complicated, we can follow the same thinking process to extract more domain concepts, too.

Summary: Modeling Business Error as Data/Objects

In the end, tagged tuple in Elixir and Result Object in Ruby are both an extra level of indirection. They both wrap the original results and provide an additional responsibility (to say if the operation succeeded or not or if the data is valid or not). Again, they both demonstrate the power of modeling business knowledge as data. Hope this technique can help you tackle other hard business problems as well!

Growing a Result-Driven Mindset

Yiming Chen — Tue, 31 Dec 2019 00:00:00 +0000

Today is the last day of 2019.
It's time for me to review the last year.
Put everything aside, I think the biggest change of me is the mindset transition from a process-focused one to a result-driven one.

Connecting the dots (that led to a mindset shift)

Inspired by The Engineer/Manager Pendulum written by Honeycomb CEO Charity Majors, I've been doing more management work in the past 12 months. Management works like hiring, on-boarding, organizing meetings, planning, and goal setting really opened my mind. When looking back, I realize that I've been unconsciously pushing myself and my team to a more result-driven style of working. Let's see what I've done in a chronological order:

Setting up a goal-driven on-boarding program

On-boarding is the most important thing if we want everyone to become a superstar in this team. So I think of it as the highest leverage task for a team. The goal of on-boarding in my team is to help the new-hire get familiar with our team and our way of work, so he/she can start delivering value as soon as possible.

After some trial and error, we developed an on-boarding program for our team. This program looks like this:
1. On the first day a new member joins our team, a mentor would help him define a two-month roadmap. This roadmap would define goals and tasks for the trial period. For most developer roles, one of the most important goals is to deliver a feature to production (within one or two weeks). Tasks are just checklists for guiding the new member to achieve the goals.
2. At the end of every week, the mentor would have a catch-up meeting with the new member. In this quick catch-up, they would review the past week, see if goals are achieved and maybe adjust/define the next goal.
3. By the end of first month, the new member would join a more formal feedback meeting. In this meeting, we would share feedback to the new member. And the new member can share his/her feedback to the team, or the company.
4. By the end of second month, our team would decide whether this new member passes the probation period or not. Note that the decision is mostly based on if he/she has achieved the goal defined previously.
With this program, we successfully on-boarded four backend developers. They now all work at their full speed in various projects. We also identified a mismatch between another web developer and our team. He/She didn't achieve the goals we defined together even after we adjusted the goals accordingly. Sadly, we had to let him/her go.

As you can see, the goal plays an important role in the whole on-boarding process. And we found that this process worked the best when the goal is clear and it's reviewed often. If the goal is not clear, on-boarding would take much longer.
Defining expected results when writing a Tech Plan

Turning the focus back to our internal team, I noticed that two of our major projects have the same delivery issue: a high rework rate.

Team members rarely discuss the code change they are gonna make until they need to deploy it to production. So, rework is often required during the code review phase (if there is one). Or worse, sometimes hotfix is needed after deploy the change to production and there is a bug.

This time, I introduced Tech Plan (or RFC) to the team. The purpose of defining a Tech Plan is for team members to have a common understanding of and how we are going to solve a problem.

After introduced Tech Plan to more team members, we realized that the most important thing to do when defining a plan is defining the problem. Or put it in another way, defining the result we want to achieve. Only when the result is clearly defined, can we come up with different solutions targeting at the same result, compare them, and make tradeoffs.
Clarifying expected results for each Sprint

Next, I found our sprint meeting was not delivering the value as it should be. Team members only go through the todo lists together, report the progress over the past sprint, and finally define the tasks for next sprint. To me, the missing piece is the expectation of results, again. People know what they should do (add sorting to a table), but they don't know why they should do it or what result it is for (users can find the info they want more easily), so they can't provide a better solution (add search functionality).

So I helped clarify the purpose of sprint meeting: sprint meetings is for reviewing the results from the previous sprint and getting aligned on the goal for the next sprint. Most importantly, the goal here is not only a feature, but a hypothesis that if we ship feature A, then we can get outcome X. Without this hypothesis, we don't know what we are targeting at.

(And it also led me to Say NO to Processes.)
Implementing OKR to define clear expectations

Finally, I started working at every level of our team to help define OKRs for the next year/quarter.

The idea is the same as before: to reach a common understanding of what results we want to achieve.

There are still many challenges along the way, like how to define measurable key results for team building, design, project management. Let's see how we can tackle these obstacles in 2020. :)

What's the difference?

Hopefully these stories sound interesting to you. But you may ask what's the difference between action-driven (or process-focused) and result-driven, anyway?

When I started working as a developer, I treated process as the heart of our work. As long as we can use a new/fashion/professional process (like TDD), we can improve our efficiency. Better processes can definitely do that, but probably not as efficient as if people are driven by results.

When driven by results, we can experiment with different tools, different processes, compare them, and choose the best one for the current situation. Let's see a few examples:

Testing or not?

I used to believe if a project doesn't have good test coverage, then it needs to be fixed. But this thinking changed after I become more result-driven.

What's the purpose of testing? Or what's the desired results of well-tested code? Correct behaviour and easier refactoring! To achieve these result, we have way more tools than testing. For example, a well defined type system can also help us ensure our code is correct and make refactoring easy. (I guess that's why TypeScript is becoming more and more popular, and strict typed languages like Elm and Haskell can be so appealing.)

More importantly, every tool has its pros and cons. UI testing is a huge headache for automated tests. Manual testing is probably a more efficient and more cost-effective choice.

So now, I'm still a proponent of TDD, but also more open to other methods that can achieve the same result.
Detailed documentation or not?

I used to ask for comments and documentations for most methods/classes/modules. I used to copy private project discussion from Slack and paste it to our issue trackers. But again, these practices changed after I become more result-driven.

What's the purpose of writing everything down? Better communication, better understanding, and not forgetting anything! To achieve these results, detailed documentation may not help much. For example, when there's a language barrier (people can hardly read English), descriptive notes can hardly help. When others prefer discussing in person, sending an email can hardly lead to a receipt. When everyone has a different way organizing their notes, documenting everything may not help because it's only useful to the author.

So now, I still prefer to write things down, but only for myself, and I'll let everyone else to choose his/her preferable way to remember important things.
Single best solution or not?

I used to want to write the best code, the best article, etc. I used to think there is a best solution for every problem out there, everything is like an exam. But again, this thinking becomes ridiculous when I become more result-driven.

For most problems, its solution space is way larger than the problem itself. The best thing we can do is to try as many solutions we can and choose the best one among them. (In a geeky sense, most problems are NP-problems.)

Because the solution space is so large that we cannot try them all, we can never know if a solution is the best or not. There is always a better solution out there.

So now, I would start a continuous experiment for a problem if I want to get the best result.
1. Set a goal for the result.
2. Find a hypothesis that may achieve this goal.
3. Execute a plan to test the hypothesis.
4. Revise the execution and see if the goal is achieved.

As you can see from these examples, being result-driven doesn't equal to not taking actions. On the contrary, it needs more actions being taken.

Defining expectations clearly.
Finding different solutions.
Executing and comparing them. (Maybe not taking any action achieves the best result.)

These all requires more efforts than just taking actions as a nobrainer. Taking action is not the goal, achieving a better result is.

Being result-driven doesn't equal to ditching every process, neither. On the contrary, it means to always trying different solutions and always improving.

If a process leads to a good result, then use it;
If another process leads to a better result, then switch to it.

Improving process is not the goal, achieving a better result is.

Drawbacks of result-driven mindset

That being said, result-driven mindset has its cons as well. The biggest one is that the desired results need to be clear and explicit. So when facing a problem that's unclear and requires some explorations, a result-driven mindset won't help us (and may even draw us back).

Another thing is that if a goal is clear but doesn't reflect the result we want, it may lead us to a wrong direction. For example, daily page view is a common metric for marketing. And we set a goal for achieving 10,000 daily page view. But what can we get out of it? Maybe not what we want.

Finally, focusing on the result too much may let us use whatever method to achieve it. Take the same page view example. To achieve the goal of 10,000 daily page view, we can buy a lot of fake clicks. But by doing so, we've already lost what we really want.

So, setting goals is an art on its own. Let me explain it in a future post.

What I've read

Here are the books that guided me to this mindset in 2019:

Radical Focus: Achieving Your Most Important Goals with Objectives and Key Results: Christina R Wodtke, Marty Cagan: How to set clear OKRs (goals) to help you achieve your desired results.
The Right It: Why So Many Ideas Fail and How to Make Sure Yours Succeed: Experiment quickly and cheaply to test if your hypothesis can bring you your desired results.
SHAPE UP: Stop Running in Circles and Ship Work that Matters: Invest time and human resources once you've made your desired results clear.
The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win: Find your constraints/bottle-necks when you want to optimize for a better result.
Hourly Billing Is Nuts by Jonathan Stark: Focus on the results you've produced, instead of the sunk cost.

From result-driven mindset to growth mindset

A growth mindset is the ability to not be fixed on a single solution, but to be consistently testing and iterating on your idea, and improving upon what you know so that you're focusing on the problem you are trying to solve, not the solution in your head.

-- from Bot Thoughts: Growth Mindset

Result-driven mindset is only a start point. My goal for 2020 is to grow a growth mindset. We'll see what results I can achieve then. Happy New Year!

Why do we need OKR (Objectives and Key Results)?

Yiming Chen — Tue, 17 Dec 2019 00:00:00 +0000

Misalignment in a team hurts.
It hurts when a Sprint cycle ends, but none of the promised features are finished.
It hurts when asking for a reason, everyone says "I have this other project that drags me."
It hurts when project fails because we are not clear about what we really want and fight for it as a team.

I've been looking for a solution for misalignment for so long. I've tried organizing a meeting like TDD, setting Tech Plans for dev team, and defining a MVP by setting OKRs. In retrospect, they are all practices of setting OKRs. But their scopes are so small and can only cover a tiny portion of our daily works. To solve the misalignment issue once and for all, I need to apply OKR at the team level.

Fortunately, I just started working as an OKR Ambassador in my current team. My main objective is to help everyone in my team get aligned and focused with well-defined OKRs. And the following is what I show them the first: What is OKR and Why do we need it?

What is OKR?

OKR (Objectives and Key Results) is a goal system to create alignment around measurable goals. Well defined OKRs unite everyone behind the same goals (objectives). They provide focus so everyone can spend their time on things that matter the most. They keep reminding us if we are in the comfort zone or the stretching zone, and nudging us toward stretching ourselves and thus grow as a team.

Objectives and Key Results

OKR stands for Objectives and Key Results. These two elements are the core of this goal setting system. And they should always go together, as they are complementary to each other.

An Objective is a qualitative and time-bound goal which gives a vision. It needs to be inspiring and agreed by every team member. So every team member is aligned upon the same mission. A good example can be "Launch an Awesome MVP before 2020-03-31".

But it's not enough to define a qualitative vision. Everyone have a different understanding of "awesome", "MVP", and even "launch". That's why we need Key Results as a complementary.

Key Results are a few quantitative indicators which explains what the objective really means. They need to be as objective (as opposed to subjective) as possible. So every team member is aligned upon the same direction. Some good examples are:
- A web application that is used by at least 100 daily active users. ("launch" clarified)
- Users can write micro posts, follow new friends, read posts of his friends and re-post their posts. ("MVP" clarified)
- The loading time of friends' posts list is less than 100ms. ("awesome" clarified)
After the OKR is defined and agreed by every team member, we need a few other things to help us execute the OKR well.¹
Plan

Plans are actionable tasks that may contribute to at least one Key Result when finished. Inspirational Objectives and Key Results cannot be achieved by their own. So we need a road map that can lead us to our vision. Some good examples are:
- Set up a production environment before 2019-12-31.
- Design, develop, and deploy the homepage before 2020-01-10.
- Purchase ads campaign on Twitter for our new MVP before 2020-01-15.
Notice that plans may or may not contribute to a Key Result. The hypothesis behind purchasing a Twitter ads campaign is that it may help attract active users. If we find it not working as expected, we must look for another way.
Baseline

Baselines are some results that we need to keep and look out for. We ought to make some trade-offs when we try to achieve a Key Result. Baselines are here to help us make these trade-offs, or at least warn us if we make the wrong trade-off. Some good examples are:
- The overall cost must be lower than $20,000. (so we won't spend unlimited money on ads to attract active users.)
- The friends' posts list needs to show all the recent posts from every friend. (so we won't cut the loading time of this page by showing a incomplete list or even showing no posts at all.)
Future

Futures are what we need to discuss again in the future. Maybe we need to focus on the loading speed first, and postpone some new feature development to the future. So Future serves as a backlog and can be reviewed when next OKR cycle begins.

Why do we need OKR?

As it was mentioned briefly above, the main purpose of defining OKRs is to create alignment around measurable goals. And as you may see, every build block of OKR contributes to this purpose in a different way.

Objectives and Key Results defines clear direction and enables a continuous progress checking.

Missions are inspiring and agreed upon beforehand. Goals are no longer ambiguous because they are all measurable key results.

Everyone can have the same understanding of our progress so far. If the number of daily active users goes more up, we are moving closer to a successful launch. If the number goes down instead, we can review our assumptions, hypothesis, plans, or actions to see if there are other things we can do.
Plans are defined and executed by the same people.

OKR frees both managers and managees from micro-managing.

Managers knows more about higher level stuffs like strategies, missions, and visions. So they can focus on what results a team needs to achieve by defining better OKRs.

Managees knows more about lower level (but not less important) stuffs like technologies, implementations, and constraints. So they can come up with plans that match the reality the most. These plans can even be more effective or creative than what managers can imagine.

Note that doesn't mean only Managers define Objectives or Key Results, nor only managees define Plans. OKRs are defined by all team members together. And they can be defined from top down (Manager -> Managee, Objective -> Key Results -> Plan), or from bottom up (Managee -> Manager, Plan / Key Results -> Objective).
Baseline prevents us from losing the whole picture.

By human nature, we can easily lose the higher view when focusing on marching toward a narrowed goal. By setting baselines and continuously monitoring them, OKR nicely reminds us and leads us to the correct road when we make a detour.
Future helps us focus on what matters now.

We have limited resources (human power, time, money), so we have to spend them at the right places. Being able to put things in Future can help us spend our resources wisely and get better results.

More importantly, when more OKRs are defined, we can sort out priorities more easily and become more focused. How's "Launch an Awesome MVP" compared to "Build a better work place"? Some teams may prioritize prior so people may work overtime a lot before the launch date. Some teams may prioritize latter so the launch date is postponed by a month. There is no right choice here, but at least we can discuss these choices explicitly based on OKRs.

Finally, when everyone can see every team's OKR, every individual's OKR, transparency is improved by a lot. When in conflicts, people can check each other's OKR, think about how to find a plan that can contribute to each other's Objectives, or compromise to a solution based on OKR priorities (if a plan cannot be find).

Final Words

Hopefully, this short summary of OKR can help you understand this goal system better. And I'll keep posting my experiences and learnings of implementing OKR in my team. So, stay tuned!

Footnotes:

These other things are from the great management book Radical Candor: Be a Kick-Ass Boss Without Losing Your Humanity: Kim Scott.

Management, Refactor, and Prune

Yiming Chen — Tue, 26 Nov 2019 00:00:00 +0000

        <blockquote>

Being a great manager and English gardening have more in common than you might imagine.
1. small, seemingly minor activity
2. clip away the dead leaves or diseased areas of a garden to encourage healthy growth.
3. done periodically
4. The day that you stop pruning is the day that the garden is full of weeds and overrun

-- from What great managers do: Prune - Signal v. Noise

Gardening is a spectacular analogy not only for management, but also for code refactoring. We need to prune our codebase consistently. And the activity of pruning in a software project is called refactoring.

Like pruning, refactoring is a small, seemingly minor activity. Renaming a variable, making a public function private, or moving a function from a module to another. It's the accumulation of all these small, incremental steps that make our codebase a better place to live.

Like pruning, refactoring demands us to remove dead code or technical debt to encourage good design. Broken windows theory is real. Only by constantly fixing the broken windows in our codebase, can we nudge others to not break another window.

Like pruning, refactoring is a periodical activity. We cannot be refactoring all the time, or we have no time for building the real thing. Often, people schedule a repetitive event to tackle the technical debts accumulated along the way. Further more, experimental messy is a necessary evil. Because we need these experiments to tell us if there's a better design.

So if you need to explain refactoring to other non-technical people, you can use pruning as an analogy. And please remember this:

The day that you stop refactoring is the day that the project is full of tech debts and overrun.

P.S. Refactoring is everywhere in our daily life. I even treat optimizing my editor workflow as a refactoring process. Pruning is just one of the best analogies out there.

Nobody Wants To Use Your API

Yiming Chen — Fri, 22 Nov 2019 00:00:00 +0000

<p>

When writing an API endpoint or a function, I often find myself being obsessed with it, adding tons of toggles or options so this endpoint handles too many use-cases.
In the end, not all of the toggles are used by the client.
Even worse, the code is too complex to maintain.

So how do we, as programmers, avoid this pitfall? I think we should apply some product thinking here. First, think of our API as our product. Then, realize that Nobody Wants To Use Your Product. Finally, we write simple APIs that are easy to learn and easy to use.

API is the only product of programming

What is an API, anyway? According to Wikipedia, an API is a "contract" between "the client" and "the server".

Notice that "the client" and "the server" don't need to be talking over a network. So we are defining an API ("a contract") at every level of our application:

When writing a function, we are designing an API for the callers of this function.: What's the name of the function? What arguments does this function need? What results would this function return?
When writing a library, we are designing an API for other programmers.: What classes/modules does this library have? How to use these classes/modules? What results can these classes/modules achieve?
When writing a RESTful controller action, we are designing an API for the browser.: What's the path to this action? What's the verb to use here? (GET, POST, PUT, DELETE) What's the response structure?
When writing a page view with HTML, JS, and CSS, we are designing an API for the user.: What's this form used for? What would happen after clicking this button? What's the content of this page?

And everything else we do, is to fulfill the API ("the contract") we defined.

Write business code: Fulfill the business logic contract.
Write automated tests: Make sure we fulfill the business contract as specified.
Fix various bugs: The contract was broken, we need to fulfill it again.
Improve performance: Fulfill the performance contract (return the results within x seconds).
Document the API: Explain the contract to the user.

Or we can think it from the opposite. If you as "the server" don't expose an API to "the client", then it doesn't matter how much work you do behind the scenes. Because nobody can use it anyhow.

So API is the most important, if not the only, product of programming.

Nobody wants to use your API

After understanding that API == product in programming, we can start to think from a product perspective. In the article Nobody Wants To Use Your Product, Goran Peuc summarizes this presumption about human nature as:

People are not really into using products. Rather, people are more interested in the end result and in obtaining that result in the quickest, least intrusive and most efficient manner possible.

The same product thinking applies to programming as well. Other programmers are not really into using our API. They may use our API to get some data. Or they may use our API to avoid reinventing the wheel and thus simplify their own code. In a word, they use our API to achieve a result.

KISS (Keep it simple, stupid)¹

So what can we do as API designers? If nobody wants to use our API, and people use it only because they want to achieve a result, then we should make our API really simple. And by "simple", I mean easy to learn and easy to use. It's simple to say, but it's hard to do. The ultimate goal requires us to spend tons of efforts: people can spend almost no time to learn our API and start using it. So they can focus on their own concerns, and spend more time figuring out how to succeed in their own way.

Footnotes:

Keep It Simple Stupid (KISS) {Principles Wiki}

Fault-Tolerant GTD

Yiming Chen — Tue, 19 Nov 2019 00:00:00 +0000

How to do "Fault-Tolerant GTD"
"Fault-Tolerant GTD" in real life
- From blogging to micro-blogging
- Break a book apart
- Contribute to open source
Take it one step further
When "Fault-Tolerant GTD" is not a good choice
Why it's called "Fault-Tolerant GTD"

I use deadlines to urge me to finish tasks in time. In an ideal world, I can always finish a task before its deadline. My todo list won't explode due to the accumulated overdue tasks. And I won't declare todo bankruptcy.

But the real world is not ideal. I'm often over-optimized when setting a deadline for a task, since estimating is hard. I'm often not motivated to work on a boring task, but I have to finish it. I'm often just too busy to finish all the tasks.

Due to these various reasons, I always find I haven't finished, or even haven't started some of my tasks when they are already overdue.

This situation happens to project management as well. Often, a team cannot ship all the features they've promised before. (And it's totally fine, because we are stretching!)

But then what shall we do with these overdue tasks? Postponing them to the next cycle? No, it will lead to todo bankruptcy. Putting them back to the backlog? No, we need to finish some of them before a "hard" deadline.

Here is a useful method for you in this kind of situation. And I call it "Fault-Tolerant GTD".

How to do "Fault-Tolerant GTD"

The basic idea of "Fault-Tolerant GTD" is to use deadlines as an opportunity to review the task, and fallback to a sub-task, thus push the progress forward.

When a task is overdue, often the reason is that the task is too large. Maybe the task is too large so we have some steps left unfinished. Maybe the task is too daunting so we don't have enough motivation to start it. Maybe the task is low priority so we choose to work on other things first.

No matter what the reason is, let's take a few minutes to find an easier sub-task. The sub-task may be small enough. So we can finish it immediately and make some progress. The sub-task can be put into next cycle. So our next cycle is less daunting, and the smaller sub-task is more likely to get worked on. The sub-task can also be some planning work. Maybe the task isn't worth doing at all, so we can cancel it. Maybe our priorities need to change so we can execute on the right things.

In this way, we eat the elephant one bite at a ~~time~~ deadline.

"Fault-Tolerant GTD" in real life

I've been using this method for quite some time. Here are some examples from my real experiences:

From blogging to micro-blogging

Ideas are cheap. I can come up with three blog ideas a day, if I'm lucky. So I have many ideas I want to write about.

But writing a blog post is time-consuming. I need at least 2 hours to organize my words, polish them as a while, and finally publish it. And I can hardly write more than two blog posts every week.

So I have way more ideas than I can write about. Solving this problem with "Fault-Tolerant GTD" is simple. The only thing you need is a fallback chain.

Whenever a post idea is overdue, I'll consider the following actions:

Write it as a short Clipping
Write it briefly in a collection post
Post it on Twitter

Notice that these options are getting easier in order, so it's more likely for my idea to be out than stale in my notes.

Break a book apart

Reading is also a time consuming activity. My personal goal for reading is to finish 3 books every month. But what if the month ends but I haven't finished the book I want to read?

I would also find some easier sub-tasks for reading a book:

Read inspectionally to grasp the concept of this book, instead of reading the whole book word by word.
Extract chapters from the table of content and setting deadlines for each of them. Because chapters are the natural sub-tasks of a book, so this is easy.
Stop reading a book because it's not interesting enough for me.

With these two options, I can always finish a book without worrying if I would miss something important.

Contribute to open source

Working part-time on open source is not easy. I'm not pushed by a boss. The only thing I can rely on is my own motivations and disciplines.

Take contributing to hexpm for example. I would first take an issue as a task and set a deadline for it. When the deadline comes, and I didn't make much progress on the issue, I would fallback to an easier task: investigate the issue. Basically, it's about doing the planning work.

Understanding the background of the issue
Writing down the plan for how to solve this issue (example: store and display publisher of package version)
Replacing the original task with the first step in the next cycle (replace store and display publisher of package version with Add release_publishers table)

After these planning works are finished, the issue will no longer be intimidating anymore. Execution becomes smooth and easy.

Take it one step further

We've already seen the power and benefits of "Fault-Tolerant GTD". But it's not the end of the story. "Fault-Tolerant GTD" is about using deadline as an opportunity to review our task. We can take this idea one step further. Deadline is not merely an opportunity to review the task, but also an opportunity to review ourselves.

Whenever I apply "Fault-Tolerant GTD" to a task, I learn a bit more about myself.

"I may not be interested in this kind of task."
"I didn't plan well for unexpected events."
"I was over-optimistic of myself to finish all these tasks."

As I know myself better, I can plan better next time.

I may reject a type of tasks, because I know I'm not interested and I won't take any actions.
I may break a huge task down upfront, rather than waiting for the deadline to come.
I may put less tasks in my next cycles because I know how much I can take.

As I know more about myself, I can prevent these "faults" from happening in the future. In this way, I become more effective at getting things done.

When "Fault-Tolerant GTD" is not a good choice

Enough goodies, what are the drawbacks of this method? Or when shall we choose another GTD method?

The task has a hard deadline.

Obviously, this method cannot be used if we have a hard deadline. Because we cannot fallback to another task after the deadline passes.

"Fault-Tolerant GTD" works the best when the task has a soft deadline.¹ Like reading books, writing blog posts, and contributing to open source projects.
The review cycle is too long.

When we have a deadline that's far away (several months from now), we may have trouble applying "Fault-Tolerant GTD".

"Fault-Tolerant GTD" only contributes to your productivity when deadline is here. So if we keep doing nothing and just wait several months for the deadline, we will waste all the wait time.

When we are in this kind of situation, we may consider two options:
1. Shorten the feedback cycle.
2. Break the task down upfront.
3. Only use "Fault-Tolerant GTD" as a backstop to prevent doing absolutely nothing.
The review cycle is too short.

But "Fault-Tolerant GTD" also won't work if the cycle is too short.

Let's take it to the extreme. Say the review cycle is 1 second. That means every second is a deadline for a task. And we need to constantly review and fallback. Then we'll have no time for us to execute.

After all, "Fault-Tolerant GTD" or any GTD methods are only planning work. We'll need time to really dive into the real work and finish it.

After considering these drawbacks, I now only use "Fault-Tolerant GTD" for tasks without a hard deadline. I also follow a combination of 1-day, 1-week, 1-month, and 1-year cycles. And it works flawlessly.

If you have any questions implementing "Fault-Tolerant GTD", I'd be glad to help.

Why it's called "Fault-Tolerant GTD"

After all these explanations about "Fault-Tolerant GTD", you may still be wondering, why is it called "Fault-Tolerant GTD"?

Both the name and the idea of "Fault-Tolerant GTD" are inspired by Erlang.

Erlang is designed to be a "fault-tolerant" programming language. (Here, "fault" means errors or exceptions that occur in our programs.) What Erlang does when a program fault occurs is to fallback to an easier program task, like restarting the task.

I'm just applying the same idea to daily task management. A "fault" occurs in task management when a task dues. And to "tolerate" this "fault", we fallback to an easier task.

Most of the time, we apply what we learned from our life to programming, but we can also do the opposite. You may find some programming paradigm helpful with your life, too!

Footnotes:

This restriction also reminds me of Structured Procrastination,
a GTD tactic that also requires us to choose tasks wisely.

How to Start Consuming Less and Creating More?

Yiming Chen — Fri, 15 Nov 2019 00:00:00 +0000

        <blockquote>

The most reliable way to get better results is to just produce more crap.
Most knowledge worth having comes from practice.
- It comes from doing.
- It comes from creating.
The real tragedy of modern technology is that it’s turned us into consumers.

-- from Consume less, create more

Inspired by this article, I decided to consume less crap, and to create more crap of my own.

How to do that? It's as simple as any improvements:

Measure your baseline
Identify the bottleneck
Improve the bottleneck
Review the progress
Repeat

Below is my current status and how I plan to improve it:

Consuming

I consume a lot. Every month, I watch ~3 movies, ~10 episodes of tv shows, ~30 YouTube videos, read ~3 books, ~150 blog posts, and ~3000 of tweets.
- Movies are an important part of my entertainment life. I can't leave them once and for all. But I can be more selective in what I watch. And set a deadline for what I want to watch. If I don't watch it before the deadline, then I won't.
- TV is also a huge time consumer. So I don't watch the traditional TVs (that you can't decide what you watch) anymore. I choose what I want to watch and allocate time for them.
- Good books are a joy to read. And it can worth hundreds of blog posts due to its density of knowledge. But if I can't apply the knowledge I learned from the book, then reading it would be a waste of time.
- Blog posts and tweets are my way to follow the newest trends. But it's totally okay to not read them. We don’t miss what we don’t see. My current strategy is to read for accomplishing a goal. Google is always my best friend when I need to find some blog posts to learn about a topic.
Creating

On the other hand, I create a lot less. Every month, I open ~3 pull requests on GitHub, write ~5 blog posts, post ~25 tweets.

Given a high consume-create ratio like this, I sometimes feel that I don't get enough chances to apply what I've learned.

So, I'm trying to do my best to create more craps and make sure they are read or used by someone else.
1. Contribute more to open source projects like hexpm/hexpm.
2. Post 2 blog posts every week (= 8 posts every month).
3. Use Twitter as a micro-blogging platform. It will be a backstop for my blog post ideas. If I can't convert an idea to a blog post in time, I would post it to Twitter.

If you like the craps I've created so far, please do follow me on Twitter, or shot me an email to discuss any topics. :)

Why Billing by Hours is a Bad Idea?

Yiming Chen — Wed, 13 Nov 2019 00:00:00 +0000

        <blockquote>

No one is paid to take action, people are paid to achieve results.

I really like this quote from Million Dollar Consulting. It's the best summary for why we need to be results driven (if we want to get paid and make money). It also resonates with my terrible experiences in some hourly-billing projects. I want to share these experiences here, explain why hourly-billing caused these troubles, and what's the alternative way to hourly-billing.

The hourly-billing nightmare

An hourly-billing project is doomed from the start. But it's just the beginning. When building and delivering the project, we'll find more troubles due to hourly-billing.

A fancy plan with a tight timeline

Before a project gets started, we need to agree with the client what to build. Often, the project idea is from a simple need of the client. With hourly-billing, it's natural to build upon a simple need, adding more and more fancy features. Because why not? Client would need it, and we can bill more hours building it.

Then we need to give a proposal for this fancy plan, with a budget estimation (price / fee). To get this budget estimation, we need a time estimation, since we are billing for hours, right?

After reading the draft proposal, a developer estimates: "For these many features, we may need two years."

"What? That's impossible! The client need it as soon as possible. Can we cut it to two months so it also fits into their budget? Just use the native style and whatever third party library you can find."

At this point, the project is already doomed. After all, good, fast, and featureful. We can only pick two. With a short time frame and a tight budget, we have to cut the only thing left in the equation: software quality.

Bad executions under a tight budget

Once the project starts with a fixed budget and a fixed timeline, every minute counts.

To ship the project in two months, a lot of meetings are skipped. Because a 1 hour meeting with 5 people counts as 5 hours. It's just cheaper to dive into design and code.

After two weeks of prototyping and coding, we finally ship something, only to find it's not what the user needs.

(This kind of story always reminds me of the old saying: weeks of coding can save you hours of planning.)

Besides, no one has written a line of automated test. Why? Because programmers know the requirements are definitely going to change. Changing and maintaining these tests are going to make our budget tighter!

At the end of the sprint, what we've got is an untested, useless application. We need to hack it to make it really useful to the user. It's what we call a Death March.

Time for a time usage report

After we've finished the Death March, there's one last challenge before we get fully paid: generate the time usage report and justify for it.

Recording time usage is not easy. Most people don't have the habit of doing it. When we are asked how much time we spent on a feature, the only thing we can only rely on is our memory. So the time usage report we generate would be inaccurate.

Then we send the report to the client. Even if the report is accurate, the client would have a lot of questions. These questions would be difficult to answer, because few clients are familiar with software development. For example,

"Why did a simple feature ("centering things with CSS") took you an hour?"
"Why is the time usage not like what you estimated? Are you not following the plan?"

Why is hourly-billing a trouble maker?

Of course, these troubles are not caused solely by hourly-billing. Bad executions like poor planning, no automated testing do have their own problem, too. But with hourly-billing, it's easier to fall into these traps like unrealistic estimations, lack of planning, no tests, etc. Plus, even if everything being equal (plan well, test well, record time well), we would still be questioned for how the time is being used. So it's worthwhile to look into why hourly-billing is bad and to see if we have other alternatives.

I think the problem is at the core of hourly-billing: coupling time and money together. This coupling not only puts our relationship with our client in danger, but also makes scaling our business hard.

The tension between our client and us

Due the coupling of how much time we spend and how much money we make, we want to spend more hours on a project. But what's our clients really want? Result! All they want is a product, a solution that can make their life/business better. This misalignment is the root conflict between our client and us.

Due to this misalignment, we may prefer a fancier solution that costs more time. But what the client really needs is a simple working solution, ASAP.

Due to this misalignment, we may have less incentive to improve, which are required to get a better result.

"With this new X, we can deliver the same thing in half the time! But how can we charge it the same as before? Maybe we wait until next time."

Due to this misalignment, we may center our internal conversations around how many hours we've spent, but not how many values we can generate. People tend to be misled by these conversions and thus lose their focus on value.

Due to this misalignment, we may lose trust from our client. If we are not really thinking from the value our client can get, they will find it out one day.

A better alternative must keep us aligned with our client, by focusing on what value we can bring to them.

It doesn't scale

The coupling of time and money makes the price calculation simple: Price = Hours * Rate. So scaling with hourly-billing should be simple, too. Our only two options are obvious, but neither of them are scalable:

Spend more hours in projects.

We can do this by working overtime, which is not sustainable for our own health. And we would hit the hard limit (24-hour-per-day) quickly.

Or we can try to multi-task on different project at the same time. I don't think it's possible for me or any normal human beings. But if you know how to do it without affecting the quality of your output, please do teach me. :)

The last resort is to hire some junior people to spend their time for us. But these people raise our internal cost by a lot, not only with their hourly rate, but also a management cost.

A better alternative should decouple the time we spend and the fee we get paid, so we can get a higher leverage.
Raise our hourly rate.

To raise our hourly rate, we need to find more valuable clients. They value their projects more, they value our work more, and more importantly, they have more budget to afford a higher hourly rate.

Then the question becomes "How to justify a $1,000 hourly rate?" A smart sales person would start from our client's perspective: "Our experience can bring you tremendous value. By finishing this feature in 5 hours, you would get far more than $5,000."

This is the A-ha moment for me: if we need to "justify" our pricing from the perspective of value, then why not give a pricing based on value, so we don't need to "justify" it anymore?

Bill by the value we've generated, not the time we've spent

The solution to the coupling problem is easy and simple: just decouple time and money and stop charging by hours.

So how to decide how much we are going to charge? Again, we need to start from our client's perspective: how do they convince themselves to pay.

Our clients are paying us for a solution to make their lives better. In another word, they are paying for the value this solution brings to them. So the decision process is actually simple: if the value is higher than the cost, then it's a good deal.

This decision process gives us a clear way to set our pricing. Following this thinking, my programmer mindset shifts from writing code to a more complete one:

Identify the real problem first.
Come up with multiple solutions with their own trade-offs.
Compare different solutions and choose the best one in the current situation.
Implement the solution.
Review to see if the real problem get solved.
Iterate.

And this magical thinking is just a simple sentence: Our pricing is proportional to the value we generated.

Think of time as a series of timestamps

Yiming Chen — Wed, 06 Nov 2019 00:00:00 +0000

I've been logging my daily time usage for almost 5 years.
And by "daily time usage", I mean "every minute".
Every day, I know where I spend every minute (sleeping, working, reading, writing, coding, etc.).
As you may expected, it is not an easy task to do.

Logging daily time usage is not easy

When I first started logging time, I thought of time usage as ranges: every task T has a clock, and it starts from a point t1, ends at a point t2, lasts for m minutes. It's intuitive, but it didn't work well.

When starting a task, I need to note down what I'm going to do. But it may stop me from doing the real task.
When adjusting a range, I need to adjust two data points at the same time: change one of t1 or t2, then recalculate m.
When switching from task A to task B, I need to stop task A first, then start task B.
When multitasking, I need to keep multiple clocks running at the same time.

After several rounds of iteration, I now think of time usage as just two timestamps: every task T has a start point t1, and an end point t2.

Here is how I do it now:

Whenever I need to spend a block of time on something, I jot down a timestamp t1.
Then when I finish the task, I jot down a timestamp t2.
Finally I specify which task T I was working on, and set t1 as its start time, t2 as its end time.
If I start another task T' immediately, t2 is the start time for T'.

It's a more lightweight and flexible workflow.

Logging code time usage is not easy, either (and how Heroku solves this problem)

More interestingly, if you think of time logging beyond our daily time usage, you'll find many other things that require time logging. One of them is logging time spent in different parts of our program. Heroku faced this exact problem before. In this post, Fred Hebert shared how they solved it.

Their first straightforward solution was like how I started logging my daily time:

note t1
doing the real work T
note t2
calculate the duration m

But this solution would couple the code for time logging and the code for doing the real work together.

Then a better solution was like what I am doing now:

note t1 with a label l1: T started
note t2 with a label l2: T stopped
the analyzing library uses t1, l1 and t2, l2 to calculate how long task T took
the analyzing library may use t1 or t2 for other task calculations like T'

With this solution, Heroku now fully decouples how the real work gets done (business logic) from how the time logs are analyzed (logging logic).

The power of data and deferring decisions

Optimizing time logging reminds me of Use Return Value to Defer Decisions.

In the case of Heroku, with the help of additional label l1 and l2, we can defer the decision of how to analyze time logs to later.

Again, this is the beauty of functional programming: constraints breed creativity. To quote Fred:

it was the structural constraint of already being in an immutable context that prompted the cleaner design.

Forem: Yiming Chen

How to Partially Restore a PostgreSQL Database?

TL;DR

Problems with pg_restore -t

Introducing pg_restore -l and pg_restore -L

Lessons Learned

How to Write Elixiry Ruby - Result Object

Table of Contents

Why do we need Result Objects?

Returning nil: The Billion Dollar Mistake

Raising exceptions: The Million Dollar Mistake

Tagged Tuple: Modeling Errors in Elixir

Result Objects: Introducing an extra level of indirection

OpenStruct: Tagged Tuple in Ruby

Struct: Don't Repeat Yourself

Result Object: Adding More Behaviours to a Struct

Result Struct in Elixir?

Summary: Modeling Business Error as Data/Objects

Growing a Result-Driven Mindset

Connecting the dots (that led to a mindset shift)

What's the difference?

Drawbacks of result-driven mindset

What I've read

From result-driven mindset to growth mindset

Why do we need OKR (Objectives and Key Results)?

What is OKR?

Why do we need OKR?

Final Words

Footnotes:

Management, Refactor, and Prune

Nobody Wants To Use Your API

API is the only product of programming

Nobody wants to use your API

KISS (Keep it simple, stupid)1

Footnotes:

Fault-Tolerant GTD

Table of Contents

How to do "Fault-Tolerant GTD"

"Fault-Tolerant GTD" in real life

From blogging to micro-blogging

Break a book apart

Contribute to open source

Take it one step further

When "Fault-Tolerant GTD" is not a good choice

Why it's called "Fault-Tolerant GTD"

Footnotes:

How to Start Consuming Less and Creating More?

Why Billing by Hours is a Bad Idea?

The hourly-billing nightmare

A fancy plan with a tight timeline

Bad executions under a tight budget

Time for a time usage report

Why is hourly-billing a trouble maker?

The tension between our client and us

It doesn't scale

Bill by the value we've generated, not the time we've spent

Think of time as a series of timestamps

Logging daily time usage is not easy

Logging code time usage is not easy, either (and how Heroku solves this problem)

The power of data and deferring decisions

Further reading

Problems with `pg_restore -t`

Introducing `pg_restore -l` and `pg_restore -L`

Returning `nil`: The Billion Dollar Mistake

`OpenStruct`: Tagged Tuple in Ruby

`Struct`: Don't Repeat Yourself

Result Object: Adding More Behaviours to a `Struct`

KISS (Keep it simple, stupid)¹