Sunny

← All weeks

What I’ve been reading (W50, 2022)

How to generate good logs · How to Build Software like an SRE

How to generate good logs

  • use a standard schema/format to make them easier to parse
  • include a transaction ID/request ID to make it easier to filter for all lines related to a single transaction/request
  • include relevant information. For example, “ERROR: Invalid msg size” is less helpful than “ERROR: Invalid msg size. Msg-id 234, expected size 54, received size 0”.
  • avoid logging personally identifiable information
  • use a logging framework instead of using print statements (this helps you have things like log levels and a standard structure)

How to Build Software like an SRE

  • Coding: No in-code fallbacks for configs, never give up on local testing, avoid state like the plague
  • Merging: Use Git, Don’t waste time on code coverage, Prioritize real-world validation, Make infra changes and code changes extremely obvious
  • Deploying (no sleep til prod): Use Docker, Deploy everything all the time, Enable limited “instant” config rollouts, and Validate deployments as they go
  • Operating: Structured logs are non-negotiable

How Razorpay developed crash-free dashboards

  • Narrowed down the errors to the right owner: Built altering systems and custom dashboards to show errors to correct engineers + wrote ErrorBoundary wrapper to tag the team and set the priority automatically.
  • Automated checks to avoid new issues: Static analysis, Tests
  • Implemented error monitoring + PagerDuty to report the critical indigents
  • Built systems to rollback release faster in case of incidents
  • Reduced mean time to acknowledge incidents

Why does TikTok’s algorithm feel so different?

  • TikTok’s recommender system is not its secret: rather, it’s the design, which, of course, isn’t secret at all. More generally, in AI applications, the sophistication of the algorithm is rarely the limiting factor. The design quality, data, and people that make up the system all tend to matter more.
  • Vertical videos, automatically scrolling to the next video, and focusing on content instead of subscriptions are some design hacks they got right.
  • Despite TikTok’s design innovations being well known, other apps have trouble copying them because they were originally designed for a very different experience, and they are locked into it due to their users’ and creators’ preferences.
  • Reminds me of this old interview from one of the original founders: https://www.youtube.com/watch?v=EdddrKJgUJg

Don’t overreact to weak signals.

  • It makes you build poorly thought-out and potentially wrong things.
  • Over-reacting happens when you make decisions prematurely. Often based on incomplete or insufficient data.
  • When you don’t have enough data, you can’t tell whether what you’re seeing is a signal or just variance.
  • One way to counteract overreacting is to look at percentages more than raw numbers. Seeing that 10 orders got rejected might make you think something is wrong. But it’s not a strong signal if that’s out of 10000.
  • If you’re A/B testing something, make sure you use a calculator that tells you whether your change in performance is significant enough.

Don’t just taste the recipe, look for the ingredients

“Deconstruct the cool things you see.

If you’d like to become a better musician and you see an amazing performance, start paying attention to how they do it. How did they promote the event? What happens in the first ten seconds of each song? How frequently are they engaging directly with the audience? Is there a progression of energy throughout the show?

When something fascinates you, pay attention to the details. The person who thinks, “That was cool” is a consumer. The person who thinks, “How did they make something that cool?” is on the path to being a creator.

Don’t just taste the recipe, look for the ingredients.”