Sunny

← All weeks

What I’ve been reading (W7, 2023)

A better way to build software (chat with Karri Saarinen · Big Data is Dead

A better way to build software (chat with Karri Saarinen, Cofounder and CEO of Linear)

  • Linear is a high-frequency product, similar to email clients, with daily or multi-short sessions. It’s a heavy front-end product, and most actions involve managing things in the UI. A tool like Linear is about communicating and coordinating work. Getting people to use the tool requires making it as frictionless as possible and, ideally, something they want to use instead of having to use it.
  • Agile and Scrum both things focus on the process instead of outcomes. We often default to the most constrained approach because it uncovers more reasons why the solution is too constrained. Rather than features, it’s essential to think about workflows and how different loops play together in the software development process.
  • We want to build the best tool and believe the only way to do that is to have opinions. Having opinions is like having values; it’s hard. You have to choose them and defend them. It’s easier to go the flexible route and say, “you do you, man.” But this approach doesn’t work for a company-wide collaboration tool because it’s not productive if everyone invents their workflows.
  • In the last 20 years, our industry has had bureaucratization. We focus too much on processes and forget that we need to build stuff quickly and, ideally, the right things. We should find ways to encourage people to make real progress (problems solved, code shipped).
  • Keyboard shortcuts and command menus only feel great if your product is fast. This architecture allowed us to have a global cmd+z undo function that helps with mistakes that most products don’t have.
  • We don’t believe velocity or metrics provide clear answers for productivity but uncover problems or areas of improvement. We instruct teams not to focus too much on the numbers - like how many points you got done - but to look at the patterns.
  • We have a high bar, and technical skills are not enough. We look for critical thinking, taste, communication, and judgment in all roles. In any role, we like to think about how this person can push the role forward and execute at a much higher level than commonly expected.
  • Have as small, focused, experienced, IC-heavy team as possible. For an average software startup, pre-PMF, keeping the headcount under ten people is essential. Adding more people never makes things move faster or makes you more focused.

Big Data is Dead

  • For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size. “Your data is too big for your puny systems,” was the diagnosis, and the cure was to buy some new fancy technology that can handle massive scale. Of course, after the Big Data task force purchased all new tooling and migrated from Legacy systems, people still had trouble making sense of their data. They also might have noticed that data size wasn’t the problem if they were paying attention.
  • Just because the amount of data being generated increases doesn’t mean that it becomes a problem for everyone; data is not distributed equally. Most applications do not need to process massive amounts of data. This has led to a resurgence in data management systems with traditional architectures; SQLite, Postgres, and MySQL are all growing strongly, while “NoSQL” and even “NewSQL” systems are stagnating.
  • Ten years in, that future just hasn’t materialized. We can validate this in several ways: by looking at data (quantitatively), asking people if it is consistent with their experience (qualitatively), and thinking it through from first principles (inductively).
  • Most people don’t have that much data. Customer data sizes followed a power-law distribution. The largest customer had double the storage of the next largest customer, the next largest customer had half of that, etc. So while there were customers with hundreds of petabytes of data, the sizes trailed off quickly. Thousands of customers paid less than $10 a month for storage, which is half a terabyte. Among customers using the service heavily, the median data storage size was much less than 100 GB.
  • Modern cloud data platforms have separate storage and computing, meaning customers are not tied to a single form factor. This, more than scale-out, is likely the single most important change in data architectures in the last 20 years. Instead of “shared nothing” architectures that are hard to manage in real-world conditions, shared disk architectures let you grow your storage and computing independently. The rise of scalable and reasonably fast object storage like S3 and GCS meant you could relax many constraints on building a database.
  • The amount of data processed for analytics workloads is almost certainly smaller than you think. Dashboards, for example, are very often built from aggregated data. People look at the last hour, day, or week’s worth of data. Smaller tables tend to be queried more frequently, and giant tables more selectively. Customers with moderate data sizes often did fairly large queries, but customers with giant data sizes rarely queried vast amounts of data. When they did, it was generally because they were generating a report, and performance wasn’t a priority.
  • A huge percentage of the data that gets processed is less than 24 hours old. By the time data gets to be a week old, it is probably 20 times less likely to be queried than from the most recent day. After a month, data mostly sits there. Historical data tends to be queried infrequently, perhaps when someone runs a rare report.
  • Data is a liability. Keeping data around costs more than just storing the physical bytes. Under regulations like GDPR and CCPA, you are required to track all usage of certain types of data. Some data needs to be deleted within a certain period of time. You may violate statutory requirements if you have phone numbers in a parquet file that sit around for too long in your data lake.

[Apple Avoids Tech Layoffs Because It Didn’t Overhire Like Google, Amazon](http://(https://www.bloomberg.com/news/articles/2023-02-10/apple-aapl-avoids-tech-layoffs-because-it-didn-t-overhire-like-google-amazon?cmpid=BBD021323_TECH&leadSource=uverify%20wall&utm_source=pocket_reader)

  • Apple added fewer employees than other big tech firms. On top of that, according to data compiled by Bloomberg, the company generated far more revenue per new hire than its peers.
  • Its headcount increased just 20% from 2020 to 2022, compared with a 60% gain at Alphabet and a near-doubling at Amazon. Those two companies went on to announce layoffs of roughly 30,000 combined.
  • This signals a better quality of management at Apple compared to other technology companies that read the signals during the pandemic the wrong way.
  • Many tech companies admit they hired too much during the pandemic, betting that lifestyle changes — remote work, e-commerce spending and video-game habits — would bring a bigger windfall. Now they’re dealing with the aftermath.
  • Headcount can’t fully explain Apple’s edge over competitors. The company also generates some of the highest sales per square foot — indicating that its efficiency goes beyond hiring policies.
  • Apple also generated much more revenue per additional employee during the pandemic years than it did in the previous three-year stretch.
  • Apple is frugal by nature. It comes down to the management’s stewardship of shareholder dollars and a tight focus on what growth opportunities to invest in

Tech trends for 2023

  • The Pandemic broke all offline habits. Department stores which dominated retail for a century, started to disappear. Newspaper share of advertising continues to decline.
  • E-commerce goes back to the trend line (after the COVID peak) — 20-50% growth pulled forward
  • 250K tech layoffs were announced in the last 12 months
  • Ad revenues for Alphabet / Meta slowed down
  • Amazon has now overtaken Walmart’s revenue. Amazon doubled its warehouse space in 2019.
  • Software is eating advertising. $700bn industry turns over.
  • Amazon’s ad business overtook Amazon Prime’s revenue in 2022. They are the world’s largest advertiser. Amazon had $39bn ad revenue in 2022. Amazon Ads have much higher margins than infrastructure.
  • Chinese manufacturers and suppliers are going direct on Amazon — Anker (phone accessory company) has built a $2bn business on the Amazon marketplace.
  • Shopify powered $175n of sales in 2021 (45% the size of Amazon marketplace)
  • Meta spent $14bn on metaverse in the last 12 months and is years from consumer launch.
  • Generative AI is everywhere.
  • India and China are now almost half of the global mobile traffic.

APIs as ladders; what developers actually want from your APIs (cc: )

  • Convenient APIs are easy to get started with. Beginners don’t adopt the API if it is not convenient.
  • Gradual APIs are easy to learn. Novices find it complicated and don’t become experts if it is not gradual
  • Flexible APIs can solve a lot of problems. Experts eventually “eject” to something else to solve their problems if it is not flexible.
Substack edition

View the original post

A better way to build software (chat with Karri Saarinen, Cofounder and CEO of Linear)

  • Linear is a high-frequency product, similar to email clients, with daily or multi-short sessions. It’s a heavy front-end product, and most actions involve managing things in the UI. A tool like Linear is about communicating and coordinating work. Getting people to use the tool requires making it as frictionless as possible and, ideally, something they want to use instead of having to use it.

  • Agile and Scrum both things focus on the process instead of outcomes. We often default to the most constrained approach because it uncovers more reasons why the solution is too constrained. Rather than features, it's essential to think about workflows and how different loops play together in the software development process.

  • We want to build the best tool and believe the only way to do that is to have opinions. Having opinions is like having values; it's hard. You have to choose them and defend them. It’s easier to go the flexible route and say, “you do you, man.” But this approach doesn't work for a company-wide collaboration tool because it’s not productive if everyone invents their workflows.

  • In the last 20 years, our industry has had bureaucratization. We focus too much on processes and forget that we need to build stuff quickly and, ideally, the right things. We should find ways to encourage people to make real progress (problems solved, code shipped).

  • Keyboard shortcuts and command menus only feel great if your product is fast. This architecture allowed us to have a global cmd+z undo function that helps with mistakes that most products don't have.

  • We don't believe velocity or metrics provide clear answers for productivity but uncover problems or areas of improvement. We instruct teams not to focus too much on the numbers - like how many points you got done - but to look at the patterns.

  • We have a high bar, and technical skills are not enough. We look for critical thinking, taste, communication, and judgment in all roles. In any role, we like to think about how this person can push the role forward and execute at a much higher level than commonly expected.

  • Have a small, focused, experienced, IC-heavy team as possible. For an average software startup, pre-PMF, keeping the headcount under ten people is essential. Adding more people never makes things move faster or makes you more focused.

Big Data is Dead

  • For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size. “Your data is too big for your puny systems,” was the diagnosis, and the cure was to buy some new fancy technology that can handle massive scale. Of course, after the Big Data task force purchased all new tooling and migrated from Legacy systems, people still had trouble making sense of their data. They also might have noticed that data size wasn’t the problem if they were paying attention.

  • Just because the amount of data being generated, increases don’t mean that it becomes a problem for everyone; data is not distributed equally. Most applications do not need to process massive amounts of data. This has led to a resurgence in data management systems with traditional architectures; SQLite, Postgres, and MySQL are all growing strongly, while “NoSQL” and even “NewSQL” systems are stagnating.

  • Ten years in, that future just hasn’t materialized. We can validate this in several ways: by looking at data (quantitatively), asking people if it is consistent with their experience (qualitatively), and thinking it through from first principles (inductively).

  • Most people don’t have that much data. Customer data sizes followed a power-law distribution. The largest customer had double the storage of the next largest customer, the next largest customer had half of that, etc. So while there were customers with hundreds of petabytes of data, the sizes trailed off quickly. Thousands of customers paid less than $10 a month for storage, which is half a terabyte. Among customers using the service heavily, the median data storage size was much less than 100 GB.

  • Modern cloud data platforms have separate storage and computing, meaning that customers are not tied to a single form factor. This, more than scale-out, is likely the single most important change in data architectures in the last 20 years. Instead of “shared nothing” architectures that are hard to manage in real-world conditions, shared disk architectures let you grow your storage and computing independently. The rise of scalable and reasonably fast object storage like S3 and GCS meant you could relax many constraints on building a database.

  • The amount of data processed for analytics workloads is almost certainly smaller than you think. Dashboards, for example, are very often built from aggregated data. People look at the last hour, the last day, or the last week’s worth of data. Smaller tables tend to be queried more frequently, and giant tables more selectively. Customers with moderate data sizes often did fairly large queries, but customers with giant data sizes rarely queried vast amounts of data. When they did, it was generally because they were generating a report, and performance wasn’t a priority.

  • A huge percentage of the data that gets processed is less than 24 hours old. By the time data gets to be a week old, it is probably 20 times less likely to be queried than from the most recent day. After a month, data mostly just sits there. Historical data tends to be queried infrequently, perhaps when someone runs a rare report.

  • Data is a liability. Keeping data around costs more than just storing the physical bytes. Under regulations like GDPR and CCPA, you are required to track all usage of certain types of data. Some data needs to be deleted within a certain period of time. You may be violating statutory requirements if you have phone numbers in a parquet file that sit around for too long in your data lake.

Apple (AAPL) Avoids Tech Layoffs Because It Didn’t Overhire Like Google, Amazon

  • Apple added fewer employees than other big tech firms. On top of that, according to data compiled by Bloomberg, the company generated far more revenue per new hire than its peers.

  • Its headcount increased just 20% from 2020 to 2022, compared with a 60% gain at Alphabet and a near-doubling at Amazon. Those two companies went on to announce layoffs of roughly 30,000 combined.

  • This signals a better quality of management at Apple compared to other technology companies that read the signals during the pandemic the wrong way.

  • Many tech companies admit they hired too much during the pandemic, betting that lifestyle changes — remote work, e-commerce spending and video-game habits — would bring a bigger windfall. Now they’re dealing with the aftermath.

  • Headcount can’t fully explain Apple’s edge over competitors. The company also generates some of the highest sales per square foot — indicating that its efficiency goes beyond hiring policies.

  • Apple also generated much more revenue per additional employee during the pandemic years than it did in the previous three-year stretch.

  • Apple is frugal by nature. It comes down to the management’s stewardship of shareholder dollars and a tight focus on what growth opportunities to invest in

Tech trends for 2023

  • The Pandemic broke all offline habits. Department stores which dominated retail for a century, started to disappear. Newspaper share of advertising continues to decline.

  • E-commerce goes back to the trend line (after the COVID peak) — 20-50% growth pulled forward

  • 250K tech layoffs were announced in the last 12 months

  • Ad revenues for Alphabet / Meta slowed down

  • Amazon has now overtaken Walmart’s revenue. Amazon doubled its warehouse space in 2019.

  • Software is eating advertising. $700bn industry turns over.

  • Amazon's ad business overtook Amazon Prime's revenue in 2022. They are the world’s largest advertiser. Amazon had $39bn ad revenue in 2022. Amazon Ads have much higher margins than infrastructure.

  • Chinese manufacturers and suppliers are going direct on Amazon — Anker (phone accessory company) has built a $2bn business on the Amazon marketplace.

  • Shopify powered $175n of sales in 2021 (45% the size of Amazon marketplace)

  • Meta spent $14bn on metaverse in the last 12 months and is years from consumer launch.

  • Generative AI is everywhere.

  • India and China are now almost half of the global mobile traffic.

Originally on Substack